My face, with triangles and points plottedIn this project, we experimented with manipulating images via their geometries: identifying key points, manipulating them, and then mapping the pixels onto the resulting new geometry. We used these techniques to generate smooth animations morphing one face into another, to calculate population means, to map one face onto the geometry of another, and to generate caricatures by extrapolating from the mean.
I chose points manually for the image of me and the image of George, in the same order each time. I focused on edges and important facial features: top of the forehead, eyes, nostrils, edges of the mouth, and bottom of the chin. I then used scipy's Delaunay implementation to generate triangles.
My face, with triangles and points plotted
Generating the midway face required implementing a function morph, which takes two images, their corresponding keypoints, a triangulation, and two fractions (one for warping, one for dissolving), and outputs a single image representing the combined image at the two fractions. I used RectBivariateSpline (one spline for each channel) to interpolate pixels and settled on multiplying the destination triangle's vertex position matrix by the inverse of the source triangle's vertex position matrix - "fancier" techniques involving linear algebra solvers gave me nothing but trouble.
Choosing good points turns out to be a pain! George Clooney and I come from somewhat different genetic backgrounds. He's also much older - he has bags under his eyes that are more visible than mine, and it was hard to figure out a mapping that accounted for that.
Half Abe, half George
Half-ish Abe, half-ish George (warp_frac = 0.3, dissolve_frac = 0.5 - seems to prevent the brighter pixels in the George image from overpowering mine)
Generating a morph sequence was a simple matter of calling morph 45 times with smoothly interpolated fractions and saving the result as a GIF with imageio.mimsave.
Warp speed!I chose to work with the Danes for this part, since the dataset is annotated well, available freely, and almost a little tiny bit diverse (I think I've seen less diverse groups of Danish AI researchers, at least!). I did encounter some issues with the points that the researchers chose: they ignored the forehead completely, leading morph results to look a bit alien and rarely flattering.
Nevertheless, manually re-annotating the dataset was not within the time constraints of this project. I created keypoints for my own photo corresponding to those used in the Danish dataset.
The average Danish AI researcher, circa 2004
One AI researcher warped to the mean - this one is quite natural
Another - not so natural
And another - artifacts from the limited keypoints are clear here
Me if I were a Danish AI researcher in 2004 (and a severe demonstration of the forehead issue)
And vice versa - not flattering
A caricature of my face's most asymmetrical features, created by mapping my face's points onto those points plus 80% of the difference between them and the Danes' mean. I'll blame it on lens distortion from using my phone's front camera.