CS 194-26: Image Manipulation and Computational Photography

Project 4: Face Morphing

William Tait

Fall 2017


Overview

This project served as an introduction to 2D affine transformations. This is a class of operations that can "move" vectors and other data in space – scaling, rotating, translating, and the like. Earlier in this class we focused heavily on image blending; we now shift to such transformations to talk more about image morphing. Rather than manipulating frequencies or gradients, we'll now focus on twisting geometry and time to transition from one image to another!

Part 1: Face Morphing

Step 1: Define Correspondences

Transformations need anchor points that define where from/to things will be transformed. And different transformations need different numbers of anchor points. Simple linear transformations like scaling require only 2 reference points, but a shear transformation needs at least 3 points. So, we need at least 3 points to be able to compute any kind of affine transform on a certain shape.

In terms of our goal to morph one face into another, the fact that source and target image are both faces grants us some power in their symmetry. Setting these anchor points at dominant features like the nose, eyes, and mouth will make the transformation more natural. Moreoever, we can create a triangulation on this set of points. Instead of trying to define a complex transformation for the entire image, we can just compute a simple affine transform for each triangle formed (we'll always have 3 points!) and simply apply the transformation for the pixels in that triangle.


Step 2: Warp the Mid-Way Face

Now Presenting: George Clintwood! Given the control points for both images, the "average shape" is just the midpoints between each pair of corresponding points (ex. the right corner of George's and Clint's mouth). The triangulation of this point set is the one actually used to calculate the affine warps. But warping each image into this average shape still leaves 2 separate images; the average face is just the average of these resulting pixels.


Step 3: Creating a Morph sequence

We've defined how to warp 2 images into a result image given a certain warp ratio (the averge face warps each face by 1/2). This morph function can be used to create N frames that interpolate over this warp parameter from 0 to 1, resulting in an animation that smoothly morphs one face in the other. Here N = 45 so frame t warps George's face by a factor of t/N and Clint's by a factory of 1 - t/N. Note it may take 8-10 seconds for the GIF to load.


Frame 1

Frame 9

Frame 17

Frame 25

Frame 33

Frame 41

Part 2: Population Means

Population Shape

We can also extend these warping and morphing procedures can be extended to an entire population. For this part I used a face dataset of Danish men and women which was handily provided with 58 pre-defined control points around the eyes, nose, and chin. Note that there were 40 people in the dataset, but there were 6 pictures in different poses. The pictures to the left for instance are the "neutral" poses. This will be important soon!

Given the geometry of all 40 faces, we can easily compute the average face of the entire population by again averaging the x- and y-coordinates. This gives us a sense of what an "average Danish person" might look like. Even better, we can visualize this difference – by warping an individual into this average shape to see the change.




Warping each member of the population into the average geometry and cross-dissolving the pixels from every warped portrait shows us the average face of the population. The average face appears male simply because the dataset of 40 faces contained 33 males but only 7 females.



I'm Turning Japanese Danish


My face
Danish average face

Me, but more Danish
The Danish, but more Me

Caricatures

Up until this point we've warped images by interpolating between control points. But what happens if we instead extrapolate beyond the given range of values? Say we take the difference between the facial geometry of me and the Danish average, and add that back to my own geometry times a parameter alpha (α). Warping my face to this extrapolated geometry will accentuate the features of my face that differ from those of the Danish - essentially creating a "caricature" of my face. The higher the value of α, the more extreme the differences.


α = 0

α = 0.25

α = 0.5

α = 1

α = 1.5

Now consider the same alpha values but negative – this narrows the gap between my face and the Danish face. This is similar to the last part, but with an added degree of freedom.


α = 0

α = -0.25

α = -0.5

α = -1

α = -1.5

☆ Bells and Whistles

Changing Genders

After finding the avergae face of an entire population, let's look at the interesting capabilities of sub-population means. Remember that the average face of all the Danish samples had an eerily neutral and symmetric expression. The full population average allowed me to make my face look more or less "Danish"; the same can be done with more carefully selected groups. If I wanted to make myself "more female", for instance, I can just collect all the female Danes, compute their average geometry, and cross-dissolve my picture with that sub-population average.


My face, again
Female sub-population average face

Blending just shape
Blending just color
Blending with both

Changing... Emotions?!

It's exactly as cool and creepy as it sounds. I actually like this example a lot for several reasons. Fisrtly, the end result looks pretty despite how much the warp screwed up the guy's nose, which goes to show that even though the affine transformations in the warp are doing most of the computational heavy-lifting, the simple cross-dissolve step is just as important in the end. Also, the emotion transfer worked well even though the average face is showing teeth but the example face has his mouth shut.


Danish man (21-1m, to be exact)
Average of Danish 'happy' poses

Blending just shape
Blending just color
Blending with both

Conclusion

The 2nd half of this project was a great hands-on experience in the power of data driven methods. There were so many relations just to explore between members of the population, i.e. warping the Danes into the average population shape. But the sub-population warping was really fascinating to me, and I'd like to use the code I wrote to examine other sub-groups in the future, and maybe even try to extend these methods to other data formats.