Face Morphing

Hanlin Chen

Defining Correspondences

To perform a face morph, the first step is to define corresponding points on the images and produce a triangulation. We use cpselect to pick the points by hand and generate the list of points. Then, we calculate the Delaunay triangulation based on the average between the two sets of points to avoid potential triangle deformations. Here is the triangulation applied to points on both images:

me_tri
triangulation on my face
george_tri
triangulation on George's face

Computing the "Mid-way Face"

Now that we have the corresponding points and triangulation on the two faces, we need to warp the two images into the average shape, then average over the colors to compute the mid-way face. To help with computing the warp, we define computeAffine(tri1_pts,tri2_pts), which will compute the affine transformation from tri1 to tri2. We will then, for a given image, use the affine transformation to compute the inverse warp by repeating the following on every triangle:
Compute the affine transform from image triangle to the average triangle and take the inverse.
Find the pixel coordinates that correspond to the average triangle coordinates using skimage.draw.polygon.
Apply the inverse affine transform to the average pixel coordinates to get the image pixel coordinates.
Finally, we set the values at the average pixel coordinates to equal the values at the corresponding image pixel coordinates.
That gives us an image warped to the shape of the average. So, we do this for both images so they fit the average shape, then simply average over the pixels to obtain the mid-way face.

me
image of me
midway_face
mid-way face
george
image of George Clooney

The Morph Sequence

Now, we want to create a generalized version of the mid-way face, but now with the ability to control how much we warp the shapes and how much we dissolve the pixels. We define morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac) to produce this result. The process is very similar to the one for the mid-way face, but with two key differences:
When calculating the points to define the shape of the morph, instead of taking the average, we weight the points such that we take warp_frac of im1_pts and (1-warp_frac) of im2_pts.
When cross dissolving the pixels, instead of taking the average of all the pixels, we again weight the pixel values such that we take dissolve_frac of im1 and (1-dissolve_frac) of im2.
Note that we can now simply use the generalized morph function with warp_frac=dissolve_frac=0.5 to compute the mid-way face.

To create the morph animation, we call morph 46 times, with warp_frac=dissolve_frac advancing in equal increments from 0 to 1 to create 46 frames (0-45) of animation. On frame 0, the image shown is 100% im2 (me), and on frame 45, the image shown is 100% im1 (George). Here is the warp animation:

morph
morph animation

"Mean Face" of a Population

We will now calculate the mean face across the FEI face database. The data set comes with 200 smiling images and 200 neutral-faced images, as well as annotated corresponding points. All of the provided points lie direclty on the face, so for each image/set of points, I added points in each of the four corners, as well as the vertical center on the left and right sides. I calculated the average faces across all images, all smiling images, and all neutral images:

avg_all
average over all
avg_smile
average over smiling
avg_neutral
average over neutral

We can also apply a warp of each image in the data set to the average shape. Because of the smiling/neutral difference, I decided to apply the average smiling shape to smiling images and average neutral shape to neutral images. Here are some of the results:

1a
original neutral image
1a_to_avg
average neutral shape applied
1b
original smiling image
1a_to_avg
average smiling shape applied
39a
original neutral image
39a_to_avg
average neutral shape applied
39b
original smiling image
39a_to_avg
average smiling shape applied

Additionally, we can warp my face into the average shape of the data set, or we can warp the average face into the shape of my face:

me_avg_shape
my face warped to average shape
avg_me_shape
average face warped to my shape

Caricatures: Extrapolating from the Mean

We can also take my face and the population mean from the last step and extrapolate to create a caricature. In a morph of the two with dissolve fraction fixed at 0, if the warp fraction at 0 gives my original face, we now set the warp fraction to some negative value; I chose -0.5. This gives a version of my face where my features that differ from the mean face are exaggerated. We can see the result clearly as my eyes and mouth get even smaller than they already are.

caricature
caricature of me

Bells and Whistles

For bells and whistles, I decided to change the gender of my face. I found images for the average Chinese man and woman (from here: https://pmsol3.wordpress.com/2009/10/10/world-of-facial-averages-east-southeast-asia-pacific-islander/) to do so.

chineseaveragewoman
average chinese woman
chineseaverageman
average chinese man

First, I took the difference between the average woman and average man to get the "vector" that would make a man look more like a woman. Then I used the "vector" and my face and defined corresponding points, then performed half-way morphs with just shape, just color, and both shape and color

me
original image
me_woman_shape
image warped with half woman shape
me_woman_color
image dissolved with half woman color
me_woman_both
image morphed with half woman image