CS194 Project 4: Face Morphing
Matthew Waliman, cs194-26-afq

In this project we produce various morphs (like the one seen above) and calculate mean facial images of a population. I revisited the Rihanna/Girl with Pearl Earring muliresolution blend example in proj 3 here with great results!

A morph is a simultaneous geometric warp and color value warp. The color value warp is really just a cross dissolve between two images. Getting the points to match up to the correct locatins is slightly more difficult.

Defining Correspondences

First, we need to define pairs of points that correspond to similar locations between the two images. Then, we compute the average geometry of these correspondences by simply averaging each pair of points: mean = p1*warp_frac + p2*warp_frac. Now, we have a handful of points that match up in each image, but the correspondance of the rest of the image is unknown. We need to break up the image into corresponding image patches. We do this, by computing the delaunay triangulation of the mean points. A key step here is to compute the delaunay of the midway face, and not that of either "source" set of points in order to minimize distortions in the geometric morph. Now that's we've computed the delaunay triangulation, we have map to go from each triangle in the mean triangulation to each source image.

Note the subtle differences in triangulation in the cheek area!

Delaunay Triangulation computed on source points, applied to rihanna.jpg points
Delaunay Triangulation computed on mean points, applied to rihanna.jpg points
Delaunay Triangulation computed on mean points, applied to pearl.jpg points

Producing the "Mid-way Face"

Now to do the actual heavy lifting of transforming from a triangle patch in the source triangulation to the corresponding mean triangle, we use an affine transform, definined below.

We compute the affine transformation from each triangle in the source triangulation, then apply it to get the corresponding coordinate in the mean triangulation image space. However, this comes with a caviat. The point of a morph is just that! To morph! So while we have a discrete coordinate grid in "source triangulation space" the computed affine transformation to "mean triangulation" space may not be discrete, thus falling between pixel values. This is fine if we are working purely in continuous geometric space, but the final source image morphed to the mean geomtery needs to be displayed as set of pixel values. To remedy this we apply a color interpolation so that each warped pixel draws its color linearly from its neighbors. Then we simply morph away, transforming each source triangle safely transformed to its destination mean geometry. To create the "midway" face I just morphed each face with a warp_frac=0.5 and cross dissolved (averaged) the intensities.

The Morph Sequence

In order to produce a morph sequence I simply varied the warp_frac minutely from frame to frame, slowly nudging the transformation.

To calculate the average face of a population, here a danish dataset, I calculated the average face geometry by averagin all the points and triangulating the resultant mean points. I then used this triangulation to warp each of the faces into the average face shape, then averaged the intensities. The average face and triangulation calculated is shown below

Original souce images

Source images warped to mean

My face morphed to average Dane
Dane morped to my geometry

One application of these collected points and warping skills is the ability to create caricatures! In essence a caricature is an exaggerated verision of a person, so we try to mimic this by pushing points towards or away from a target triangulation. We do this by subtracting the difference between the target points, here I used the average asian male, and the source image/pts, I used myself. We then scale it by a small factor add it to the original source points, and use these new set of points "nudged in the direction" of the target source to create caricatures that are pushed in the direction of the target image. We can also define a directionality and compute "forward" and "backward" caricatures by applying a positive(adding the difference) or negative(subtracting the difference) to create morphs that or more or less like the target image.

Backwards Caricature
Forwards Caricature

The danish dataset also included images and points of the same people smiling, so I went ahead and computed the average smiling face, seen here on the left. The right image is the result of "forcing" or morphing the average neutral face to the smiling face geometry.

Just Color Morph

Color and Geometric Morph

Actual Smiles

I had a lot of fun morphing some characters from Sherlock and Doctor Who into their respective animals.

I had so much fun on this project. I'd like to build something related to this in the final project, perhaps trying out a different morphing algorithm. I also learned how labor intensive it is to map points on an image, especially when you're not just morphing something with distinguishable landmark features. You have to really think about what the triangulation is going to do and what features are important to conserve. In this vein I think it might be interesting to implmente automatic morphing.