Homework 4: Face Morphing

Jacob Green | cs194-26-afl

Part 1: Defining Correspondences

For my correspondences, I defined 9 points for each eye (one in the center, 8 surrounding), 8 points surrounding the mouth, 8 points surrounding the nose, 9 points outlining the cheeks and chin, and 5 points outlining the forehead, for a total of 48. I manually annotated some images to blend between, and then saved the correspondences to csv files so that I would not have to annotate each time I ran the program. Below are the two annotated images.

Jacob Face
Hein Face

Part 2: Mid-Way Face

To create the mid-way face, I defined two helper functions. The first function takes in two sets of correspondences and computes the weighted average of each pair of correspondence points, based on an input weight. Next, I defined a function that takes in two triangles (sets of three points), and computes the affine transformation matrix to convert the second triangle into the first. To do this, I set up the system of equations for each point, and encoded them into a matrix and vector (similar to homework three) and solved them using a linear solver. Finally, to compute the complete warp, I take in two images and two sets of correspondence points. I compute the average correspondences, with a weighting of 0.5, and then calculate a triangle mesh using scipy's Delaunay function. Then, for each triangle, I calculate the affine transformation from the output image space into both of the image input spaces, and save each of these transformations in a dictionary to look up later. Then, for each pixel in the output image, I first lookup which triangle in the Delaunay structure contains that point. Then, for both input images, I retrieve the transformation according to the found triangle, calculate the point in the input image that corresponds to the pixel, and append it to a list of points to query. Finally, I query these points all at once by using scipy's RectBivariateSpline to define a function over the input images and then query all of the points at once to map them into the output image space. Finally, I add the two warped images together according to a cross dissolve weight. For the midway image, I also set this value to 0.5. When doing this, I created the image below:

Jacob Image
Midway Image
Hein Image

Part 3: Morph Sequence

Once I was able to calculate the midway face, extrapolating this to create a morph sequence was mostly trivial. I simply applied my warp function at different dissolve and warp weightings spanning between zero and one (after calculating the triangle structure and keeping it constant across all calls). Below was the result.

Part 4: Mean Face

For my mean face calulations, I used the Danes dataset, taking only the forward facing and colored images (which came to 37 faces at the end). To calculate the mean face, I first took the average of all the correspondences for the dataset. Next, I iterated over every image, warped each of them into the average shape bsed on the average correspondences and using the same logic as from the warp function, and then averaged the warped images to get the mean face.

Below is the average face over the Danish image set (pictured middle) as well as 8 faces from the dataset morphed into the shape of the average face (surrounding)

Below is my face warped into the shape of the average face (left) and the average face warped into the shape of my face (right).

I also calculated a few blend images at different weights spanning from zero to one.

Average Face Weight = 1
Average Face Weight = 0.8
Average Face Weight = 0.6
Average Face Weight = 0.4
Average Face Weight = 0.2
Average Face Weight = 0

Part 5: Caricatures

To create a caricature, I wanted to emphasize the features that stand out from an average face. To do this, I simply used the warp function but instead of chooseing a weighting between 0 and 1, I used a negative weighting for the mean face, which accentuates the individual characteristics. I tried this for just the warping and for warping plus color. The negative color turned out to be a very bad choice...

Negative warp weightings, color not changed

Average Face Weight = 0
Average Face Weight = -0.2
Average Face Weight = -0.4
Average Face Weight = -0.6
Average Face Weight = -0.8
Average Face Weight = -1

Negative warp weightings with color changed equivalently (yikes, this was a bad idea)

Average Face Weight = 0
Average Face Weight = -0.2
Average Face Weight = -0.4
Average Face Weight = -0.6
Average Face Weight = -0.8
Average Face Weight = -1

Part 6: Bells and Whistles

For Bells and Whistles, I decided to try blending my face into several different races. Therefore, I downloaded a set of mean faces for many different populations, cropped them and annotated them, and performed a set of warps between each of them and my face. The results are below.

India:
Israel:
Samoa:
West Africa: