CS194-26 FA21 Project 3: Face Morphing

By Austin Patel

Overview

The goal of this project is perform face morphing, which is a smooth transition from one face to another by transforming both shape and appearance. The goal is also to investigate what the average face of a dataset of images looks like.

Results

I start with the two images we are going to morph together.

Next I annotate corresponding keypoints on each image, trying my best to key the keypoints locating the same features on both images. I setup my code to automatically add keypoints to each of the four corners, which is needed so that every part of the image (including the background) is part of a triangle once we do triangulation. The average keypoints (found by average the corresponding keypoints from each image) are also presented.

Using the average keypoints, I use a Delaunay triangulation algorithm to create a triangulation. This triangulation indicates which keypoints are part of what triangles in the mesh. Next, I apply this same triangulation to the keypoints from the Austin and George images.

Next, for each triangle in each source image, I compute an affine transform between the triangle in the average keypoint image to a triangle in the source image (inverse warping). This affine transformation is computed by creating a system of six linear equations formed the three corresponding sets of points from the source and destination triangles. I go from average triangle to input image triangle because the affine transformation does not perfectly map pixels to integer locations and therefore we can do bilinear interpolation on the color values of the input image. Using these affine transformation and sampling techniques I am able to perform the image warp from each input image to the triangles formed from the average keypoints.

With both input images mapped to the average shape, the appearance forms a valid subspace and thus we can average both average shape input images to get a fully morphed image. Looks pretty good except for some parts of the hair and the right shoulder.

Now that we have the tools to morph to a given set of keypoints, we can move linearly from the Austin keypoints to the George keypoints and compute a morph along all the midway keypoints. The blending also is happening as a weighted linear combination of the shape adjusted input images. NOTE: These YouTube videos might require you to be signed into a Berkeley email address to view!

The next task is to use images from a dataset that already have keypoints labeled and try to compute the average face. The data set I used is the IMM Face Dataset (https://web.archive.org/web/20200207215115/http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=3160). I specifically selected out the images of males looking forward with a neutral expression to use for my analysis (30 images, but 3 were black and white and thus I removed them leaving 27 images used here). Here are some examples of the data from the dataset with keypoints annotated don top of the images.

I average all the keypoints across the subset of the dataset I used to produce the average keypoint locations.

The average keypoint locations can then be used to create a triangulation for the average face shape.

I then apply the same image morphing procedure as was done for the Austin and George images to map each input image of the dataset into the average shape. Here are 3 examples.

Now that we have normalized for face shape, we are able to take the average of the color values for the shape adjusted images to produce what is called the average face of a population.

Next I map my own face to the average face shape of this dataset. I do this by adding keypoints to my face in the same way that keypoints were added to the images in the dataset. I then morph to the average shape. My eye looks a litle strange due to the labeling by hand that I did not being perfect.

Now I try the reverse: morph the average face of the dataset onto the shape of my face. Here is the triangulation generated from the keypoints of my face.

Then we morph the average face onto my geometry. The face does not look as good as before since we lose symmetry and evenness from average a lot of face shapes. Again we see the slight errors in my annotation causing distortion in the eye.

The next step is producing a caricature of my face. This can be done by first subtracting the average keypoints of the dataset from the keypoint positions of my face. Thus my keypoints are represented as the average keypoints + 1*direction_vector. To produce a caricature we can change the multiplier to be more than 1. For the video below I transform from 0x to 2x. For each shape I compute, I recompute a new triangulation given those keypoints. It would have been interesting to try not recomputing a new triangulation each time. I then map my face onto the new exaggerated shape structure.

Here are the keypoints from the caricature changing from 0x to 2x. Notice the triangles jump around occasionally as the triangulation is recomputed for each new shape.

Bells and whistles: I applied principal component analysis to the dataset of faces by first flattening each WxH matrix into a single column WxH length vector and then stacking the vectors side by side as columns of a matrix. I use a library to compute the PCA. The top 9 eigenvectors (corresponding to largest 9 singular values) are displayed below. Image 3 shows lighting on the face. Image 5 clearly shows the vector encoding eye brightness. Image 9 we see beard presence encoded.

Top 9 singular values plotted. The sharp dropoff after the first two singular values indicates that the top two eigenvectors are already able to encode a lot of the variance in the space of faces from this dataset. I did not try generating new faces from these eigenvectors, but that would have been cool

Bell and Whistle: The next thing I wanted to try is to see if I could change my appearance from male to female. Here is another picture of me as well as an average image generated by taking Miss California from 1997-2020 and computed a face average using the similar approaches I used above. The average face image is used from https://griffonagedotcom.wordpress.com/2019/12/19/miss-america-face-averages-of-candidates-by-state-and-by-year-1997-2020/.

First I need to do the same keypoint labeling procedure

I then do the same approach of averaging keypoints and computing triangulation

The first thing to visualize is shape transform from my shape to Miss California shape.

Here is what it looks like if we just do appearance blending

And last, but not least, if we do shape and appearance blending we get some very convincing results. Note that there is a tranform from both male to female and from neutral expression to smiling expression! The teeth do appear strangely since teeth appearance do not form a subspace (there is no in between of having vs not having teeth). The shirt transform looks slightly strange as the average image did not have a super clearly defined shirt location.