CS194-26 Project 4: Face Morphing

YiDing Jiang

Overview

In this project, I implemented a morphing algorithm that warps individual triangular meshes through affine transformation. With this technique, I produced several morphing sequence from faces to other faces. I also used the technique to calculate a "mean face" of 30+ people and use it to create a caricture of myself.

Part 1: Face Morphing

The first step of face morphing involves label two photos with important feature points and compute the average shape of the two sets of feature point to produce a "mean face":

Professor EfrosMidway FaceJunyan

This midway face is made by computing Delaunay triangulation on the averaged feature points:

Then I used inverse warping to warp both images into the midway shape and then average the two result to produce the midway face.

Now this technique can be used with with linear interpolation with weighted average to produce a cross dissolve over time from one face to the other one (here every gif contains 45 frames):

Now, a couple more examples:

My face to Danish averageMeme lord

Part 2: The Mean Face of a Population

For this part, I first computed the mean face of all the neural and color male faces (30 of them) in the Danish dataset by first computing their avergae geometry and then inverse warp all of them to that geometry so they can be averaged.

We can look at some of the individual data points in the dataset morphed into the mean face (the top row is original face and the bottom is the face morphed into the average geometry of the population):

Male 5Male 10Male 13Male 27

And then I warped my face into the average Danish face and vice versa:

My face to avg. Danish faceAvg. Danish face to my face

I further calculated the average face of the same male face but smiling. Using the smiling average I can compute the difference vector between the my face and the mean face to be the deviation of my face from the mean. Adding a scalar multiple this vector to my original face, I can create a "caricture" of myself.

Original FaceCaricture at alpha=0.5
Caricture at alpha=-1.5Caricture at alpha=2.0

Bells and Whistles

In this part, I calculated the mean smiling male face from the Danish dataset and compute the difference vector between the avg. smiling Danish face and avg. normal Danish face. This vector information can be used to add to my face to make myself "smile".

Mean normal faceMean smiling face

While the difference shape vector can be directly added to the geometry of my face, the color difference vector cannot because the geometry is not aligned. To dddress this, I first transformed my face to the mean geometry of the the population, add the color difference, and then transform the result back to geometry of my face. For transformation with both color and geometry, I transform the color difference vector into the geometry of my face and then transform the result to geometry of my face plus the smile difference vector:

Original FaceShape Only, alpha=0.9
Color Only, alpha=0.9Shape and color, alpha=0.9

As you can see, these transformations aren't exactly the best looking or natural. One reason for this phenomenon might be because the mismatch of the underlying distribution because the means are computed from a small subset of Danish and I am … clearly not Danish.

However, I did notice that shape works somewhat okay and I did some more exploration and found this amusing effect:

My face "unsmiled"Danish face "unsmiled"

This is done through substracting the smiling difference vector from the geometry, which essentailly "unsmiles" the face to produce a "sad/wronged" face, which surprisingly works better than trying to make a face smile :)

Finally, here is some stacked non-linear but local linear transformation of the gods of Deep Learning, downsampled becaused the space constraints…:(