Project 3: Face Morphing

In this project, we use our knowledge of face populations along with shapes and appearances to create a new, totally realistic face. We do this primarily by create correspondences to triangulate our pictures, and then creating affine maps between the triangles in order to smoothly morph between the two faces. Then we do some large-scale averaging with more correspondences to have fun with.

Defining Correspondences

The first part we needed to do is write code for selecting facial keypoints to use for our triangulation of the faces. I wrote up a simple script which displays the images side by side and lets you define the correspondences one pair at a time. The picture belows gives a simple example of me using it live to define correspondences between me and George. It was robust enough that I could start from either side when marking a pair of correspondences, and would save the correspondences to a numpy array on disk.

An example of my correspondence tool using ginput.

From here, it was a matter of calling scipy's delaunay function to get the triangulation of the points. Then we used the triangulation to create a map from the source face to the target face. Here is the triangulation that I generated overlaid on George's face. Notice that I added correspondence on the corners so that the morph was smooth over the entirety of the image.

A triangulation of George's facial points.

Computing the Mid-way Face

Now that we have the correspondences, we can use them to create a map from the source face to the target face. This is done by creating a map from the source face to the mid-way face, and then applying the map to the target face. The mid-way face is the average of the two faces.

It was easy to obtain the shape of the mid-way face, as it was plain averaging. Obtaining the color was more difficult, and required me to implement an affine map between two triangles. I followed the geometrical method suggested in class by mapping one triangle to the origin triangle then mapping that to the other triangle, making it very simple to compute the transformation. Here is the result of creating the mid-way face.

The progress of faces to the mid-way face..

The Morph Sequence

The final morph sequence isn't hard to create once we've done this, as we just repeat the steps but with different intermediate faces (in different ratios). Here's my output gif, created using the same values for warping and dissolving. I reversed it at the end so that it's a smooth video. The entire compilation took < 2 mins since I vectorized my code, using matrix multiplication to find the inverse maps, and vectorized the interpolation calculation with ev.

The progress of faces to the mid-way face..

The "Mean Face" of a Population

For calculating the mean face, I chose to use the Danes from the IMM Face Database of 240 face images. Parsing the annotations took some scripting work, but in the end they all looked like this.

An example annotation from the IMM Face Database.

Computing the average shape only required averaging the shape vectors then, after which I warped all the faces onto the average shape. Here are some examples of what the faces looked like.

An example of a good warp.

An example of a bad warp.

An example of another bad warp.

The last two images didn't turn out well, and this is because the dataset consists of faces that are turned, which clearly require funky warps in order to return them to front facing pictures. Here is the mean face of this population.

The mean Dane.

I then collecting feature correspondences on my own picture similar to how they were collected for the above dataset, and then warped them onto each other to see what the outcome is. They are displayed below.

My face warped to the mean face (like I just got bit by bees!).

The mean face warped to mine.

I believe some of the issue can be attributed (obviously) to poor correspondence choice, but also to the angle at which I took my picture being different, as well as starting with a portrait picture which required some manipulating to get working.

Finally, here is a caricature I made of myself by subtracting from the mean and boosting it. It accentuantes my brows and other weird features, as well as becoming brighter due to some other artifacts from how I had to handle the portrait.

A caricature of me using an extrapolation of 1.6 from the population mean.

Bells and Whistles

For my bells and whistles, I used a population average picture to change my ethnicity. I found a picture of an average Indian person, so I warped it with my results from above and got an Indian picture of me.

Indian Average Male.

Me warped to be 70% of the indian average male.