Face Morphing

CS194-26 Proj 4 (Project Web Page) : smooth face morphing, population averaging, and caricature creation

Project Overview

In this project, I morphed one face to another, used a population dataset to get the average image that represents the dataset, used the average to develop a facial caricature with exaggerated facial features, and used a subpopulation of smiling people to convert my neutral face to a smile (and vice versa)!

What is an image warp? Basically, if we want to morph one image into another, we need to map and transform the featurews from the original image to the position in the destination image in a continuous fashion, and then dissolve the colors from each color continuously as well.

Morphing

For the next three parts I morphed a picture of me (on the left) to the CEO of Microsoft, Satya Nadella (on the right). I chose to morph myself into him because I worked at Microsoft recently while he's in charge, share ethnic and facial similarities, and believe he's an all around awesome guy!

Me and Satya Nadella

Me
Satya

Part 1: Defining Correspondences

This part of the project required me to select points along the key features like my nose, jaw line etc. in my image, and then get the exact same corresponding points in Satya's image. This was a very manual process, but I wrote code to store and backup these points as I select them. Here's how they turned out.

Me
Satya

Notice how the ordering of the points in each of the photos matches in each of the pictures. Here is the Delaunay triangulation of each of the images. The Delaunay triangulation is the set of triangles connecting a set of points with the minimum perimeter. Here's what they look like on both my and Satya's face

Me with Traingulation
Satya with Triangulation
Mean Triangulation with My Face as Reference

Part 2: Computing the Midway Face

The midway face requires finding average shape (i.e. the average of all the points selected on both images), transforming each triangle on the source and destination onto the corresponding triangle in the halfway image, and then averaging the colors from each triangle.

I usesd the `draw_polygon` function and defined my own affine transformation that maps one triangle to another to get a pixel mapping for each pixel in the halfway image

I put the original images from the top as reference. You can see how well they're blended

Me
Satya
Me and Satya Morphed: Who's who?

This turned out better than I expected and I really cannot identify who was the 'source' image. It looks like a prietty good blend of me and CEO of Microsoft!

Part 3: Morph Sequence

The goal of this question is effectively to define the morph function, so that we are able to generate an image that is some fraction of the source image, and also a fraction of the destination image.

Here's a few example of the individual frames that we outputed before I show you a cool GIF

t ~ 1/4
t ~ 1/2
t ~ 3/4

And now here's the GIF for submission

The Transformation from Me to Satya

And a cooler back and forth loop

The Transformation from Me to Satya and back and forth

Part 4: The "Mean Face" of a Population

The main goal of this part is to find the average image of many images. The theory behind this section is the same as that in part 2 where we averaged two images, except instead of taking the average of two, we're taking the average of many images, when we get the new points.

I used the Danish face dataset that had several danish images. They were mostly of white men but there were some women. The Danish dataset came with a list of all the key points in a text file. I had to parse it. Here's an example of what it looks like below:

Danish example

The blue points are the marked key points that wer provided.

Part 4.1: Compute the average face shape of the danes

First I had to take the average of all these special marked points. Here's what the average points of all the images look like relative to one sample image

Average special marked points for all the Danes

Part 4.2 is after 4.3 where I show the morph from some of the Danish people into the average

Part 4.3: Average Face of the Population

Here's what the average Danish person looks like

Average Danish person

Part 4.2: Morph Each of the Faces In the Dataset to the Average

Example 1
Example 2
Example 3

Part 4.4: Morph of my face into the average geometry and back

Doing this took quite a bit of work from remapping the points on my phase and running the morphing function again

Dealunay of the average dane
Dealunay of the mean of the average dane and me (my image)
Dealunay Triangulation of me
How my face morphs into a danish guy

Here is what I would have looked like if I were half danish!

Half indian half danish

Part 5

Based on the mean of the Danish, we can use the mean structure to emphasize certain features in my own orginal image. Basically, I take my current special points, and then add the scaled difference from me and the average Dane to my image.

Here's what the caricature looks like:

aricature with Accentuated Danish Features

While I was debugging this code, I actually generated some pretty cool caricature like images. Check them out!

[Bells and Whistles]: Part 6: Adding a smile

For the final part of this project, I decided to try to take my neutral face and add a good smile to it.

I can fine tune the alpha parameter as I show below. With a high alpha value, there is a higher effect of the smile on the original image. With a lower or negative alpha value, it takes away the smile...and you can kind of see a frown!

Alpha = 0 (no added smile effect)
Alpha = 1
Alpha = 2
Alpha = 4 (high smile effect)
Alpha = -1
Alpha = -2
Alpha = -4 (frown)