Face Morphing

Avni Prasad, CS194-26-aej

In this project, I worked on image warping to morph one or more images into another. Here, I implement face morphing between faces, computing the mean of population of faces, and extrapolating caricatures of images. To begin, let's look at the case of trying to morph these two images together.

George Clooney

Barack Obama

Morphing - Problem Statement and Intuition

Input: George image and Obama image

Output: a fluid morph from George's image into Obama's image

Morphing comprises of the following steps:

Defining Correspondences: in this step, I want to mark key features of the face (e.g. eyes, mouth, nose, etc.) to later use as a mapping from one image to the next (e.g. I want to warph George's nose into Obama's nose).
For each frame at time t in the range [0,1]
- Step 1: Calculate a weighted average shape based on time t (closer to time 0, we want to warp images to a face shape closer to George's face shape then as time moves towards time 1, we want to warp images to a face shape closer to Obama's face shape)
- Step 2: Once both images have the same face shape, then we can cross dissolve the color by interpolating the two images

I defined 56 pairs of correspondences in both images, including 4 points in the corners and edges of the images which will help give a smoother transition. I did this through ginput which allows you to manually select the points. After getting these points, I then defined a triangular mesh over the points in each image using Delaunay triangulation. Triangular mesh gives us triangle-to-triangle correspondences between the two images, including all pixels contained in the triangles.

Suppose I am interested in getting the mid-way face of these two men. I will want to first take the mean of their correspondences points to get a feel for what the mid-way shape of these two men are

As you can see the average triangular mesh over the two original images do not match up well. Out next task will be morphing the original images to have a shape that matches well with the average triangular mesh AKA morphing to the average face shape.

Before moving on to the whole morphing sequence, I computed the mid-way face of the two images

Step 1: Morphing images to Mid-way shape

Taking the correspondences recieved from the previous step, I now want to map corresponding image triangles to its corresponding average triangles (eg. Obama's nose above is wider than George's nose, so we want to reshape Obama's nose to be the size of the average nose size of the two men). We can do this by looking at the transformation across corresponding triangles in the images. Specifically, I use the inverse of the affine transformation matrix T that goes from source image to the midway image in order to find the corresponding points in the source image and perform bilinear interpolation on the coordinates to get the closest pixel value. This approach, also known as Inverse Warping, eliminates cases where holes (black squares in the photo below) may show up in the warped image since pixel values at locations that no coincide with pixel coordinates can be obtained from the source image using bilinear interpolation.

My results of this process

George Clooney

Original Face Shape → Mid-way Face Shape

Barack Obama

Original Face Shape → Mid-way Face Shape

Step 2: Cross-dissolving images with same face shape

Now that we have an Obama and George image with the same face shape, we can cross-dissolve the images to average the colors. In other words, here we are simply outputting the mean of the images from the previous step

Finally, we are ready to create a fluid morph from George's image into Obama's image. For the mid-way face, we computed the average of George and Obama's face, but for creating a fluid morph we want to create a range of images that initially look like George and then start to look more and more like Obama. To do this, we use a weighted average to dictate how similar the output image should look like George versus Obama.

At each timestep t in the range [0,1]:

I warped the images into a weighted average shape: (1- t) * (George's face shape) + t * (Obama's face shape)
Then, I cross-dissolved the images based on the weighted average image: (1- t) * (George's warped face) + t * (Obama's warped face)
At time t = 0, we'll get George's image and at time t = 1, we'll get Obama's image

Here, I'm computing the average face of the Danes dataset. I specifically looked at the images of females smiling (suffix: 2f).

Step 1: Morph images to average shape

First, I took the average of ALL the correspondences to get the average shape of this dataset. Then I morphed the faces of each image into that average shape

Original image → Average face shaped image

Step 2: Cross-dissolve all images to get the "Average Image"

To do this, we simply average all the warped images resulting in the average image of smiling females from this dataset:

Notice that this image is quite symmetric and unblemished. We can understand this as a result of asymmetries of the woman in the dataset and their blemishes more or less average out to create an image like this.

Step 3: Compare Average Image to portrait

I calculated the mid-way face between a portrait of me and the average face calculated above. Here were the results:

My portrait

Mid-way of average face and my portrait

Average face

Then, I warp my face into the average geometry from the image above and warp the average face into your geometry. The results of this:

My face warped in average geometry

Average face warped into my geometry

Finally, for this last part, I used the average fact I got from the Danes dataset to generate a caricature of myself. To exaggerate my features, I essentially upped my warp factor to be over 1 (or 100%) and to understate my features, I will decrease my warp face to be below 0 (or 0%). Here are examples of different alpha's I tried:

-40% Me

-20% Me

100% Me

150% Me

200% Me

I tried this same process with Obama and George's photo to get caricatures of Obama relative to George's image. Of course, we can't get enough of Obama... 200% is just not enough

-40% Obama