Project 4: Face Morphing and Modelling a Photo Collection

Jose Chavez

cs194-26-adu

A face morph is where we warp the facial structure in one image smoothly into the facial structure in another image. The trick is figuring out how to warp a static image. In this project we examine the steps necessary to smoothly create a face morph. In addition, I will look at morphing faces based on the averages of populations.

Defining Correspondences

The first step towards morphing one face into another is defining correspondence points. These are points over the face that capture the overall facial structure. Using Python ginput, we select the points in a certain order for image A, and then we select the corresponding points along the face for image B in the same order. This way, we can produce a list of pair coordinates for both A and B. For this project, I would like my image A to be a portrait of George Clooney and my image B to be of me, matching the aspect ratio.

However, we can't morph using only points. We can shift point positions all we want but we would have no way of changing the pixel values to reflect the warp. We need a triangulation, specifically a Delaunay triangulation. A Delaunay triangulation essentially produces a triangle mesh given certain data points, which can be found efficiently and can have "good" triangulation, in which smallest angles are maximized. Below are the Delaunay triangulation for both George and Jose, found using scipy.spatial.Delaunay.


Correspondence points for George
Correspondence points for Jose
Delaunay Triangulation for George
Delaunay Triangulation for Jose

Computing the Mid-way face

Computing the mid-way face of our two images is a good place to start. This involves computing the average shape, warping both faces into that shape, and averaging the colors together. Warping to the average shape is essentially applying transforms on each triangle in image A and image B to produce the triangles in the average shape. These transformations are affine transformations which can account for rotation and transformation.

To code an affine transformation, I essentially coded the math in this link.

But what about the pixel values after the warping? Can we just apply the transform on a pixel inside an image A triangle to an averaged triangle? This isn't a bad approach but what if we apply a transform on a pixel and the end position is somewhere in between two pixels? How do we get the pixel value then? To work around this, we take the inverse transformation, where we map a pixel in an average triangle to a corresponding location in an image A triangle. If that position happens to be in between two pixels, we can just interpolate the pixel values near it. To do this, I used the scipy.interpolate.interp2d function. Below is my mid-way shape and face with the pixel values from image A and B averaged.

Triangulation for mid-face
Mid-face

The Morph Sequence

Now that we know how to warp two images to a certain shape, the intermediate shapes can be computed to compile a smooth morph. Instead of taking just the average shape, we use a value within [0, 1] to produce an intermediate weighted average shape. This is essentially linear interpolation:

(1 - t) * p + t * p'

Here, t is a value in between 0 and 1, p are the points for image A, and p' are the points for image B. As t steps from 0 to 1, we morph to that shape and essentially use the same formula to interpolate the color values, a cross-dissolve!

Below is the result of taking 45 steps in between 0 and 1 to produce the morph sequence from George to Jose.

The Mean Face of a Population

As mentioned earlier, there are other applications to morphing. One application is finding the average face of a population. For this project, I took the grayscale dataset from The FEI Face Database. These images are fairly small and are only focused on the faces. Each face is annotated with correspondence points. To calculate the mean face I compute the average face shape of the whole population, morph each of the faces in the dataset into the average shape, and then compute the average face of the population. Below is an average shape of all the facial correspondence points in the FEI dataset.

Triangulation of average points across data base

Below is the mean face of the total population, including a few examples of how some faces appear after being morphed to the mean face shape.

Full mean of the population

Original image
Face morphed to mean
Mean face

Original image
Face morphed to mean
Mean face

Original image
Face morphed to mean
Mean face

Below is my face morphed to the average shape and the average shape morphed to my shape.

Mean face
Jose morphed to average
Mean face morphed to Jose

Caricatures

Using the mean of an entire population, I can produce a caricature by extrapolating from that average. Basically, I adjust my features towards the direction of the average, but sometimes way beyond it. For example, to extrapolate my face shape towards the average male shape, I take the difference between points on my face and the points of the average shape. I then take that difference and multiply it by a scaler, alpha, and finally add that result to my facial points. Depending on the alpha, this can produce some interesting results. My example below is using the mean male face shape. I used an alpha of -0.3 to extrapolate away from the mean. Below is the result.

Mean face of male population
Caricature with alpha = -0.3

Bells and Whistles: Changing Gender

Using extrapolation, we can also adjust the shape and color towards a population mean. For example, if we wanted to make a male face look more female, we can morph its shape to the average female shape and then extrapolate its color. Below are the results with morphing the shape, extrapolating the color, and then both combined. The result looks like a convinving female version of the original image!

Mean face of female population

Man morphed to female average shape
Man color directed towards female average color
Man color and shape towards female average face