Face Morphing

Completed by: Lisa Jian

Defining Correspondences

Defining corresponding pixel (x, y) coordinates between two images is essential if we want to morph one image to the other, or if we want to take the average of a face. In this discussion, we will use corresponding points and feature points interchangeably. To get corresponding points between two images, we simply used matplotlib.pyplot.ginput to click and get the points. An example of corresponding points for two headshots would be the center of the chin for image 1 and the center of the chin for image 2.

Computing the "Mid-way" Face

Suppose that we two images, im1 and im2, and the feature points for both, im1_pts and im2_pts. To compute the mid-way face, we first take the average of the feature points. From these averaged points, we can get a Delaunay triangulation, which we will use on all our point sets (i.e. im1_pts, im2_pts, and the set of average points). Next, we have to perform an inverse warp on im1 and im2 to get them to "average features". Once we have the two warped images, we can simply cross-dissolve the two images to get the resulting mid-way face. Cross-dissolving simply means taking some weighted average of two images. Here, since we are computing a mid-way face, we would use a strict average (as opposed to a weighted average) of the two warped images.

We will discuss inverse warping a bit more in depth. We start with some source image with the corresponding source points, and we want to warp the image to fit some destination image with some destination feature points. To do so, we will also need some triangulation that defines the corresponding triangles in the source and destination images. We have all the pieces except the destination image, which we are solving for. The idea is such:

  1. For every pair of corresponding triangles, compute the affine transformation that would let you transform a point in the destination triangle coordinate space to the source triangle coordinate space.
  2. For every pixel in the destination image, mark the triangle and its associated affine transformation matrix it belongs to.
  3. For every pixel in the destination image, determine which coordinate it falls in the corresponding source triangle coordinate space. Use this coordinate to take a bilinear interpolation of the neighboring pixels RGB values.

Below are some sample midway images created with this algorithm.

Image of me! (Lisa) Image of Tina Fey Mid-way face between me and Tina Fey

Here are midway images for characters from Bob's Burgers.

Image of Tina Image of Linda Mid-way face between Tina and Linda

The Morph Sequence

This extends the idea we had from the previous section. The only difference now is that instead of taking an average between im1_pts and im2_pts, we take a weighted average. So at some time t (some fraction between 0 and 1), our morph sequence from image 1 to image 2 would be (1 - t) * im1 + t * im2. This weighted average is used not only in computing the weighted "mid-way" points, but also in performing the cross dissolving part. We do this repeatedly with evenly spaced t values to get some number of frames, which we can combine to the morph sequence.

Below is the morph sequence from my face to Tina Fey's.

Infinite morph sequence from my face to Tina Fey's

The "Mean Face" of a Population

For this part, we looked at an annotated dataset of faces (we used the Danes dataset). The annotations include correspondence points for the jawline, eyes, eyebrows, and nose. From these correspondence points, we can get the average of them - this is the average face geometry of the population. With this average geometry, we can use the warping process we described above to warp each of the faces to the average geometry. Then we can take the average of these warped images to get the average facial tones of the population.

In this case, we worked with a subpopulation of white males. The mean face (average geometry and average facial tones) below was created with 33 images.

Subpopulation average face

Some faces from this subpopulation was also morphed to this average geometry. These results are included below.

(1) Original face image from dataset (1) Face image warped to average geometry
(2) Original face image from dataset (2) Face image warped to average geometry

In addition to working with the examples from the dataset, we have morphings with my face. Below are images of my face morphed to the average geometry, and the average face morphed to my geometry.

Original image of me My face morphed to selected subpopulation geometry Average subpopulation face morphed to my geometry

Caricatures: Extrapolating from the Mean

We can create caricatures by extrapolating from the population mean from the previous part. Denote the subpopulation mean as avg_x and the face we want to turn into a caricature as x. Denote the feature points of each with f(avg_x) and f(x), similarly denote the tones/colors of each with c(*). If we want to accentuate the features of the subpopulation, we would warp both x and avg_x to have the feature points k * (f(avg_x) - f(x)) + f(x) for some k greater than 1. Then using these warped images, denote them with x_w and avg_x_w, we can accentuate the tones in subpopulation average warp in our caricature. The tones in the caricature would be p * (c(avg_x_w) - c(x_w)) + c(x_w) for some p greater than 1. The caricature's geometry would be k * (f(avg_x) - f(x)) + f(x).

This was done to my face using the subpopulation average above. Below are the results from just modifying the geometry, and from modifying the geometry and the tones.

Original image of me Caricaturized version of me (geometry only) Caricaturized version of me (geometry and tones)

For fun, I also morphed the average face's geometry to mine and added its tone to my face. The result is as follows.

Subpopulation's tone added to mine

Bells and Whistles: Changing ethnicity

This is similar to the last section. Instead of using the previous subpopulation mean, we used this average of 64 female faces (source).

Average female face

From this, I can try to modify my appeared ethnicity. Included below are the morphings with geometry only, tones/appearance only, and both. Note that the geometry morphings appear a bit strange because the feature points did not exactly correspond along the jawline. Additionally, since no feature points were included along the hairline, the top of the head looks dented. This effect also appeared in earlier sections (i.e. "The "Mean Face" of a Population" and "Caricatures: Extrapolating from the Mean")

Original image Geometry morphing only Tonal/appearance morphing only Geometry and tonal morphing

Citations

Styling of this page is modified from bettermotherfuckingwebsite.com.