CS194 Project 3:

Face Morphing

Ahmed Malik || Spring 2020

Overview

In this project, we explore an image warping technique for smoothly morphing one image into another image. This can be achieved by simply cross-dissolving the two images, but this generally produces a poor morph sequence if the subject matter of the images are not aligned. Therefore, we will first compute the average shape of the two input images and perform a cross-dissolve for each frame of the animation.

We then extrapolate this further by applying similar methods on a larger dataset of collected images of a certain population type, in this case using the Danes population set. Here, we will compute the mean face of such a population and use it to morph images to new facial geometries, as well as produce caricatures by weighting certain features.

Defining Correspondence Points

To align the subject matter of the images we must identify points of correspondences. In this case, the subject matter of both the input and output image are faces, so we aim to match facial features from the first image to the facial features of the second image. To that end, we use plt.ginput() to identify 24 points of correspondence noted as follows:

Head/Face: 10 points
Eyes: 4 points per each eye
Nose: 3 points
Mouth: 3 points

We also add the 4 corners of the image canvas for a total of 28 points. For the morph sequence, we use portraits of George Clooney and President Obama.

We define the points of the mid-way face to be the average (arithmetic mean) points of the two sets of correspondences:

(x^{'}, y^{'}) = (\frac{x_{1} + x_{2}}{2}, \frac{y_{1} + y_{2}}{2})

We wish to warp the pixels from one image to another by using an affine transformation, but this requires at least 3 points in each basis. To obtain 3 points, we use Delaunay triangulation (scipy.spatial.Delaunay). Delaunay triangulation is particularly useful because it maximizes the minimum angle of all angles of triangles, keeping the triangles from becoming too skinny and consequently producing unwanted artifacts in the morph. The triangulation for Clooney and Obama is shown as follows:

Computing the Mid-Way Face

The mid-way face is formed by warping each input image to the average shape, determined by the arithmetic mean of the two sets of correspondence points as noted earlier. We achieve this by computing the affine transformation matrix of each triangle in the image's Delaunay triangulation. Specifically, we perform an inverse warp where we use the inverse of this affine transformation matrix to get the positions of the pixels in the orignal image. Inverse warping ensures that all pixels in the final, morphed image are filled in. The operation to compute the inverse transformation matrix is defined as:

T = (X A^{-1})^{-1}

where

X = [\begin{matrix} x_{2,1} & x_{2,2} & x_{2,3} \\ y_{2,1} & y_{2,2} & y_{2,3} \\ 1 & 1 & 1 \end{matrix}]

A = [\begin{matrix} x_{1,1} & x_{1,2} & x_{1,3} \\ y_{1,1} & y_{1,2} & y_{1,3} \\ 1 & 1 & 1 \end{matrix}]

and the coordinates $x$ and $y$ correspond to vertices of the triangles in the first image (for $A$ ) and second image (for $X$ ), and $T$ is the transformation matrix used to apply the affine warp to all pixels within the triangles. Thus, we can apply this method to the portraits of Clooney and Obama to achieve their mid-way face:

Overall, the alignment is very good, though we do see a ghosting effect at the outer edges of their heads, likely due to specifying the majority of the correspondence points around the center of the face. The actual faces match up remarkably well in terms of both shape and color.

Morph Sequence

In order to produce a smooth morph between Clooney and Obama's faces, we will create 46 frames, where each frame is shown for 1/30th of a second. To create these frames, a morph must be computed at each point of the sequence, and we control this with a warp fraction warp_frac and a cross-dissolve fraction dissolve_frac.

We define the average shape of each frame to be:

avg_shape = im1_pts + warp_frac \cdot (im2_pts - im1_pts)

As the warp fraction increases, the average shape errs closer and closer to the shape of the second image. Using the average shape, we can compute the morph of both input images to this intermediate average shape, and scale by the dissolve fraction (same as the warp fraction for our case):

morphed_img = img1_morphed + dissolve_frac \cdot (img2_morphed - img1_morphed)

Similary, as the cross-dissolve fraction increases, the morphed image errs closer and closer to the second image. The result of this is as follows, where my .gif animation goes back and forth between Obama and Clooney for a smoother transition:

The "Mean Face" of a Population

To compute the "mean" face of a population, we use the Danes population dataset. We extrapolate the average shape method from earlier to expand to averaging several faces. The average face is then the sum of all faces in the dataset, warped to the average shape, with an arithmetic mean applied. Here we present a few example faces from the Danes population set warped to the average shape, as well as the average face of the population set as a whole: dane1_to_avg

We note that the average face looks primarily male, likely due to the fact that there were far more images of men in the population than women. Also, the average face is quite rectangular/narrow, and we see this affect the resultant morphs which are slightly squished in width and also elongated.

We can now use the average shape of the Danes population set to morph Obama's portrait to the facial geometry of the average Danish person, as well as morphing the average Danish person's facial geometry to that of Obama's:

For the most part, the results look as expected. Note that the portrait of Obama used for this has a very different scale and orientation than that of the images in the Danes population set, so we do see some effects of incorrect scaling creeping in at the top of Obama's head. However, overall, the portion that corresponds to Obama's face (rather than his head as a whole) is morphed to match reasonably close with the average Danish facial geometry. Similary, sans the scaling effects of Obama's larger head per the size of the canvas, we see the average Danish face's facial geometry resemble Obama's when morphed as such.

Caricatures

A caricature is an image where certain unique features are exaggerated. Here, we present two caricatures of Obama's face: one that exaggerates his personal facial features, and one that tries to find a happy medium between his facial features and that of the average Dane's facial geometry. We define the correspondence points in the caricature as:

caricature_pts = base_pts + scale \cdot (base_pts - avg_shape)

where in this case, the $base_pts$ are the correspondence points of Obama (using Danes point system), and $avg_shape$ is the average Dane's facial geoemtry. A positive $scale$ scale factor will enable morphing Obama's caricature to exaggerate his personal features, while a negative $scale$ factor will err Obama's face closer to the average shape (Dane's average facial geometry).

Here, we show an a caricature of Obama using a $scale$ factor of 0.75 to exaggerate his personal features, and another using a $scale$ of -0.6 to err him closer to the average Dane's facial geometry:

Due to the difference in scaling of Obama's portrait and the images in the Danes population set, we again see some effects that blow up proportions, but the faces do resemble what we expect. The first image errs closer to Obama's shape, widening his head further, while the second is a more controlled morph of Obama's face to the average Danish face. It's interesting that we can use this idea of a caricature to control just how much of certain features we're adding or removing from a face.

(Required) Bells & Whistles: Changing Gender

Building off the idea of extrapolated features from caricatures, we can apply this to change the appearance of gender in an image. Again, we use the Danes population set and apply morphs to Obama's portrait.

To extrapolate male or female features, we need to obtain the average male and female shapes (geometries) and form the average male and female faces. The dataset is split up accordingly, and the average male and female faces are morphed to the average Dane's facial geometry for alignment. The results of the average male and female faces are as follows:

To morph Obama's face into the average male or female face, Obama's correspondence points are added to the difference between the average male shape and average female shape (for a resultant male face), or to the difference of the average female shape and average male shape (for a resultant female face). Similarly, the average male color and average female color can be computed as the difference between one and the other, and then added to the resultant image. Here, we present a morph of just shape, a morph of just color, and a morph of both shape and color, first for the male face, and then the female face:

The results are quite interesting. We don't see too significant of a change, but it is still noticeable, especially in the female morph cases. In the case of the male morph, Obama's face is made slightly more recatangular, and his mouth looks to be smiling just a slight bit. In the case of the female morph, Obama's face is stretched further and made more round. In both of these cases, this makes sense, since we can see that the average male Danish face is slightly rectangular, while the average female Danish face is more round (perhaps due to wider smiles). In both cases, the color is similar, where Obama's skin color is noticeably lighter, though we seem to see effects of a scaling mismatch creeping in again, where there is a little bit of green likely coming from the background used in the images from the Danes population set.

Summary

Overall, I found this project very challenging, but also very worth it since the results are so interesting. I think the coolest thing I learned in this project is the general idea of how a face can be abstracted to certain features (and from there, certain pixels), and we can modify these features using a simple affine transformation.