CS 194-26: Computational Photography, Fall 2018

Project 4: Face Morphing

Varsha Ramakrishnan, CS194-26-aei


In this project, we computed a morph sequence of faces by first defining a set of points on two faces, then calculating the warp between both those faces and a median face, and finally warping at different proportions of each image and cross-dissolving between the result images for the sequence. We extrapolated this concept (the notion of faces as a vector of points), to calculate the mean face of a population, swap the gender of a face, and calculate caricatures by increasing the component of the face that is most different from the average image of a face.

Section I: Process of the Morph Sequence

Part 1: Defining Correspondences

The first step to generating a morph is to define a correspondence between faces. I did this by manually selecting points according to a predescribed set of rules, as shown below. The inclusion of the four corners allowed me to think of the background as a set of triangles and therefore allow the algorithm to take care of transferring the background as well.

After this, we calculated a Delaunay triangulation between the points in this correspondence, as shown in the image on the right. This finds the triangles between the points that best represent the image, and returns them as a list of point numbers. It is important to keep the same triangulation between images, as these tirangles will be transformed into the corresponding triangles in the other image.

Me with Defined Correspondence
Me with Delaunay triangulation

Part 2: Computing the Mid-Way Face

In order to compute the mid face between two images, I first found an average shape for the two images in points. Then I warped both source images A and B to follow the points of the average image and then finally use half of each image in the final. The two source images, those images warped to the midpoint, and the merged image, are shown below.

In order to make my algorithm run faster, I iterated through all the triangles in the image, found the masks corresponding to each triangle, warped each mask back to the position it would be in the source image, multiplied to find the corresponding pixels of the source image, and used those source points to fill in the triangle location in the mid-way face. I also kept track of the masks in order to make sure that if a point in the midway face is computed more than once, the value is divided by the appropriate number.

As a result of my vectorization, I can run an image in less than a second, and a sequence of 45 images (as described in the morph sequence below) takes about 30 seconds to compute. This made it very easy for me to run multiple sequences, and the main stopping point was choosing all the correspondence points on each image.

Source A
Source B
Source A, morphed to mid face
Source B, morphed to mid face
Cross-dissolved midface

Part 3: The Morph Sequence

To implement the morph sequence, I simply ran the same algorithm as mid-way face, but with a different alpha constant for each step in the sequence. Varying the fraction of warp and dissolve uniformly between 0 and 1 made for a good sequence (in the midway face, these constants are both 1/2). Here are a few examples

(Note: this can be really trippy to look at, watch out) (Note: You may have to wait for some of these gifs to load).

Me to Robert Downey Jr
Me to George Clooney
Me to Mark Ruffalo
Me to Mark Zuckerberg

Section II: Average Faces and Not-So-Average Faces

Part 4: The Mean Face

I calculated the mean face by first finding the average points of all the images in the category, then warping each image to the average, and finally averaging the pixels of all the images. The results are displayed below.

Face 1 source.
Face 1 warped to average.
Face 5 source.
Face 5 warped to average.
Average Man.
Average Woman.

I also warped my face to the points of the average face and vice versa. The results were very interesting, as my face is markedly different from the average older Danish woman. I have learned from this project that my eyes are uncomfortably large when compared to average Danish women.

My face.
Cropped average face.
My face warped to average.
Average face warped to mine.

Part 5: Caricatures

I computed a caricature by subtracting my face's points from the average face, multiplying that difference by a caricature fraction (about 0.5), and then adding the points back in to the original image points and warping my face to fit those points. I have discovered from this that my observations above are correct, and that I do have very large eyes.

My face, caricatured with alpha 1.5.
A sequence of caricatures, from 0.5 alpha to 1.5

Section III: Bells and Whistles

Part 6: Gender Swap

As an extra idea, I experimented with modifying my perceived gender by subtracting the difference between the points on an average boy and and an average girl, and then multiplying that difference by a factor (about 0.3) and adding it in to my face's original points. The results can be seen below.

Average boy.
Average girl.
Cross dissolved boy.
Cross dissolved girl.
Warped to boy coordinates.
Warped to girl coordinates.
Warped and faded boy transformation.
Warped and faded girl transformation.

I also tried changing my race by warping my face to the standard coordinates of an average asian face. The results are shown below, and the warp factor I used was (0.5).

Average asian woman face.
Cross dissolved me and asian woman.
Warped to fit the average asian woman points.
Full warp and dissolve with average asian face.

Part 7: PCA Basis

I calculated a PCA basis for this system by flattening every set of points in the image set into a 1 dimensional vector, stacking them, and calculating the PCA transformation. I tried one set keeping 30 components (since I only had 36 images, I couldn't get too much more precise), and another set using 10 components.

I then computed an average in the PCA basis and found that the returned points were remarkably similar. This is because I'm using the PCA basis on an image that is comprised entirely of information that the PCA basis has "seen before".

I also separated out one image (the first image) from the set, calculated the pca basis without it, and then caricatured that image. The results are shown below. You can start to see the difference in 1* warp. In the 2* warp, it's clear that with only 10 components, there wasn't enough variation maintained in the top left corner to morph his head inwards in the caricature, as is present in the 30 component image. We can see some information has been lost.

The results are not necessarily better than the original, but they do take less time to compute as you're computing in a lower dimensional space. Since my algorithm already runs pretty fast, (about 30 seconds for a 45 image warp sequence) this was not that noticeable.

Average computed by pca.
PCA caricature, 30 components, 1*warp
PCA caricature, 10 components, 1*warp
PCA caricature, 30 components, 2*warp
PCA caricature, 10 components, 2*warp

Part 8: In-class Morph Chain!

We made a morph chain! In case the video embedding below doesn't work, here's the link: