Project 3: Face Morphing

Omkar Waingankar - Spring 2020

Overview

In this project, I produced a "morph" animation between two images of faces sourced from National Geographic, computed the mean of a population/subpopulation of faces, and extrapolated facial features from these population/subpopulation averages to create a caricature of myself.

Part 1: Defining Correspondences

The first part of this programming assignment involved providing manual input to our algorithms to provide sufficient data to carry out our affine transformations and computations in the following parts. I began by using the CPSELECT library to map approximately 40 correspondences (pictured below) between my two source images, carefully focusing on key facial featuers such as their lips, eyes, nose, corners, and the outline of their heads.

From there, I was able to leverage SCIPY's Delaunay triangulation library function to split the source images into a series of triangles, as pictured below, for eventual warping. Delaunay triangulation is a triangulation method which seeks to minimize the maximum edge length across all triangles in the graphic (avoiding long, skinny triangles), and is implemented naively by randomly swapping edges within quadrangles and seeing if the local edge lengths are reduced. This method is excellent for our use case because it will help construct a more even grid across our source images.

Part 2: Computing the "Mid-Way Face"

The second part of this programming assignment is at the core of this unit of material, and involves computing the average face from the two source images above (in both geometry and appearance). First, I found the average keypoints between the two image's keypoints - this would serve as the geometry of the mid-way face. From there, using the triangulation I computed in the first part, I iterated across the triangles in both images one at a time and computed a separate warp from each triangle's geometry to the corresponding triangle from the average geometry.

After correctly computing the inverse warp of both faces to their average geometry, I needed to correctly combine their RGB values. To perform this cross-dissolve operation between the colors of the two images, I simply weighted each of the warps equally and took their sum.

Part 3: The Morph Sequence

After computing the mid-way face, which can be thought of as an interpolation between the first face and the second face with equal weights of 0.5, I refactored my code to accept any weight, W, and computed distinct interpolations between the first face and second face at 45 evenly spaced intervals of W in the range [0,1]. When strung together at a high enough frame rate, the result is the full morph animation, as displayed below.

Part 4: The "Mean Face" of a Population

The next phase of this programming assignment involved generalizing the functions and techniques from the midway image and extending to computing an "average face" across any number of N images.

Female Subpopulation Average

I sourced 200 normalized, aligned, and annotated headshots of Brazilian students and staff from the FEI Face Database, and chose the subpopulation of all females in the dataset (approximately half of the images). I generalized my code from previous parts to compute warped geometries for each image to the average, and then took an average across all the layers at the very end.

Morph to Subpopulation Average

Next, I took random photos (both male and female) from the original dataset, and computed a full warp from each of their geometries to the geometry of the average subpopulation. The first row are the original images, and the second row are the warped faces.

Upon inspection, we can see how these warps influenced certain facial features across all of the images. In general, wider noses and faces were squeezed together, taller noses were shrunk (and smaller ones were elongated), distances between eyes and eyebrows were made more consistent/symmetric, and jawlines/facial structures were pulled towards a more oval shape.

Population Average

Below is the population average: computed similarly to the subpopulation average except across all 200 images.

Personal Warping with Subpopulation Average

In this final step, I warped my facial geometry to approximate that of the average female subpopulation, and also vice versa. My eyes, nose, and smile are slightly more asymmetrical than the average female subpopulation, and one can see this reflected in the warp from the mean geometry to my geometry.

Part 5: Caricatures: Extrapolating from the Mean

Now that we have computed a female subpopulation average above, we can use this image and its keypoints to caricature other images (i.e. extrapolate and extend certain aspects of facial features that are more prominent/present in this subpopulation average).

To perform this transformation, I computed the difference between the subpopulation average keypoints and my profile image keypoints, and rather than interpolating to some weighted average between the keypoints as in the morphing procedure before, I extrapolated the difference by a factor ALPHA on top of my profile image keypoints. In this manner, using a very similar warp function as before, I set the warp target to an exaggerated caricature of myself in the direction of the subpopulation mean.

Notice how as ALPHA increases, the morph characteristics we observed in Part 4 are further and further exaggerated (lengthening of the nose, forcing of facial structure to a more oval shape, etc.).

Bells & Whistles

For the required bells & whistles in this project, I teamed up with 15+ other students in CS 194-26 to create a video montage of chained morphs from one classmate to another. I contributed by computing and recording the the morph from Bernard Zhao (classmate) to myself, pictured below.

The link to the full video can be found here.

Conclusion

My favorite parts of this project were the ones where I could use images of me and my friends to learn and work with something new from the geometrical and appearance representations of images. For instance, I was able to compute a full morph animation between me and my close friend, Bernard, and seeing the animation come to life was simultaneously eerie and exciting for both of us! The parts of this project I found most challenging were the conceptual parts (initially, I accidentally computed a forward warp initially instead of an inverse warp), and I found the parallelism tasks with numpy to be relatively easier.