Project 3 - Face Morphing

Project overview

In this project, we worked with face images. By finding the correct (weighted) average of two faces through affine transformations, we are able to seamlessly transform one face into another. Faces essentially span a subspace, so some combinations of faces in a population could actually generate any other face in the given population. We worked a little bit in that realm by determining what the mean face of a population (for me, this was the Brazilian faces dataset) looked like.

Defining correspondences

I started out the project using cpselect to select points, but it was crashing my kernel regularly, so the implementations in the later parts use ginput. I selected the corresponding key points on both images, and then used scipy.spatial.Delaunay to create a Delaunay triangulation for ONLY the first image (George). This ensures that both images have the same triangulation pattern.

Computing the midway face

Obtaining the midway face consisted of two steps, averaging the triangluation geometry, and averaging color. For each triangle pair (the corresponding triangulations on both images), I first computed the midway geometry that corresponded to the pair, and then used inverse warping to map an average of the orginal triangles' colors onto this transformed midway triangle. The midway triangle is an affine transformation of the coordinates of original triangle to midway triangle, which we can represent with A. Our inverse warp is A^-1. This allows us to pick a pixel in the midway triangle, find the corresponding pixel in the original image, and color our midway pixel with the same color.

Morph sequence

Here is the full morph sequence between George Clooney's face and my face. I used 60 frames, with each frame displayed for 1/100 of a second.

Mean faces

I found the average shape of the population, tranformed each image's geometry to the average geometry. Here are the intermediate results:

Then, we simply average the entire set of images that has been transformed to the average geometry. The colors/brightnesses are an average.

Here are some transformations! The first 2 rows are people in the dataset, and the last row is me transforming to the average. For these transformations, I used 45 frames, with each frame displayed for 1/60 of a second.

Caricatures

In addition to finding the mean face of the entire set of smiling faces, I also found the mean face for men with neutral expressions. I attempted to use this to create a caricature of myself, transfroming from a neutral face of myself to a male version. I extrapolated the shape such that the final image was 1.3(mean_male) - 0.3(lindsay). Overall, we can see the over masculine features added on. My jawline has become much wider and the shape of my face is dramatically different.

Bells and Whistles

Class morph

I participated in a class morph! Here is the Youtube link .

Big thanks to George Wang for organizing the group! The participants were George Wang, Andrew Lee, Lindsay Yang, Deepshika Dhanasekar, Reini Lin, Jonathan Tan, Alex Kassil, Annie Nguyen, Lauren Go, Sofie Yang, Henry Xu, Nancy Shaw, Bernard Zhao, Omkar Waingankar, Jared Rosner, Matt Owen, Boren Tsai, Adel Setoodehnia, Ani Nrusimha, and Ryan Koh.

Facial expression change

I did a facial expression change, transforming my neutral face to smiling using the average smiling face obtained previously. I transformed my smile shape to the average smile shape, and colored the remaining in using only my face's original coloring

Conclusions and Takeaways

Very cool project, though it did take a bit of time for me to understand that a face warp first involves a change of geometry to the weighted average, and then you inverse warp for the coloring. I also did some experimentation before realizing the best way to create a caricature was to exclude coloring from the average man image I was using. Otherwise, I could really only see the man's face because its edges/colors were dominating.

Lastly, for fun, and it is really not related to the class, enjoy this face morphing video I saw on Reddit