CS194-26: Image Manipulation and Computational Photography, Fall 2017

Project 4: Face Morphing

Sudeep Dasari (cs194-26-aae)



Face Morphing

Overview

This project seeks to create face morphing algorithm that allows smooth animated transitions from one image to another. First, the start and end images are annotated with matched keypoints. From these keypoints an “average geometry” is computed that breaks the image into non-overlapping triangles. The triangulation is such that each triangle in the first image is paired with a triangle in the last image that has the same keypoints as vertices. Note these keypoints are in different locations in these images so the triangle’s geometries are different. The animation is generated by gradually warping triangles in the first image into the shape of the last while cross-dissolving the pixel values. Thus, a smooth animation is created by gradually warping the geometry and blending the pixels simultaneously.



Defining Keypoints

Matching keypoints are manually annotated on both the start and final image. Below are two examples of such matched annotations.

Annotated Start Image
Annotated Final Image


Generating Mid-Way Image

To generate a Mid-Way image the average geometry of the image has to be computed, and the keypoints have to broken into triangles. The average geometry is found by averaging the corresponding keypoints in the first and last frames. A Delaunay triangulation is computed on the averaged points. This triangulation "groups" vertices into non-overlapping triangles, in such a way that eliminates skinny long triangles (which could cause visible artifacts). Note that while the triangulation is kept constant through the entire animation, the vertices' position in each frame changes! This change over time captures the change in geometry during the animation. Below is a visualization of the triangulation overlaid on both start and end frames.

Triangulation on Start Image
Triangulation on Final Image

The Mid-Way face is created by fitting an affine transformation that warps the original triangles from start/end images into their location in the Mid-Way image. After the geometric warping is complete the two triangles (one from start frame and one from last frame) are cross-dissolved 50%. This creates the perception of blended geometry AND blended coloration, which will be crucial to create the final animation. Below is an example of a generated half way image, with average geometry overlaid on top.

Mid-Way Image


Image Morphs

All the framework is laid to generate image morphs! Instead of generating just the mid-way image, N images are generated that vary (in our case linearly) from start to end. I set N to 45 and playback the animation at 30 FPS. Below are some cool GIFS I generated!

Barack Obama to Jordan Peele
Morphing of 170 Professors

Bell and Whistle: Video Sequence

Using the image morphing algorithm I created a themed music video of morphs between animals in the cat family. Here is the video link. Enjoy!

Mean Images

Overview

In this section data driven methods for computational photography are explored. The mean face of a human population is computed, and used to create a caricature of my face. The Dane dataset was used for all computations, and only the neutral faces (photo 1) were used.



Mean Face

The Dane dataset is composed of pictures of Danish computer scientists. Each picture is annotated with the same set of keypoints. The average face is generated by computing the mean geometry of the dataset (average of all keypoints), warping each face in the dataset to the mean geometry, and then averaging all the warped faces.

Various Sample Faces Warped into Mean Geometry
Original Sample 11
Morphed Sample 11
Original Sample 20
Morphed Sample 20
Original Sample 27
Morphed Sample 27
Original Sample 27
Morphed Sample 30
Original Sample 33
Morphed Sample 33

Here is the generated average face.

Dane Average Face

After generating the sample mean image and sample mean geometry, I took a picture of myself and annotated it with the same keypoints used in the Dane Dataset. I then morphed my image into the geometry of the Dane set and morphed the Dane image into my geometry.

My Face
My Face Warped Into Average Geometry
Sample Mean Warped Into My Geometry

Caricatures

Caricatures are generated by warping my face into the average geometry, calculating the difference between my face and the average, and then adding back the difference. At this point I both warp back into my face’s old geometry, and also warp the image into my shape plus the difference between my face’s geometry and average face’s geometry (with scaling factor of course). Below are the results.

Caricature in Average Geometry
Caricature in My Geometry
Caricature in 'Caricaturized' Geometry

Comparing the caricature in my geometry to the original image the most noticeable differences are due to color. Obvious the background is different due to changes in background, but additionally my skin tone is darker than original image! This is expected as my skin tone is darker than the sample mean’s.

After warping into the caricaturized geometry changes in my face shape are apparent. My nose is noticeable larger and my face is longer! This seems to match the qualitative difference between my picture and the average Dane picture.