Face Morphing

Phillip Kuznetsov

cs194-26-aea

Blending between images cleanly can be a complicated process. For some images, simply playing with the opacity of an overlaying image can produce a very natural looking blend.
A timelapse that demonstrates how blending can look if an image is aligned.
However for others, this technique fails - primarily because the images are not aligned properly - causing "ghost" artifacts to appear during the blending process. A quick solution to this problem would be to align the two images prior to running the cross-dissolve. This works via a simple translation for the images above. However, this technique fails to work in general, especially for images like faces.
badclooney
Failed blending because features are not aligned.
What if instead of trying to find a single transformation that works for every single pixel, we allow for many transformations to be applied across the image. This method looks to modify the shape of an image along with the color while performing the cross-dissolve and produces visually satisfying results. The process works as follows: a user annotates keypoints for both images. These points act as the correspondence between the two images. For morphing between faces, the user might want to outline the eyes, nose, and other important features using the keypoint correspondence.
correspondence
Then from there, the program defines a triangulation of the image. An efficient method that I used for this project is known as Delauney triangulation.The triangulation should be identically constructed in each of the images - such that each image has corresponding triangles.
triangulation
Between each of these triangles, we can calculate a transformation based on the vertices. Thus the number of transformations is equivalent to the number of triangles in the resulting triangulation. The transformation is then applied to each of the pixels within the triangle, giving us a visually appealing morph between images.
badclooney

Correspondence UI

To build a coherent correspondence, I hacked together a fairly simple correspondence picker using matplotlib. The basic motivation was that I wanted to make sure the correspondence between two images was roughly aligned and wanted an easy way to ensure this.
corrui
The UI allows a user to define a correspondence with a set number of points, visualizing past points with a marker. Then, the UI allows a user to define a second correspondence, using the first as a reference. The user can toggle between sequential mode annotating one point at a time and batch mode - annotating all points at once.
corrui

Mid-Way Face

The first step in making a cohesive morph is to create a "mid-way" face. This requires us to find a face that has the average shape between two images. For this part of the project I morphed an image of myself and an image of george clooney
The resulting midway face simply requires us to morph each image to the "average" geometry, then we perform a simple cross-dissolve -taking the average of the colors of each image

Morph Sequence

With this technique in hand, we can then make a cool morph sequence that demonstrates a gradual transition from one image to another. Basically, we gradually change the weighting of shape and color between frames, creating a subtle transition between the images.
morph_sequence

"Mean Face" of a Population

With this new technique in hand, we can go further and make an average face for an entire population. I used this dataset containing a whole collection of Danish computer scientists.
avg_dane
Average face of danes
Now we can then morph images from this population into the mean face
Original image
Image mapped to average danish shape
Original image
Image mapped to average danish shape
Original image
Image mapped to average danish shape
Original image
Image mapped to average danish shape
Additionally, we can isolate different portions of this population such as the men and the women, giving mean populations for these specific subgroups. As a note, the dataset is leaning much more towards men than woman (33 vs 7), which is why the average image looks more manly and looks really close to the image of the average male.
avg_dane
Average female danish computer scientist.
avg_dane
Average male danish computer scientist.
Original image
Image mapped to average danish female geometry
Image mapped to average danish male geometry
Original image
Image mapped to average danish female geometry
Image mapped to average danish male geometry
Original image
Image mapped to average danish female geometry
Image mapped to average danish male geometry
Original image
Image mapped to average danish female geometry
Image mapped to average danish male geometry
To try and see what a balanced population might look like, I also made an average face balancing both genders in the subpopulation used to generate the mean face.
avg_dane
Average danish computer scientist - genders balanced
Original image
Image mapped to average danish shape
Original image
Image mapped to average danish shape
Original image
Image mapped to average danish shape
Original image
Image mapped to average danish shape
My original face
My face mapped to average danish face
My face mapped to average female face
My face mapped to average male face
My face mapped to average balanced face

Extrapolating from the Mean

Not only can we map to the means, we can actually extrapolate away from these means to create caricatures of our images. We run this process by simply taking the image and subtracting or adding the mean by some constant factor such that $$X_{new} = X + \alpha * mean$$
Original image
Image extrapolated to female geometry with $\alpha = -0.5$
Image extrapolated to male geometry with $\alpha = -0.5$
Original image
Image extrapolated to female geometry with $\alpha = -0.5$
Image extrapolated to male geometry with $\alpha = -0.5$
Original image
Image extrapolated to female geometry with $\alpha = -0.5$
Image extrapolated to male geometry with $\alpha = -0.5$
Original image
Image extrapolated to female geometry with $\alpha = -0.5$
Image extrapolated to male geometry with $\alpha = -0.5$

Bells and Whistles

Crazy Face Warps

Face Blow Up

Growing a new head

My objective for this B&W was to make a morph that looked like I regrew my head. To start, I cropped out the background of the original image (it caused problems in a few of my tests) and then also cropped out my head for one of the pictures.
The source images for this morph and the headless image's correspondences
Although I did get a neat effect, I wasn't very satisfied with the result (seen in the image on the left below). However, I did discover a variation that I thought was very amusing, especially when run at high speeds.
The original intention was to make a creepy-looking head regrow.
The accidental product
The accidental product - now with more speed!

Home Alone

Nightmare fuel