Project 3: Face Morphing

Overview

For this project, I was tasked with producing a morph animation of my face into someone else's face, computing the mean of a population of faces, and extrapolating from a population mean to create a caricature of myself.

Face Morphing

The process of creating the morph can be summarized as follows

  1. Defining Point-wise correspondences between images
  2. Computing a "Mid-Face" for the two images to meet
  3. Creating a video/gif of the above blending process

Defining Correspondences

To create a proper blend between two images, it is insufficient to naively stack scaled (between 0 to 1) images on top of each other. As seen in project 1, translation alone is often insufficient to find a proper alignment between two images. Another insight from project 1 is that it is often more efficient to identify image correpondence based on image features rather than pixel values.

Herein, when defining correspondences between images, I focused primarily on key features that would best ensure proper image alignment. Take for example the following photo:

Assuming we have another image that is similarly labeled, we no longer align based on pixel value similarity. Rather, we align based on the more important features of eyes, ears, mouth, face, head/hair, nose, etc.

To define these pointwise correspondences, I used the python tool cpselect.

After identifying (i.e. suffering through an all-too-extended period of seemingly mind-less clicking). I averaged the sets of points from both of the orignal images to compute a meeting point (i.e. where the points from both the source and target images contribute equally to the blended photo). The aforemented set of averaged points was used to create a fixed triangulation. Below I show images of the normal triangulation of points on the left and the average triangulation of points on the right

Let's briefly analyze the significance of the above operation. Segmenting an image into many triangles that will each be transformed is easier than transforming a whole entire image. Having a shared set of images segments (i.e. triangles) is much easier to blend together than if we had 2 distinct sets of image segments.

Making the Mid-Face

With the shared set of triangles, I then proceeded to calculate the change of basis necessary to complete the warps necessary from each source image to the midpoint. As touched upon in class, the transformation needed for the morphing operation is affine. Further, the, inverse warping technique (i.e. from destination image to input image) tends to the best results. This required solving a linear system of equations to calculate the coefficient matrix needed to map the sources faces to the mid face. I then used the inverse of this found coefficient matrix to complete the inverse warp from the source to mid-face. Below, I've included photos of the source images and the calculated mid-face

The Morph Sequence

The warping function that I developed in the mid-face portion of this project was easily adapted to compute the morph sequence. By introducing a scaling factor (bound between 0 and 1), I was able to place more weight one specific source images depending on the the stage of the morph. Specifically, in the early stages of the morph, I placed more weight on the points and pixel values of my face. In the middle of the warp, both my face and my target's face shared equal weight. In the latter phase of the morph, my target image's point and pixel values dominated. Here is the resulting morph.

The Average Danish Face

In the mid-face section of this write-up, we saw how we easy it was to find the average between two faces (even of different ethnicities!!! - Bells & Whistles??). Herein, there stands no reason why we cannot employ a similar tactic to multiple faces. In the following section, I find the average face of 40 male and female danes. The dataset was obtained from here . This dataset was already labeled according to the following 58 point scheme. To permit smoother transitions and the inclusion of the background, I made a slight modification to the original dataset and included the 4 corner points of the scene.

Just like before, I calculated the average for all the annotated forward facing images (both male and female alike), used a shared set of triangles to complete my warps, and finally ended up with the following image. May I present the average Danish face:

Given that this is the average across a very male dominant datast, the features of the resulting average images are very male dominant. However, this is not to say that warping male faces to the this average face results in normal looking people. I've included a number of photos below to show that this is, in fact, in no way a guarantee.

Creating Caricatures

For this portion of the project, I experimented with morphing my face to that of the average Dane. The result is ~terrifying~ below:

From this photo, it can be deduced that I ~photoshopped my face to fit the danish dataset image dimensions~ have a narrower face than the average dane and that the eye level of my photo was lower that of the photos in the original dataset.

Just for fun, I've included the result of warping the average danish face to my own

Bells and Whistles

I teamed up with a number of other classmates to create a music video morph between all of us. My contribution was creating the morph from Matt Owen to me. Here is the morph that was included in video

The entire video is embedded below!

In [4]:
from IPython.display import HTML
HTML('<iframe width="640" height="480" src="https://www.youtube.com/embed/sTT0-nN3vtc?rel=0&amp;controls=0&amp;showinfo=0" frameborder="0" allowfullscreen></iframe>')
Out[4]:

Extrapolating Female Features

I used this site to select a random subset of female faces and find their average. I then plotted points of correspondence between my face and the average and then female-i-featurified my face. I look terrible

In [ ]: