Project4 Face Morphing

Mian Zhong cs194-26-acm

Part 0: Find Correspondence

The first step to prepare image morphing is to annotate correspondence points on two(or more) images. I used cpselect in Matlab and exported the two sets correspondence coordinates in this project. The common and smooth practice is trying to map exactly from one object localization onto the other, for example, eye to eye, ear to ear, lip to lip and etc. Here is an example of me and Mayu Matsuoka (Photo credit: Magnum Photographer) :

For the entire project I use Delaunay triangulation.

Part 1: Mid-Way Face

The morphing process can be considered into two parts: shape and appearance(color). Two important concepts are interpolation of two objects and affine transformation.

Interpolation

Interpolation is the process to use given inputs to approximate something within the range of input. Interpolation can happen between any two objects if a valid multiplication and addition defined on them. In our case, we have two input images, and we are trying to interpolate the shape using coordinates, and color using pixel values. The mathematical expression for interpolation between P to Q with interpolation factor t is:

Affine Transformation

Affine transformation has degree of

I = tP + (1-t)Q

In calculating mid-way face, we simply set t to be 0.5. The real implementation are roughly three parts:

  1. Compute average shape: midway_corr = 0.5*(corr1) + (1-0.5)*corr2
  2. Warp two source images into the average shape: use affine transformation from midway coordinates to im1 and im2 separately
  3. Cross-dissolve the color together

Below are the original images of me(L) and Mayu Matsuoka(R). The mid-way image is in the middle:

Part 2: Morph Sequence

In previous part, we fix t to be 0.5. Now I calculate a sequence of morphed images at different t between 0 and 1. I wanted to show the morph sequence from me to Mayu, but the file size is too large to fit here. Instead, I put another two morphing sequences: Mean Danish Female to me, and modelshow series (in progress).

Part 3: Mean Face of Population

In previous parts, I morphed from one image to the other using different interpolation points. Now, I will calculate the average face from a dataset of face photos with similar idea. I use Danes Dataset to compute the average 37 faces over them. The datasets provide 58 correspondence points for each image. It includes 30 males and 7 females.

I use the provided correspondences of each image to calculate the mean correspondence coordinates. Then I got the triangulation for the mean correspondence coordinates. I used the same triangulation to select the triangle vertices from mean correspondence coordinates and a sample image, and then calculate the Affine Transformation from mean to sample. Therefore I compiled a list of lists of Affine matrices. Each list of affine matrices will transform all triangles in triangulation from mean correspondence coordinates to all corresponding triangles in each sample.

I calculated the average face among both male and female, among male and among female.

Part 4: Caricature

Now, let's extrapolate! In all previous parts, we set the warp fraction to be from 0 to 1 to do interpolation. If we go out of the range, the calculation extrapolates and thus results in some caricature visual effects. Here is a caricature of me morphed in the mean face of danish female:

Bells and Whistles:

Change gender (accidentally ethnicity, smile...)

I use the male mean face from Danes dataset I calculated above and my picture to see the effect of changing gender. Below I will show the original, images with only changing shape, only changing appearance and both. For shape only, I set the warping factor to be 0 and dissolve factor to be 1, so it only interpolates the shape and keeps all my pixel values. For appearance only, I set warping factor to be 1 and dissolve factor to be 0, so it fills all pixels values from male mean face into my face shape. The both is a mid-way face of my face and male mean face. I originally only thought to experiment on changing gender, however, I realized that using these two source images, I also included ethnicity and smile.

I still like to morph both shape and color. The shape only takes me with thinner and longer face with less smiling effects. The appearance only takes the mean male appearance to be more smiley =). When I applied both morphing, I look like a bit hybrid.

Acknowledgement:

I had a lot of fun with this project and I am still working on making a series of morphing of a modelshow with ~10 models (See my last work morph sequence). The key idea of the entire project is to find the proper interpolation(extrapolation) shape and use it to inversely warp to source image, and then do cross-dissolve. For time constraints, I might omit some photo credits but I sincerely acknowledge all wonderful photography work to help my project visually better! As promise, special thanks to our four corner hero Mrina!

Now I am gonna go ahead to clean all my code and submit it before 12 am!!!