Proj3 Face Morphing

The goal of this project is to produce a "morph" animation of my face into someone's face, compute the mean of a population of faces, and extrapolate from a population mean to create a caricature.

Defining Correspondence

First, define pairs of corresponding points on the two images by hand. When labeling the two faces, consistency is essential. Here I write a function to complete this task by using $\textit{ginput}$. To insure consistency, I first draw the correspondence pairs on the image and label them in order, and then use $\textit{ginput}$ to pick the points. Second, create a triangulation at the midway shape (mean of the two point sets) by using $\textit{dalaunay}$ function. The result of triangulation is shown below.

Computing the Mid-way Face

This part involves computing the average shape, warping both faces into the shape, and averaging the color. To compute the average shape, I simply take the mean of each pair of correspondence triangles. To warp the two faces, first compute the affine transformation from the average image triangle to the corresponding triangles in the two faces. There are two ways to compute the affine transformation, one is to use linear algebra, and the other is to use coordinate transformation (change of basis). Here, I used linear algebra approach. Then, compute the points enclosed by the average image triangle using $\textit{skimage.draw.polygon}$. Using the affine transformation to find the corresponding points in the two faces, and average their intensity value. The resulting average face of my image and George's image is shown below. The defect at the bottom left corner of the average face is due to the correspondence pair of the corner of coats. The color is resulted from 1/2 of the black color from George's coat and 1/2 of hte light blue color of the background of my image.

The Morph Sequence

In this part, I will produce a sequence of 46 frames of animation numbered 0 - 45, where frame 0 is identical to my image and frame 45 is identical to George's image. Then use python image processing library $\textit{Pillow}$ to create an animation of those 46 frames. I use the same warping factor and dissolve factor, $t\in [0, 1]$. The result is shown here.

The Mean Face of a Population

I used the IMM Face Database to find the mean face of a population. First, I write a function to parse the $\textit{.asf}$ file and obtain the relative coordinates. Then, convert those coordinates to pixel coordinates (column, row), and using the previous procedure to produce a mean face. Only female faces are considered. The most important thing for this part is to add the four corners to the correspondence pairs manually; otherwise, the mean face obtained only contains the face surrounded by dark background.

Looking at the results of warping the images to the average shape, the faces are transformed to the average shape. No matter the relative size of their faces in the original image, the warped image have similar size of faces, which is comparable with the average face. The mean face is shown below, which looks more appealing compared to individual faces.

Now, I am going to show the warp between my face and the mean face by following the same procedure as in the previous part, i.e., finding corresponding pairs, warping my face to average face, and warping average face to my face. Warping my face into average geometry, my face looks skinner and from more round to more rectangular; however, the neck is distorted due to lack of corresponding pair. Indeed, it is hard to estimate correspondence at the shoulder of the average image. Warping the mean face into my geometry, it loses it symmetry.

Caricatures: Extrapolating from the Mean

To create a caricature, use the same function for warping. The only difference between interpolation and extrapolation is the warping factor. For interpolation, warping factor $\alpha \in[0,1]$, whereas $\alpha$ can be any value for extrapolation. Few caricatures with different $\alpha$ values are shown below. When $\alpha = 0$, it is my original face. When $\alpha = 1$, it is my face warped to mean shape. When $\alpha = 1.5$, I have a very cool hairstyle.

Bells and Whistles: Make Myself Happier

I used the IMM Face Database to find the mean face of a population of happy expression. The geometry of morphing can be treated as change of basis. Using $A, \ A_h$ denote the current face and happier current face, $M_n,\ M_h$ denote the mean neutral face and mean happier face, they have the following relation:

$A \xrightarrow[{}]{T_1} M \xrightarrow[{}]{T_2} M_h \xrightarrow[{}]{T_1^{-1}} A_h$

where $A,\ A_h, \ M, \ M_h$ can be shapes or pixels. The result is shown below. Only shape change makes my face kind of distorted but a bit happier. Only appearance change make the smile more natural but with some defects on color. Applying both shape and appearance change produces a decent happier face.

References

M. B. Stegmann, B. K. Ersbøll, and R. Larsen. FAME – a flexible appearance modelling environment. IEEE Trans. on Medical Imaging, 22(10):1319–1331, 2003