Project3 Face Morphing

Zhenkai Han

Overview

In this project, I learned how to morph face. The idea is simple: linear interpolation and extrapolation. With triangulation and some linear algebra knowledge, I can improve morph a lot and create funny pictures and generate mean face for population.

Part 1: Defining Correspondences

I wrote my own little tool (set_points.py) to define pairs of corresponding points on the two images. I labeled "my face" in same order and similar position. For triangulation, I used Dalaunay provided by scipy.spatial

"My face" from internet. I resize it to fit George's face with Photoshop.
Corresponding points on "my face". My little tool can display serial number of a point, which is useful to define another face in the same order.
Triangles on "my face".

Computing the "Mid-way Face"

I computed the average shape with linear interpolation. In mid-way face, Its wrap fraction is 0.5. For wrapping face, I need to wrap source triangles to averaged triangles one by one. From my triangulation on start face, I can get indices of each triangle with method simplices of Delaunay. I did following job for each triangle in triangulation:

Since I defined points in order, so the indices also was the indices of corresponding triangle in the averaged points and I could get the three points which define the triangle. With method skimage.draw.polygon, I could convert the three points of triangle to a matrix contains all coordinates of points in the averaged triangle (mask). I wrote a function compute_affine to get the transformation matrix T which transforms source triangle to destination triangle. With linear algebra, the inverse of T helped me to get a matrix contains all coordinates of points in the source triangle. So I could fill the averaged triangles region with color of points in source triangles.

At last, I added all filled averaged triangles together to get mid-way face.

"My face" from internet. I resize it to fit George's face with Photoshop.
Beautiful portrait of George downloaded from project page.
The mid-way face of "My face" and George. Warp fraction and dissolve fraction were both 0.5.

The Morph Sequence

The morph sequence is simple: computing the "mid-way face" with different warp fraction and dissolve fraction. To get the best morphing result, warp fraction and dissolve fraction should be the same for one "mid-way face". After that, I made these "mid-way faces" into an animated gif with python library imageio.

"My face" morphs into George Clooney with 24 frames.

The "Mean face" of a population

The "Mean face"

The mean face is simple: it is the mean of all faces. It is similar to computing the "mid-way face". But for each images, their warp fraction and dissolve fraction are the same: one over how many face in total.

I used dataset from Danes. Since this dataset mainly contain male's face, I calculate the mean face with all pictures matched regex *-1m.jpg. I also read their defined points in corresponding asf file.

A sample face I used to get the mean face.
The mean face I got with all full frontal male faces with neutral expression.
Triangles on mean face.

Warp into geometry

Geometry of a face can be seems as its triangulation. To warp face A into geometry of face B, I need to define points in the same order and number as points in face B so that A's triangulation can match B's triangulation. This time warp fraction is 1 to use B's geometry completely; Dissolve fraction is 0 to use A's color only.

"My face" from internet. I resized and cropped it to fit Danes's mean face with Photoshop.
"My face" warped into the mean face geometry. Now "my face" has a longer jaw.
The mean face. I resized and cropped it to fit "my face" with Photoshop.
The mean face warped into the "my face" geometry. Now the mean face has a shorter jaw.

Caricatures: Extrapolating from the mean

Extrapolating from the mean face is simple: warp fraction should be over 1. It is easy to think that it will make feature of the mean face more obvious.

"My face" from internet. I resized and cropped it to fit Danes's mean face with Photoshop.
The mean face I got with all full frontal male faces with neutral expression.
"My face" with extrapolating from the mean face. Its warp fraction is 1.1.
"My face" with extrapolating from the mean face. Its warp fraction is 1.2.
"My face" with extrapolating from the mean face. Its warp fraction is 1.5.
"My face" with extrapolating from the mean face. Its warp fraction is 2. Larger jaw!!

Bells & Whistles

Change gender and ethnicity of "my face"

I downloaded average face of west African women.

Original average west African Female face from internet
I resized and cropped the average face to fit "my face".
"My face" from internet.
Morphing just the shape. The warp fraction is 0.5.
Animated morphing just the shape.
Morphing just the appearance. The dissolve fraction is 0.5.
Animated morphing just the appearance.
Morphing both the appearance and the shape. The dissolve fraction and warp fraction are 0.5.
Animated morphing both the appearance and the shape.

A morphing music video

I picked some photos of George Clooney from here. After reading his biography, I think everything will be good in the future!