Irene Geng | i-geng

Part 1. Defining Correspondences

In order to morph one face to another, we need to define pairs of corresponding points on each of the two images, in the same order. I used ginput to select points on an image.

Once we have two sets of correspondences (one for each face), we can take the mean of the correspondences and compute the Delaunay triangulation on the mean set of key points.

Here are Martin Schoeller’s photographs of George Clooney and Heath Ledger.

I chose 76 different correspondences on each face. Here is the mean triangulation drawn on each photograph.

Part 2. Computing the Mid-way Face

To compute the mid-way face of two different faces:

1. Compute the average shape

2. Warp both faces into the average shape

3. Average the colors of the warped faces (cross-dissolve)

The average shape is just the average of each correspondence—the mean correspondences that we used for a Delaunay triangulation.

Warping

To warp a face into the average shape, we need to warp each triangle of the triangulation into the shape of the corresponding triangle in the average shape, using the affine transformation matrix that transforms the source triangle into the target triangle.

The skimage.draw.polygon function generates a triangular mask given the three vertices of a triangle, and we can use this mask to warp all points within a triangle at once (instead of needing to iterate over individual pixels).

We will use an inverse warp to choose the color of each pixel in the warped triangle; if a target point lies between pixels, we can use bilinear interpolation (or nearest neighbor sampling for faster calculations) to sample the color.

Here are both faces warped into the average geometry. The most noticeable differences are the changes in hairline and eyebrow shape.

Cross-Dissolving

The final step is to cross-dissolve the two warped images, which is just linear interpolation of the color values from both images.

Part 3. The Morph Sequence

To create a morph sequence from one face to another, we can repeat the morphing process from Part 2, but with a different fraction of warping and cross-dissolving. The parameter that controls the amount of warping and cross-dissolving lies in the range [0, 1]. When the parameter is 0, the result is the original first face; when the parameter is 1, the result is the original second face.

• The full morph sequence from George Clooney to Heath Ledger is linked here: https://www.youtube.com/shorts/2a4PJa7JCfA

• Obligatory roommate morph sequence: https://www.youtube.com/shorts/gKgimYiZwGg

• Morph sequence of actors who have played the Joker in TV/film: https://www.youtube.com/shorts/zID_cePwajQ

Part 4. The Mean Face of a Population

FEI Face Database

I used photographs from FEI face database (https://fei.edu.br/~cet/facedatabase.html), which is a Brazilian face database that contains images of 200 individuals taken at the Artificial Intelligence Laboratory of FEI in São Bernardo do Campo, São Paulo, Brazil. There are 100 male and 100 female subjects, with ages ranging from 19 to 40 years.

I separated the dataset into two subsets: neutral faces and expressions.

The FEI database images are also annotated with 46 correspondences (I also added in the 4 corners of each image as correspondences). Here are two example triangulations:

Mean Face

The compute the mean face of a population:

1. Find the average shape by taking the mean of all correspondences.

2. Warp each face into the average shape.

3. Take the average color of all warps to get the mean face.

Here are a few examples of faces in the dataset that have been warped into the average shape, for both neutral and expression subsets.

And here are the average neutral and average expression faces.

Here is my face warped into the average geometry, and the average face warped into my face’s geometry.

Another Dataset

Many iterations of the Joker exist in different forms of media, including comic books, television, film, and video games. I chose a subset of 16 Jokers and labeled 66 correspondences on each face.

Part 5. Caricatures: Extrapolating From the Mean

Using the FEI database population means, we can produce caricatures by extrapolation.

Treating the correspondences selected on my (neutral) face as some vector p and the correspondences on the average expression face as some vector q, I calculated p + 1.5(q - p) as the target correspondences, found the extrapolated triangulation, and then warped my face to these targets.

This warped my face to have a smiling expression, although there are some artifacts around the forehead and hairline because the FEI database’s annotations do not include correspondences on the forehead.

Principal Component Analysis

We can take an m x n image and represent it as a vector of pixel values of length m * n. Then, our dataset can be represented as a matrix of dimensions K x (m * n), where K is the number of images in the set. If we perform PCA on these face vectors, we can recover a set of eigenvectors (or “eigenfaces”).

Here are 16 eigenfaces with the largest singular values for the FEI database neutral and expression subsets.

Neutral Eigenfaces

Expression Eigenfaces

Using the Eigenface Basis

By choosing random weights to multiply each eigenface by, and then adding together the weighted vectors, we can generate new faces. Here are two examples, with a basis of 32 eigenfaces and random weights chosen in range [-1.0, 1.0].