Introduction
We seek to morph a face to another face, which can be done via linear interpolation. However, merely performing linear interpolation in the pixel space is incorrect- two faces have different geometries so you will see "ghosts" of both face's features (eyes, noses, mouths, etc.) if they're in different locations.
\[\bar{I} = 0.5 I_1 + 0.5 I_2\]
| George | Pixelwise Average | Andrew |
|---|---|---|
 |
 |
 |
Instead, we can interpolate in geometry first, then the pixel space. What that means is warping each face's shapes to some intermediate geometry such that their eyes, noses, and other features all line up, then performing the averaging in the pixel space.
Defining Correspondences & Applying Interpolation in Geometry and Color
For each face, we define 43 correspondence points for eyes, noses, mouths, ears, facial perimeter points, and other features common to all faces. We use these points to define a Delaunay triangulation, splitting each face into sets of triangular regions that correspond to one another.
For each pair of triangles (red and blue), we can linearly interpolate between their points to construct an "average" triangle (dotted purple). We can then warp both triangles into the shape of the average triangle via an affine transformation.
More formally, we represent a triangle's vertices \(A, B, C\) in 2D homogenous coordinates: \[A = \begin{bmatrix}
x_A&
y_A&
1
\end{bmatrix}^{T}\]
Then, for vertices of our source triangle and destination triangle \(A, B, C, A', B', C' \in \mathbb{R}^3\), we can obtain the affine transformation \(T \in \mathbb{R}^{3 \times 3}\) mapping points in \(\Delta ABC\) to \(\Delta A'B'C'\) as follows: \[\begin{bmatrix} \mid & \mid & \mid \\ A' & B' & C' \\ \mid & \mid & \mid \end{bmatrix} = T\begin{bmatrix} \mid & \mid & \mid \\ A & B & C \\ \mid & \mid & \mid \\\end{bmatrix}\] \[T = \begin{bmatrix} \mid & \mid & \mid \\ A' & B' & C' \\ \mid & \mid & \mid \end{bmatrix} \begin{bmatrix} \mid & \mid & \mid \\ A & B & C \\ \mid & \mid & \mid \\\end{bmatrix}^{-1} \]
Then we can take two corresponding triangles from the two images and create an intermediate triangle \(\Delta \bar{ABC}\) with each vertex being a convex combination of the corresponding vertices of the two: \[\bar A = (1-\alpha)A_1 + \alpha A_2\] and then warp \(\Delta ABC\) to \(\Delta \bar{ABC}\) and \(\Delta A'B'C'\) to \(\Delta \bar{ABC}\).
However, the way this is actually implemented in practice is by inverse warping, not forward warping. This lets us compute the coordinate in the source triangle \((x_s, y_s)\) for the coordinate of every pixel inside the destination average triangle \((x, y)\). \[\begin{bmatrix} (x_s)_1 & \dots & (x_s)_N\\ (y_s)_1 & \dots & (y_s)_N\\ 1 & \dots & 1 \end{bmatrix} = T^{-1} \begin{bmatrix} x_1 & \dots & x_N\\ y_1 & \dots & y_N\\ 1 & \dots & 1 \end{bmatrix} \] Then, for each pixel in the destination triangle, we can get the color values from the corresponding pixel in the source triangle, interpolating as necessary for non-integer coordinates.
With both source triangles warped to the same average geometry, we can then cross-dissolve their color values using the same weighting coefficient \(\alpha\) from above.
Midway Face & Morph Sequences
George to Andrew:
| George | Midway Face (\(\alpha = 0.5\)) | Andrew |
|---|---|---|
|
 |
|
The morph isn't exactly perfect- the correspondences are only defined around the face so we will still notice ghosting in the neck and shoulder regions. Also, because of different hairstyles some triangles will include hair in one face and not in the other, leading to ghosting around the hairline as well. There's a small artifact in the left eye which could be attributed to very narrow triangles in that region in the triangulation, likely causing an irregular appearance when we apply warping.
Mean Faces & Caricature (Brazilian Faces Dataset):
We construct a "mean" Brazilian face from the Brazillian faces dataset by warping all 200 faces to their average geometry (by averaging all 200 sets of correspondences). We then weight each face equally and average their color values together. This yields the following face:
We can visualize the morphing process for some example faces in the dataset:
1a.jpg to mean 11a.jpg to mean
We can also show my face warped into the mean face geometry, and vice-versa:
| Andrew | Andrew Brazilian | Brazilian Andrew | Brazilian Mean |
|---|---|---|---|
 |
 |
 |
|
Observe that the differing positions of our eyes means that the warp will naturally distort my eyes up and the mean face's eyes down.
If we relax the constraint for the weighting coefficient \(\alpha \in [0, 1]\) we can weight my face further away from the Brazilian mean face (using \(\alpha=-0.25\)), creating a caricature where the features of my face that are different from the mean face are amplified.
Ignoring the artifacts which are likely caused by pixel value overflows, we can observe that these effects are sort of the opposite of warping my face into the Brazilian mean face. The eyes are pushed down and the eyebrows are flattened, and regions of my face that are more gray relative to the Brazilian image (lower cheeks, chin, upper lip) are exaggerated.
Bells & Whistles: Face Morph Movie
20 members of the class submitted headshots and generated morphs from the previous face to their own. Thanks to George Wang and Lindsay Yang for organizing!
The resulting morph video can be found below:
Conclusion
I learned a lot about the power of affine warping and how it is possible to view multiple spaces (pixel space, geometry) as candidates for linear interpolation.