Face Morphing

Morph one face into another, compute the average face in a sample, and extrapolate from the mean to generate caricatures.

A CS 194-26 project by Kevin Lin, cs194-26-aak

Classic image blending techniques like cross-dissolves don't necessarily create a believable transition between two subjects. To create a believable transformation sequence, the shape of the image must also change.

By combining changes in both shape and appearance when morphing two subjects together, we present a believable face morph procedure defined on a sparse set of correspondence points.

Getting Started

The package is written in Python 3 and requires a recent version of numpy, scipy, and scikit-image. Invoke the main.py file to run each tool.

main.py contains several different sub-utilities for defining input correspondence points, morphing one shape into another, computing the mean of a population, an optimized procedure for applying a shape-warp without a cross-dissolve blend, and a method for extrapolating a caricature based on existing geometry.


usage: main.py [-h] {input,morph,mean,warp,caricature} ...

Utilities for face morphing

positional arguments:
  {input,morph,mean,warp,caricature}
                        Compute face morphs
    input               Input data points
    morph               Morph one face to another
    mean                Compute the mean face
    warp                Warp shape to geometry
    caricature          Extrapolate from the mean shape

optional arguments:
  -h, --help            show this help message and exit
            

Defining Correspondences

Cross-dissolves are fairly straightforward: at each pixel, we just need to take a weighted proportion of each image. The challenge, then, is to come up with the correct change in shape.

In order to define a change in shape, we must first identify the shapes themselves. Because most real-world samples are not simple enough to solve using a global transformation for each pixel, so we'd like to be able to define localized transformations on each part of the input image. But defining the localized mesh grid becomes a problem in its own as it's difficult to define each change in the dense grid.

We instead use a sparser representation based on correspondence points to capture matching features between the two subjects. Landmarks are used to annotate each subject's features to obtain a one-to-one mapping of each feature point. Then, we generate the mesh grid from the correspondence points by employing a Delaunay triangulation across the points.


(109.23750099999998, 166.57581),
(143.9027425, 166.92649499999993),
(77.73999100000006, 204.2258675000001),
(114.54258500000013, 192.1678624999999),
(126.69349500000004, 195.5741399999999),
(139.16443999999973, 192.3297299999999),
(174.13690250000002, 203.9827350000002),
(142.55071249999995, 227.9684275000000),
(110.34089600000009, 227.7870049999998),
(50.61375299999999, 105.18867475000005),
                  
Correspondence points are (x, y) pairs
From sparse points...
to a dense mesh grid

Mid-Way Face

In order to produce the full morph procedure, we first needed to define a function which could compute each intermediary step of the animation. This "mid-way face" is computed by determining the affine transformation between the points of each triangle from the source image to the destination image.

The affine transformation can then be applied to 'pluck' the right colors from the corresponding points in the source and the destination. This captures the shape of the mid-way face. To make it believable, we finally cross-dissolve each pixel to blend the two warped shapes together.

Me
A sample from the FEI face database
The mid-way face

Morph Sequence

The mid-way face represents a single frame in the final animation, so we can create the full animation by computing the mid-way face for each timestep between the source image and the destination image.

The morph animation from me to the FEI sample

"Mean Face"

We then extended with the mean command, which allows us to compute the average over not just two images, but any arbitrarily-large set of images. This allows us to generate interesting visualizations of an entire population of images.

A sample morph from the FEI face database
A sample morph from the FEI face database
A sample morph from the FEI face database
A sample morph from the FEI face database
A sample morph from the FEI face database
The average face across all faces in the FEI database
Triangulation overlaid on my face
Warping from my face to the mean face
Warping from the average face to my face

Caricatures

We can also generate caricatures by extrapolating shapes to and from the mean. Instead of warping between two images, we start with the input, and stretch the transform to exaggerate the image.

Here is a collection of images extrapolating toward the mean face shape of the Brazilian FEI database.

Me at 0.0 (starting position)
Me at 1.0 (at mean shape)
Me at 2.0 (beyond mean)

Here is a collection of images extrapolating away from the mean face shape of the Brazilian FEI database.

Me at 0.0 (starting position)
Me at -1.0 (negative mean shape)
Me at -2.0 (beyond negative mean)

Shape and Appearance

We can also reimagine our own faces using face morphing. Instead of warping between faces, if we hold the appearance or shape constant while modifying the other variable, we can reimagine our faces under in a different gender, ethnicity, or age.

We first collected a sample of three average face images found online.

The average face of ethnic Chinese actresses
The average face of a Taiwanese female
The average face of a Tibetan man

Using the same procedure above, we generated correspondence points. Here, we show the correspondence points triangulated over my face.

Triangulation of the average Chinese actress
Triangulation of the average Taiwanese female
Triangulation of the average Tibetan man

We can then observe the outcome as we hold the appearance constant, but warping the shape into my face.

50% shape morph into the average Chinese actress
50% shape morph into the average Taiwanese female
50% shape morph into the average Tibetan man

Or, if we hold the shape constant but dissolve the corresponding colors into my face.

50% color morph into the average Chinese actress
50% color morph into the average Taiwanese female
50% color morph into the average Tibetan man

We can also morph both shape and appearance at the same time, similar to the morph procedures we've demonstrated previously.

Shape and color morph into the average Chinese actress
Shape and color morph into the average Taiwanese female
Shape and color morph into the average Tibetan man

Using the same technique, we can also generate animations of the morphing process using both appearance and shape.

Shape and color morph into the average Chinese actress
Shape and color morph into the average Taiwanese female
Shape and color morph into the average Tibetan man