Project 3: Face Morphing
cs194-abp
Overview
In this assignment, we produced a “morph” animation of a face into another face, computed the "mean" of a population of faces, and extrapolated from a population mean to create caricatures.
A morph is a simultaneous warp of the image shape and a cross-dissolve of the image colors.
To warp an image, we need to define correspondence between the two images. Generally, the more points mapped, the better the transformation.
I used photos by Martin Schoeller) to do the morph sequence (notably the photo of George Clooney and the photo of Justin Timberlake.)
Part 1: Defining Correspondences
I wrote a short script a’la the last project’s selection script using ginput in python. I decided to use 38 points (ears, eyes, eyebrows, hair, face shape, lips) and have included the annotated image of George:
| George (Annotated) |
 |
Next, I calculated the midway points according to some warp_frac using a helper function:
def interpolate_pts(f1, f2, warp_frac)
Finally, I called the Delaunay function to compute a triangulation of the midway points.
Part 2.1: Computing the “Mid-way Face”
There were three things I had to carefully handle:
- Include the corners (and the half-points of each side) in the points. This made sure all of the image was triangulated, and subsequently warped.
- When you look at
im.shape, really im.shape[0] is the number of rows, which means the height (maximum y-value) and vice versa for im.shape[1]. This caused quite a bit of grief with the usage of the results of polygon, as in the sample code in their docs they do im[rr, cc]. For our use case, we jmust use it as im[cc, rr] when our points correspond to (x, y).
- The images must be the same size and cropped such that it’s approximately the same amount of space between the chin and the bottom of the image, etc. For this particualr issue, I had to re-do my points a number of times.
For the affine matrix, I used the formula in this link to figure out how to calculate it. Essentially, it’s just T_inv = XA_inv; where T is the transformation matrix, X is the target points, and A are the input points. We want T_inv because it’s a little less lossy to map from the target (midway) image back to the original images.
For each triangle, we calculate the inverse transformation affine matrices from the midway triangle to image1 and image2. Then, for each image, we use polygon on the mapped triangle coordinates to get all the points inside the triangle. Next, we can use numpy indexing with the result of polygon to get the values at those points in the image, i.e the corresponding pixels in the image. Finally, we index into the coordinates all the points of the midway triangle and set the value to the average of those pixels.
This is the result of the morph:
| Justin |
George |
Midway Face |
 |
 |
 |
Part 2.2: The Morph Sequence
For the morph sequence part, I generated 51 images. In the gif, each frame is displayed for 3ms.
I used the morph function written in part 2 as a generic helper function, only changing the warp_frac and color_frac parameters with a 0.02 step.
def morph(im1, im2, im1_pts, im2_pts, warp_frac, color_frac):
Part 3.1: The “Mean face” of a population
I chose the Brazilian dataset of annotated faces (linked here).
I used a short script to extract the annotated points before computing the average face.
A sampling of faces morphed into the average shape:
Featuring my little brother (his face is thin, but it’s also the angle and/or not cropped closely enough, which is why the results are mostly amusing):
| Average Face (Neutral) |
Average (Neutral) to David |
David to Average (Neutral) |
David |
 |
 |
 |
 |
Incidentally, I also calculated the average smiling face for usage in 3.2 and the Bells and Whistles.
| Average Face (Smiling) |
 |
Using the average smiling face from part 3.1, caricature of David:
| David |
Smile Caricature |
|
 |
Using the average woman face from the Bells and Whistles section, caricature of David:
| David |
Woman Caricature |
|
 |
Bells and Whistles
For bells and whistles, I chose to do the first: a change-gender warp of my little brother’s face using a composite image from this blog I found on Google.
The warp & color fractions were set to (0.5, 0); (0, 0.5); (0.5, 0.5) when calling morph to generate these images.
| David |
Average Woman (CEO) |
Both |
Just shape |
Just appearance |
|
 |
 |
 |
 |
| Average Face (Smiling) |
Just shape |
Just appearance |
Both |
David |
|
 |
 |
 |
|
This isn’t technically a caricature, but I learned that bad point correspondences make for really terrible and cursed-looking images (featuring my older brother @iamjakelong and the average face):
| Jake Long |
Jake Long and Average Face (Neutral) |
Jake Long and Average Face (Smiling) |
 |
 |
 |
Created by writing MarkDown on HackMD.io and converting it to HTML.
Project 3: Face Morphing
cs194-abp
Overview
In this assignment, we produced a “morph” animation of a face into another face, computed the "mean" of a population of faces, and extrapolated from a population mean to create caricatures.
A morph is a simultaneous warp of the image shape and a cross-dissolve of the image colors.
To warp an image, we need to define correspondence between the two images. Generally, the more points mapped, the better the transformation.
I used photos by Martin Schoeller) to do the morph sequence (notably the photo of George Clooney and the photo of Justin Timberlake.)
Part 1: Defining Correspondences
I wrote a short script a’la the last project’s selection script using ginput in python. I decided to use 38 points (ears, eyes, eyebrows, hair, face shape, lips) and have included the annotated image of George:
Next, I calculated the midway points according to some
warp_fracusing a helper function:Finally, I called the Delaunay function to compute a triangulation of the midway points.
Part 2.1: Computing the “Mid-way Face”
There were three things I had to carefully handle:
im.shape, reallyim.shape[0]is the number of rows, which means the height (maximum y-value) and vice versa forim.shape[1]. This caused quite a bit of grief with the usage of the results ofpolygon, as in the sample code in their docs they doim[rr, cc]. For our use case, we jmust use it asim[cc, rr]when our points correspond to(x, y).For the affine matrix, I used the formula in this link to figure out how to calculate it. Essentially, it’s just
T_inv = XA_inv; where T is the transformation matrix, X is the target points, and A are the input points. We wantT_invbecause it’s a little less lossy to map from the target (midway) image back to the original images.For each triangle, we calculate the inverse transformation affine matrices from the midway triangle to image1 and image2. Then, for each image, we use
polygonon the mapped triangle coordinates to get all the points inside the triangle. Next, we can use numpy indexing with the result ofpolygonto get the values at those points in the image, i.e the corresponding pixels in the image. Finally, we index into the coordinates all the points of the midway triangle and set the value to the average of those pixels.This is the result of the morph:
Part 2.2: The Morph Sequence
For the morph sequence part, I generated 51 images. In the gif, each frame is displayed for 3ms.
I used the morph function written in part 2 as a generic helper function, only changing the
warp_fracandcolor_fracparameters with a 0.02 step.Part 3.1: The “Mean face” of a population
I chose the Brazilian dataset of annotated faces (linked here).
I used a short script to extract the annotated points before computing the average face.
A sampling of faces morphed into the average shape:
Featuring my little brother (his face is thin, but it’s also the angle and/or not cropped closely enough, which is why the results are mostly amusing):
Incidentally, I also calculated the average smiling face for usage in 3.2 and the Bells and Whistles.
Part 3.2: Caricatures: Extrapolating from the mean
Using the average smiling face from part 3.1, caricature of David:
Using the average woman face from the Bells and Whistles section, caricature of David:
Bells and Whistles
For bells and whistles, I chose to do the first: a change-gender warp of my little brother’s face using a composite image from this blog I found on Google.
The warp & color fractions were set to (0.5, 0); (0, 0.5); (0.5, 0.5) when calling morph to generate these images.
This isn’t technically a caricature, but I learned that bad point correspondences make for really terrible and cursed-looking images (featuring my older brother @iamjakelong and the average face):
Created by writing MarkDown on HackMD.io and converting it to HTML.