Overview
In this project, we were able to apply some very interesting transformations to faces. First, we selected correspondences between a pair of images, then we used the selected keypoints to generate a triangulation. Given the final face shape we wanted to end up with (some linear combination of the two face shapes, usually), we divided the original image into its triangulation and morphed each triangle to its new shape using an affine transformation. Finally, we cross dissolved the colors of two images with the same face shape. Using these techniques, we were able to create a video morphing one face to another, create caricatures, and more!
Defining Correspondences
I decided to use an image of me and my dad, shown below. To select correspondences, I used matlab's cpselect tool. I selected 38 correspondences, plus four more since I added in the corners of the images. Then I generated a Delaunay triangulation using the mean keypoints of the two images, using Scipy's Delaunay function. The triangulations on both images are also shown below.
Computing the "Mid-way Face"
To compute the midway face between me and my dad, I first needed to morph both my image and my dad's image to a midway shape. To do this, I found the average of the keypoints in my image and my dad's image to get the midway keypoints.
Then, I repeated the following process for both images to produce two morphed images:
Once I had the morphed images, I simply averaged their colors to get a midway image. Below, I show the two original images, and the midway image.
The Morph Sequence
Similarly to how I created the midway face, I was also able to create faces that were x% me and (100-x)% my dad. To do this, instead of using the mean keypoints to generate the two morphed images of the same shape, I used warp_frac * my keypoints + (1 - warp_frac) * dad keypoints, where warp_frac varies between 0 and 1. Also, instead of averaging the colors of the two images, I took dissolve_frac proportion of my color and (1-dissolve_frac) proportion of my dad's color. In my case, I used the same value for both warp_frac and dissolve_frac to generate a single midway image. By varying these fractions between 0 and 1, I was able to create a sequence going from me to my dad. The results are shown in this gif.
The "Mean Face" of a Population
For this part, I used the Danes dataset, which already had annotated keypoints for a set of faces. I found the average face shape by averaging the keypoints from all of these faces, and using the same method as before, I warped all of the faces in the dataset to the average shape. A few examples are shown below.
Original Faces Faces Morphed to Mean Shape
Once I morphed all the faces in the dataset to the mean shape, I averaged their colors to get an overall mean face, which is shown below.
After generating the mean face, I also morphed my face to the mean face shape, and the mean face to my face shape. To do this, I first made the image of me rectangular to match up with the mean face image shape. Then, I used matlab to select correspondences between my face and the mean face, then used the same face morphing technique as before. The result for my face warped to the mean shape didn't turn out great, because I was smiling and the mean face was not.
My image, made to be rectangular by adding white space around the edges
Me warped to mean face shape
Mean face warped to my shape
Caricatures: Extrapolating From The Mean
To generate a caricature, we can use a similar process as before, but instead of using warp_frac * im1 shape + (1 - warp_frac) * im2 shape, where warp_frac is between 0 and 1, we can use im1_shape + alpha * (im2 shape - im1 shape), where there is no limit on alpha. By setting alpha > 1, we are essentially "going past" im2's shape and exaggerating the difference between im2 and im1. By setting alpha < 0, we are doing the opposite and "going past" im1. I used my face as im1 and the mean face from before as im2, so that alpha > 1 emphasized the features that made the mean face different from my face, and alpha < 0 emphasized what made me different from the mean face. Below, I show the results for a few different values of alpha.
alpha = -1
My eyes are bigger than the mean face, and so is my smile!
alpha = 1.5
This made my eyes and mouth much smaller.
Bells and Whistles
For this part, I decided to try and change my image to look more like a baby! I found a picture of the average baby online (source: https://i.pinimg.com/originals/27/0b/d5/270bd529a4ea04cee6ed85d66335827e.jpg). This image, along with my image, is shown below.
I selected correspondences between my image and the baby image, and then used those to generate a triangulation. Then, I morphed my image towards the average baby face, using 40% of the baby's shape and 60% of mine. I did the same thing with just color, by morphing the baby image to my face shape and then cross dissolving my image and the average baby image (using 70% of my color and 30% of the baby image's color). Finally, I tried using both shape and color: I morphed both my image and the baby image towards the warped shape, then cross dissolved our colors, using the same shape and color parameters as before.
Me to baby shape, with 60% my shape and 40% baby's shape
Me to baby color, with 70% my color and 30% the baby's color
Me towards baby shape and color