RDJ
CHRIS
Face Morphing
COMPSCI 194-26 // Project 3 // Spring 2020
By Naomi Jung // cs194-26-acs
The goal of this project was to morph faces using triangularizations and affine transformations. While I morphed several pairs of people together by the end of this project, my first input images were Robert Downey Jr. and Chris Evans.
We began by defining correspondances in the two images that we were aiming to morph. These correspondance points included easily matchable features such as points on the eyes, eyebrows, noses, mouths, chins, etc. The labeling had to be consistent, so that we knew which points matched to which in the two images. From here, we were able to create a Delaunay triangulation using a scipy function, which created a triangulation based on the input points we passed in. This triangulation is important because we wanted both of our input images to be morphed into the same triangulation in order to create the morph effect.
Below are screenshots of the triangulation for RDJ, for Chris, and for the middle image, which was defined by averaging the correspondance points of RDJ and Chris. I am including the input image triangulations just to show what they look like, but for the rest of this project, I mostly used the average triangulation, since it made more sense for both images to be morphing into the average, as opposed to only have one input image morphing into the shape of the other.
To compute the "average" face, we began by computing the average face, which involved multiplying the correspondance points of both images by 0.5 and then adding them together to get a set of points that were averaged between them. The next step was to warp each image into the average shape individualy. This involved computing an affine transformation between correspondance points from the image. To calculate an affine matrix, we need to find a matrix A such that AX = X', where X is the original set of points and X' is the new set of points. So given the three points in a triangle (x1, y1), (x2, y2), (x3,y3), we needed to find the 3x3 matrix that would transform these points into (x1', y1'), (x2', y2'), (x3', y3'). To solve for A, we can do A = X'X^-1.
Once we computed the affine matrix, we could use that matrix to transform our points. If we multiply our original points by the affine matrix, we would get the new points in our warped image, mapped back to the original points. However, this would make it difficult for us to figure out what values the points in our new warped image should take on. Instead, if we do an inverse affine transformation, where we multiply the points in the warped shape (which we can attain by using the polygon functions from skimage), we will be able to map each pixel in the warped image to a corresponding pixel in the original image, so we know what values each new pixel should take on. The inverse warp is also better because it eliminates potential holes in our warped image, since the inverse guarantees that
Once we iterate through each of the triangles and inverse warp them as I just described, we will have our warped image as shown above. Once we have a warped image for both of our input images, the last step is to cross dissolve them, by multiplying each image by 0.5 and adding them together. This will result in the midway face as shown below!
This next task was to create the morph sequence from one image to the other. The mid-way face computed in the previous section would be the image that occurs halfway through the morph sequence. To do this, we create a morph function that took parameters for warp_frac and dissolve_frac which were in the range of 0 and 1. These numbers described the fraction of warping/dissolving of the first image, and 1 minus that fraction would describe how much to warp/dissolve the second image. We called this function several times with different fraction values, to produce a sequence of images that morphed from Chris to RDJ over the sequence. The first image was 100% Chris, and as we increased values of the fraction to 0.2, 0.5, 0.8, and so on, the image morphed more and more toward RDJ.
To do this, we simply adapted some of our code from the mid-way face, to use the fraction parameters - we morphed each image into the shape formed by warp_frac * img1 + (1-warp_frac) * img2, and we cross dissolved the images using dissolve_frac * img1 + (1-dissolve_frac) * img2. The resulting images are shown below, at various warp/dissolve parameters. In addition, the GIF demonstrates the morph sequence over 45 frames!
Next we calculated the "mean face" of a population using a dataset from IMM. I chose to calculate the average face for the smiling women in this dataset. Below are some examples of these input faces that I used.
We first computed the average face shape of the population by averaging the annotated data points from each image. We warped some of the input faces into this average shape and produced the following results.
After warping each of the faces into the average shape, we computed the average face by cross-dissolving all the faces together to produce the following average face!
And here is my face warped into the average geometry, as well as the average face produced above warped into my face's geometry!
Finally, we also produced some caricatures by extrapolating from the mean, meaning that if we changed the values of the warp factor to be outside the range of 0 to 1, we would be able to extrapolate the input images. Below are a couple images of me extrapolated from the mean, with different values for the warp factor.
I created a several morph images between many of my friends and created a music video of all of their morphings! Below are some still images of the morphs that I thought were particularly good, as well as a link to the video!
