Me
Nikhil
In this project, I got to produce "morph" animations that transformed my face into those of others and vice versa.
A morph involves a simultaneous warp of the shape of the image and a cross-dissolve of the colors of the image. To perform the warp, I had to define a correspondence between the start and end photos, which mapped corresponding facial features between the faces. The cross-dissolve involves changing the amounts of colors pulled from the two different images as the morph occurs.
For my first morph, I used myself as the start photo and my friend Nikhil as the target.
The first step was to define a correspondence betwen our faces. For this, I used ginput and plot in Python to select points on notable features of our faces (for instance, eyes, mouth, nose, and jaw). Once I selected around 65 points that I was happy with, I saved those points in a NumPy array so that I wouldn't have to go back and reselect them if I used the images in the future.
Once corresponding features were labeled, I calculated the average position of each of those pairs of points in order to find the midway shape of our faces. I then found a Delaunay triangulation of those points, which is ideal since it minimizes the number of skinny triangles that can produce a visually unappealing morph. Once I had this triangulation, I was able to compute mappings between these triangles and corresponding ones in the original images, using the points I had selected in those original images.
After mapping all the triangle correspondences, I computed affine transformations between corresponding triangles. Using these affine transformations, I was able to determine the exact pixels from the start and end images to insert into the midway face. I used this information to fill out the empty pixels in the midway face with the correct pixel colors, which gave me the midway face.
The next step was to figure out all of the intermediate frames between the start and the end image, instead of just the middle one. This involved the same general process as computing the midway face did, except with the cross-dissolve and warp fractions rising along a scale for each frame. These values dictated "how much" of each image contributed to the structure and color of the intermediate frame faces.
Another morph I made as I was testing my code transformed Vladimir Putin into Pitbull - aka Mr. Russia to Mr. Worldwide.
For this part of the project, I decided to find the mean face of the Danes dataset. This dataset contained images of many people with different facial expressions. I used the neutral photos of all of the people.
First, I computed the average face shape of this population. This was done by taking the average of the x- and y-coordinates of all corresponding points in all of the images. There were 58 points given for each image, and I added an extra point in each corner.
Average face shape of the Dane population
Next, I morphed every image in my population to the average shape. You can see that all of these different faces transform into one that is more rectangular with a defined jawline.
After this, I computed the average face of this population. You can see that there are some similarities between the average face and the faces that specific people were morphed into above, with the rectangular face and defined jaw.
Average face of the Dane population
Then, I created two new transformations. The first was my face warped into the average geometry, using the average face shape determined earlier. This transformation does a pretty good job of translating my features to to Dane population average face. It incorporates the general features of the average Dane face, such as the neutral mouth expression, smaller eyebrows, and squarish jaw.
The other was warping the average Dane face into my face geometry. This transformation works well, and like the previous one, is a pretty good portrayal of a combination of one face's components with another's structure. It turns the mouth into a smile, increases the eyebrow size, and makes the jaw more pointy like on my face.
For the previous transformations, I would have weights of faces on a scale from 0 to 1, where 0 represents completely using the first face, and 1 is completely using the second. For caricatures, we want to heavily exaggerate a specific face's features. For instance, if we want to caricaturize face 1 using face 2's features, we can think of this as pulling the structural weights from those of face 1 toward and even past those of face 2. This can be represented by weighting the shape of face 2 by some value over 1, and those of face 1 by a negative number.
Here are a couple of caricatures of my face done by extrapolating from the population mean I calculated in the last step. Again, in extrapolating features, we're applying very heavy weighting of the Dane population mean face structure for the combination.
Somewhat convincing: Dane weight = 1.5
Unrealistic: Dane weight = 2
The first result is not bad. We see that compared to my earlier transformation that warped my face into the average geometry, the first caricature takes it to the next level with a smaller mouth and squarer jaw. There is an odd angle by the top right of my head, but nothing too bad.
However, the second result is where we can see the dangers of extrapolating from our data too much. There are significant error regions near my forehead. Part of the reason for this is that the datapoints accompanying the Dane dataset do not cover the top of the head at all, which means that it's much harder to determine good correspondences with those regions in other faces. This leads to the sides of my head awkwardly bulging out at the top. However, like the last caricature, it gets the main features of the average Dane population face correct, such as the neutral expression and thick jaw.
Midway