In this project, I implemented a morphing algorithm to change an image of a face into an image of another face in both shape (geometry) and color (pixel values). The algorithm relies on manually selected correspondence points in each face. I applied this technique to morph the face of Audrey Hepburn to the face of Marilyn Monroe. I also calculated the "mean face" of a data set of Danish computer scientists. Having these tools in hand, I created caricatures of my face, based on average Danish face. Additionally, I changed my own ethnicity by making my face more Vietnamese, made a video of my face morphing into another student's face, and finally made a music video of characters morphing into each other based on my favorite video game. See sections below for details.
For part 1, I morphed a portrait of Audrey Hepburn into a portrait of Marilyn Monroe. Both images are professional headshots in which the two women are wearing black turtlenecks and have a similar forward-facing pose. The backgrounds are different shades of blue.
Both images were cropped to contain the face of roughly the same size and in the same area of the image. The cropped images were then resized to be 640 x 447 pixels each. The images are shown below.
Following the scheme shown in the template of Professor Efros, I added 6 points along the neck-shoulder line. The order of points to select is shown in the image below. A total of 49 points were selected from each image.
I used Python matplotlib.pyplot.ginput function to select correspondence points from each image. The points were saved into txt and csv files. An average set of points was computed from the two sets of points. The corner points were added to each set of points. Applying the Delaunay triangulation to this average set of points produced 50 triangles. Applying triangulation to the midway geometry reduces the risk of deforming triangles. The Delaunay triangulation of the average set of points is shown in the image below.
Computing the "mid-way" face between Audrey and Marilyn images involved calculating the affine transformation of each Delaunay triangle from each of the images to the midway image. The points inside each triangle were computed with skimage.draw.polygon function. Applying the inverse transform to each triangle produced a set of non-integer coordinates in the source and target image. To find out what color the pixels in the mid-way face should be, I used scipy.interpolate.griddata function. The 2D interpolation function returns the pixel value (the color) of the non-integer coordinates in the midway image.
One important optimization I did is I pre-computed all the non-integer coordinates in the mid-way image first (for all triangles), generated by transforming the source image into the mid-way image and the target image into the mid-way image. Then, I computed the new color of the whole midway image (once for source and once for target). This speedup was significant: the compute time decreased from 30 minutes to under 30 seconds, per image, compared to my initial implementation of shading each triangle in turn.
The mid-way face of Audrey + Marilyn was generated by morphing each image into the mid-way shape. The mid-way color was computed by adding half of each image together: 0.5 * transformed Audrey image pixel values plus 0.5 * transformed Marilyn image pixel values. The original images alongside the resulting image are shown below.
The morph sequence was generated using the same principles from Part 2, with an addition of two factors: the morph factor and the dissolve factor. The morph factor controls how much of each image is present in the shape of the resulting image. The dissolve factor influences how much of each image color is present in the resulting image.
Both the morph and the dissolve factors are real numbers between 0 and 1, inclusively. They both start at 0 at frame 0, when the entire image is Audrey, and both end at 1 at frame 45, when the entire image is Marilyn. The mid-way face in part 2 is an example of this more abstract implementation, with the morph and dissolve factors each being 0.5, which produces the color and shape that are both equally weighted in the resulting image. As the frame number increases, both factors increase linearly, which increases the proportion of shape and color of the Marilyn image in each iteration. The morph and dissolve factors for Marilyn and Audrey always add up to 1. This produced a smooth transition in both shape and color from one image to another in 45 frames.
Combining the 45 frame images into a single video produces a smooth transition effect from Audrey's face into Marilyn's. Generating 45 frames with the process described in this part took ~17 minutes to run. This YouTube video shows a morph sequence from the Audrey image to the Marilyn image in 45 frames, and then back from the Marilyn image to the Audrey image in 45 frames: https://www.youtube.com/watch?v=_WCZIhKg88c. I made the video in iMovie.
In this part, I applied the technique described in parts 1-3 to a larger data set. I chose the Danish data set because it was well-labeled and easily accessible. The data set contains 40 human faces (33 male and 7 female). In this part, I worked only with images of front-facing neutral type, labeled with "-1m" or "-1f". For example, "01-1m.jpg" is one of the images in the data set I looked at, but "01-2m.jpg" is not. Each image is 640 x 480 pixels. Annotations contain 56 correspondence points. Sample landmarks in the dataset are shown in the image below.
Examining the data revealed that images 02, 03 and 04 are grayscale. I removed them from the data set when computing the average shape and color. Morphing each individual face in the set of 37 faces took ~8 minutes. Sample morphed faces are shown in the two images below.
The average face (color and shape) of the data set of 37 Danish computer scientists (30 men and 7 women) in the neutral non-smiling forward-facing position is shown in the image below. The face, not surprisingly, is that of an European white male.
My face warped into the average Danish geometry is shown in the image below (left). Compare to my original face (right).
The averge Danish face warped into mine is shown in the image below (left). Compare to my original face (right).
In this part, I made caricatures of my face, by extrapolating the geometry difference between my face and the average Danish in the IMM data set, described in Part 4. To make caricatures of my face, based on the average Danish face geometry calculated in the previous step, I played around with the difference between my face geometry and the average Danish face geometry. I multiplied the difference between my face and the Danish face by a constant multiplier, then added the scaled difference to my average geometry to find the caricature's geometry of my face. Then, I morphed my face into this new geometry. Some results are shown below. The constant multiplier, alpha, varied from -1.5 to 1.5, in increments of 0.25. Alpha 0.0 returns the original face. Negative results are "more danish", and positive results are "less danish". For large negative and positive alpha values (less than -1.0 or more than 1.0), the geometry of my face stops being human, as it no longer lies in the "face space".
For Bells and Whistles, I implemented: 1) changing the "ethnicity" of my face to be more Vietnamese, 2) morphing my face into another student's as part of a student morph chain video, and 3) making a themed music video of my favorite video game.
In the first part, I changed my ethnicity into vietnamese. The average face geometry of the vietnamese face is from the internet, compiled by FaceResearch.org.
The average vietnamese female face is shown in the image below (left). My face is shown in the image below (right).
The result of morphing my geometry to the vietnamese average face geometry is shown in the image below.
Applying the same principles as in the making caricatures, I experimented with different interpolating and extrapolating factors. Morphing my face into the average vietnamese geometry produced the results shown in the image below. After a certain value, the face geometry starts to break down (around -1.0). More negative alpha values make my face "more Vietnamese".
To change color only, I first morphed the average vietnamese face geometry into mine. The result of morphing the average vietnamese geometry to my face geometry is shown in the image below.
Then, morphing my face into the resulting face produced a smooth sequence of color change only. The results are shown in this video: https://youtu.be/jnrzOdjAWAg.
The results of blending color and shape together between my face and the average Vietnamese female are shown in this video: https://youtu.be/K5QyMC5cnQQ.
All videos were made out of morphing sequence of 45 images, similar to Part 3 of the project. Then, I put the videos together from the images using iMovie.
In the second part of Bells and Whistles, I and 9 other students created a morphing chain video: https://youtu.be/QRuzGcu5MoE.
Student group is: Zixun Huang, Andrew Zhang, Matt Yan, Mark Chan, Chenxi Ou, Irina Hallinan, Paris Zhang, Hugo Zhan, Yi Mu, and Yueheng Zhang.
In the third part of Bells and Whistles, I made a themed music video of 12 League of Legends female characters. The song is "Lost Chapter" by Pentakill. Video link: https://youtu.be/jM1IEOYo5L4.
This is it for this project. It was both fun and challenging to get all the components of the project, but in the end, it felt rewarding to put the morphing technique into practice by making music videos and thinking about how else I can creatively use this computer vision method.