In this project, I consider morphing between two faces by wrapping between the structural similarity between two sets of images.
The project also involves looking at the average face of a population and caracetures.
Defining Correspondences
To define correspondences between two faces, I select points on the face that represent important features such as the sides of the nose or the mouth.
The two faces that I deceded to compute correspondence between is barack obama's closeup photo and david lynch. The photos are both closeups and are cropped to be the same size.
Original Images
Original Lynch Image
Original Obama Image
Following is a video of the order correspondences defined:(https://drive.google.com/file/d/1U7F6KwOXaDQxTK7VsNTNtBBqM61u7-5h/view?usp=sharing)
Obama With Points
Then I ran delauney on each image seperately to verify that it's accurate.
For the midway face/calculation, I averaged the points than used delauney on it.
Obama with delauney
Lynch with delauney
Midway Face
The midway face is computed via an affine transformation from the midway between the structures of the two faces.
To compute this transformation, we first use the delauney to get every triple computed. Since the order of the triangles is the same as the points, we get the relevant points from each image.
After adding some [1,1,1] to the points (to allow for the affine translation), we compute the inverse transform from the average points to the points of each image.
After computing the transform, we want to set all the pixels to the mixed interpolation between the two colors.
We compute the transformation on the list of pixels on the triangle and use Spline interpolation to set the color.
Midway Face. The features of both lynch and obama seem to exist in this photo
Morph
In order to compute the morph, we slowly change how much of image is blended, including how much of the points get blended. There is approximately 50 points in this.
It blends really well!
Morph between the faces
Mean Face
The mean face that I tested with is the average dane male face.
First, given the points, I compute the average delauney. Then for each face, I iterpolate with 1.0/n * color, where n is the number of images(50+).
The average face is added below:
Average face of male danes
When computing the points required, I first tried using the asf files provided in the danes dataset. However, in order to try to extrapolate the danes to other sets of images, I moved to using dlib a face landmark detection library. Note, one tradeoff with the extensibility is that dlib doesn't handle foreheads well.
The following are some images of male danes morphed into the average face. This was done by using the structure of the average face but using the colors formed from the dane's face. This was done on all images. Here are some examples:
Dane man original
Dane man morphed. It can be noticed that his face was wider than average dane and got shrunk. However, since his face is close to average dane it doesn't change as much as the other two images below do.
Dane man original
Dane man morphed. It can be seen the man's original phase was too thin compared to the average face. The morphing made him more chubby. The average dane man's mouth doesn't smile as much so the mouth was also shrunk.
Dane man original
Dane man morphed. It can be seen that the original man was chubbier that the average face. The morphing made his face less chubbier.
The next steps involve comptuing the transformation from the average face to my face and my face to average face. To represnt my face, I'm using the following image of chris pratt.
Both of these steps involve using similar logic of using the average face's structure but using the texture from the face and vice versa.
The following images use dlib. The benefit for this is that the points can be identified quickly without labeling in the same format as the dataset provided. The drawback is the lack of forehead coverage.
chris pratt(stand in for my face. "he's so cool")
Average Dane for chris face. Since chris pratt face is thin in this photo, he had to be same structure as the average dane.
chris face for the average dane. You can see that chris pratt face is thinner and less chubby than the average dane. The average dane had to be squished in order to fit the structure of the average dane.
Caricature of Face
In order to compute to the Caricature of the Face, we first take the average face points vector + gamma * (sample face points - average face points vector). This can either make an extended dane or an anti dane.
Note In both of this images the dlib is used, cauisng possible issues with the forehead. This simplified compute.
Chriss pratt's face extended in the opposite of the direction of the dane(gamm=2). You can see how chris pratt's face has become even thinner than it is in the opposite direction of the dane.
Chris pratt's face extended in the direction of the dane(gamma=-2). This is the extreme dane with the face extended. You can see how chris pratt's face has become more chubby than the average dane in the direction dane.
Bells and Whistles
Made of morph of 15+ past presidents. It has music as well.
The labels were done using dlib.
Useful files to replicate results are in the drive: https://drive.google.com/drive/folders/1oD0tUhMw7KqXCBftD1cWEgGCQeaphtKj?usp=sharing
