In this project, we are learning how to represent images as polygons and do affine transformations between corresponding polygons in different images, creating the effect of a morph.
Correspondences
For this, I used matplotlib's ginput to put in coordinates, and used the scipy Delaunay function to define the best triangulation. Initially... before we needed to weight images... I just took the mean between each point we defined, and then took the triangulation of that, then just used that triangulation to create the polygons for the images we had before.
Here is a test image.
Here is a mesh that was defined for two people
Mid-Way Face
To do the midway, I just implemented an affine transformation between each triangle, getting coordinates of all the pixels in the mask for each polygon, and using the scipy map_coordinates function to quickly go from our final coordinates transformed by the inverse of the affine transformation from polygon in image1 to the polygon in image2. I did this for every polygon, and added up to get the output midway image.
You can see in this midway image that the shirt is the middle in between the two shirt colors and the watermark is halfway there. Everything else: facial hair, skin tone, etc is all also halfway.
Morph
To morph, we just generalize the midway function, just weighting the images by a parameter for both shape and for texture. Other than that, it was exactly the same as the midway function.
Population Mean
To do this, we just generalize the morph function to be between arbitrary amounts of images. Same structure, same ideas, same code, just now for more images weighted even less each.
I used the small danes dataset to produce the following mesh and mean population image.
Here are some images from the dataset that are warped to the mean population shape.
Here is my face warped to the population mean shape.
Unfortunately, warping to the shape of the danes mean image didn't do me any favors, but it is worth noting that shape deformations of my face against this dataset were to poorer effect than color since my face is considerably thinner than the faces in the dataset.
Caricatures
Here, we just take the color vector between two population images (one of which can just be the mean global image), that have the same alignment, and then add that to another image to caricature it in some way. I did cut a corner in assuming that computing the mean of a population and morphing commute as operations, just to improve the number of these images that I was able to make. That being said, it worked ok. I took the mean population images, and then took shape vectors between the two, added this difference scaled to the input image's shape, morphed the input and 2 population means into the output image's modified shape, and then took the color vector between the population images, scaling and adding to the output image. Here I did it to myself using the danes as the male/female populations, trying to make myself more masculine.
Here is myself to begin with.
I started with a low poly mesh of my face and the mean dane face.
This gave the following results with low, medium, and high levels of caricaturization (I apologize for the nightmare fuel that is the last one).
You can clearly see that this thinness of my face is being exploited the most here. My face becomes considerable thinner with higher levels of caricaturization. You can also see that my mouth which was open becomes more and more open since the danes mean image wasn't smiling at all. The effects of my eyes being at a slightly different orientation is also changed more and more over time. My mouth was also jagged as a result of the fact that I only used 4 keypoints around the mouth. My background is white (whereas the danes was green) resulting in a blindingly white background for me. The same happened for my hair. The shadows on my face due to differences in lighting can be seen in that my skin becomes darker as time goes on. My ears also get moved up. This is an artifact of the fact that I set my ear keypoints to be a little higher than they probably should have been. You can see this in the meshes.
I tried again with a higher poly mesh to see if I could correct the last output and make it look a little nicer overall.
The fact that my background was different and hair color was different is still at play, but my mouth is a lot more rounded as a result of using 8 keypoints instead of 4. My eyes have the same effect since I used 4x the keypoints for the eyes compared to the low poly mesh.
Bells and Whistles
Gender Changing
Here, I am going to show the results from caricaturing my face to be specifically more masculine, and then turning my face into a feminine face.
Here are the mean masculine and feminine faces that we obtained from the dataset, and the mesh that was used to caricature my face.
Here are the results for a more masculine me. The first one is with my face first warped to the population mean, and the second is without that being applied.
The results aren't bad, I definitely look like I have more facial hair. You can see some artifacts behind me from the negative of all that hair in the mean feminine image being subtracted out. Having done it both ways, I think it was a better idea to warp to the population mean and then colorize since my face wasn't really in the dataset's distribution of faces, as my face is considerably thinner than any of the danes.
Now let's see what my face looks like at halfway between masculine and feminine, and what my face looks like when I'm feminine. The meshes and population images can be seen below for male and female.
Here are the feminine photos where I am warping my face to the mean shape to begin with. The first is my face warped to the population mean, the second is half feminine and the third is a full feminine photo.
I can sorta see that I look slightly more feminine over the base image. My upper lip becomes brighter from the lack of facial hair whatsoever. That being said, it's not great, so I went at it again with actual shape morphing in the feminine direction as well. The results were slightly better.
The first is my face originally, the second is a half feminine and the third is full feminine.
These look much better. The middle one is definitely gender ambiguous, and I can see the last one being more feminine. The bright upper lip is still a little jarring, but I think it does add to the effect of making myself look more feminine.
A high poly mesh might have been a little nicer, but it's only really the eyes that look distorted, and I feel like that's not the worse since eyes aren't really what differentiates between men and women.