Face Morphing

In this project, we saw that faces have extremely interesting and universal properties. We explored and implemented techniques to warp images to each other through the deployment of basic Linear Algebra and approximation techniques such as Delauney. It becomes clear that the execution of successful face morphing is not necessarily to be associated with deep learning technologies as many of these technologies have een established in the previous decades.

Correspondence Points

Defining Correspondence Points

One of the most important things before conducting any morphings is to define correspondence points. This is a way to identify commonalities and constructs the basic structure through which morphing is conducted. Per the project description's discretion, these alignments and correspondences have been picked manually. At first, I attempted to create 70 correspondence points, but after having talked to fellow students, I came to the conclusion that 49 points is enough to produce the exact same result. When conducting this correspondence basis, it is important to remember to pinpoint the 4 corners of the photograph as otherwise the morphing will leave out these areas. Additionally, before constructing any Delauney triangulation, it is important to double check that the correspondence has been put in correctly in sequence. The following correspondence down below has been used throughout the project, altogether with an example of the Delauney triangulation used in creating the Midway Face.

Correspondence
Correspondence Points (Sequentially inputted starting from the left corner and rotating clockwise inwards.
George Triangulation
George Clooney with example correspondence and triangulation
Kevin triangulation
My picture with correspondence and triangulation used for morphing
As we can see, there is no overlapping of triangles. It is important that this does not occur since otherwise we will get some 3D morphing artifact which is undesireable since both pictures exist in a 2D plane exclusively.

Midway Face

midwayface
The midway face of George Clooney and me
Constructing the midway face has been accomplished by iterating over all the triangles within our triangulations. These triangles are associated with correspondence points in both George Clooney's photo as well as my photo. Through those two correspondence points which correspond to n vectors in R2 or alternatively, 2 vectors of Rn. We calculated the (x,y) coordinates of the correspondences in the average shape: $$\text{average shape}= \dfrac{ \vec{X} + \vec{Y}}{2} $$ Then performing a warp from the original correspondences from both pictures to these average shape correspondence points yielded the final shape. For the warping procedure we used basic solving to create the transformation matrix. The only thing left to do was to add the result back through a proccess which is called cross-dissolve.

Morphing Sequence

Morphing sequence GIF from 45 pictures
This morphing picture has been generated through a changing warping and cross dissolving factor. So basically, we are weighing each picture in each frame. At t=0, we start with George Clooney and at t=end, we end up with a picture of me. In between, there are increments of 1/45 of shifts in both aforementioned factors which generate this slow transition into moving away/morphing picture A and B.

Mean Face of the Population

The following task was to create the 'mean face' of a population trying to decipher the average shape as well as the average pixels/colors. The data set used comes from the Technical University of Denmark's Computer Science faculty. It consists of 240 pictures taken from different angles of people that work at their department. The correspondences have been provided in .asf files and thus before use have been parsed and loaded into our python environment. Computing the average was not much different from calculating the midway face or morphing. The only difference here was that we needed to add 1/240 of the pixel values and morph 240 pictures to the average shape. The result down below is quite interesting. The average 'danish Computer Scientist' seems to be a male with mainly Caucasian strengths. In class we talked about how 'oddities' are washed away through taking the mean and hence this average person looking very perfect and symmetrical. Lastly, it also reveals some underlying biases as well, in this situation, showcasing a male rather than a person without leaning towards a certain gender.

The warps have been transformed through manually defining the correspondence of our average Danish Computer Scientist to the basis we created earlier. The center image is the average picture warped to my correspondence. The right picture is vice versa. My asymmetric chin becomes extremely clear. Limitations are that my ears are small and the ears in the average danish Computer Scientist is almost not clear which results in odd stuff going near the ear area.
 
Average CS Dane
The average Danish Computer Scientist.
Mean in Kevin Geometry
Morphing me to the average Danish CS. My asymmetric chin as well as my larger nose and smaller eyes become extremely clear.
Kevin in Mean Geometry
Morphing me to the average Dane geometry. We see a clear the symmetricization of the face as well as flattening of the chin.



 

 

Caricatures

Caricature
Caricature of me


Quite honestly, I don't have large clue of why my caricature looks like this. This caricature has been generated through calculating the distance of geometry of my correspondence and that of the average 'Danish Computer Scientist'. This difference has been added back to my photo. It appears that my long chin has been exaggerated as well as the asymmetry of the face.
 

Bells and Whistles: Transformation of Indian man into an East-Asian woman

Special thanks to my friend Sandeep who volunteered and agreed to be morphed into an East-Asian Woman. The general set up is that we use the average Indian Man and the average East Asian Woman pictures to calculate the distance between the two.
Average CS Dane
Average Indian Man
Mean in Kevin Geometry
Average Chinese Woman
Kevin in Mean Geometry
Original Sandeep

 

 
In terms of warping, this means that we need to add the difference in the both vector bases to Sandeep's correspondence vector basis. This resulted in the following:
Caricature
Sandeep with 'Chinese Woman' shape features. We see an enlargement of the eyes and a narrower face with a pointier smaller chin.
Similarly, we had to do the same thing for color to calculate the average color. Important for that is that both average pictures are subtracted and added back in the basis of Sandeep's Correspondence points.
Average CS Dane
Average Indian Man warped to Sandeep
Mean in Kevin Geometry
Average Chinese Woman warped to Sandeep
Kevin in Mean Geometry
Sandeep with features from the Average East Asian woman
Lastly, we combine both methods and get the following final result:
Caricature
Final result of transformation into East Asian woman


Concluding thoughts

This project was a really fun yet challenging project. It became really clear to me how much we can do with just plain linear algebra. Deep Fakes can be created through the deployment of tools and research done in the end of the previous century. Additionally, it is extremely interesting to see that faces and their elements seem to have some linear base and can be approximated linearly as a consequence too. Some limitations were definitely the backgrounds and the small differences that make the pictures not to be at their highest quality. Nonetheless, I do think the pictures turned out decent.