CS 194 Project 4

Face Morphing

By Stephanie Claudino Daffara

This assignment explored morphing animation between images' corresponding points, computing the mean between images, and playing around a bit with caricatures.

The Morph Sequence

In order to compute a morph between image A and image B, first we must find a "Mid-Face". This image is a half-way mix between two images. In order to create it I had to find the mid points between the two images' correspondence points and compute triangulation. I used scipy's Delaunay function to compute the triangulations. Then for each point in each triangulation in each image, I computed the barycentric coordinates in respect to the mid-face triangles and used those to linearly interpolate the corresponding points in image A and image B. Finally, in order to account for points that are not integers, I used scipy's interpolate function to find the best integer pixel value from both image A and image B. Finally I added the corresponding pixel values between image A and image B and averaged them to get a final mean result.

Here is a mid-face morph between Adrien Brody, photographed by Martin Shoeller, and an image of myself.

Image of myself

Image of Adrian Brody

Mid-face of Adrian and myself.

Next I worked on creating a morph sequence between two images. This was simply executing the morph function I wrote for the mid-face above, except that now I use a warp constant that goes from 1 to 0 decrementing in steps of 1/(frames per second). Some results I got are shown below.

Start image

Picture of my sister

End Image

Picture of Myself

Video of morphing my sister into myself.

I thought it would be fun to experiment on showing someone throughout time (in this case myself). So the first image is a picture of me when I was a kid, and the second picture is an image of me now. The result is kind of like a time-lapse!

Start image

Picture of myself as a kid.

End Image

Picture of Myself today

Gif of morphing my younger self into today.

Avoiding For loops

In the class slides this morph function is said to need two for loops. Prof. Efros mentioned in class that if you could do it in a single for-loop he would give extra points. Well, I made it my goal to vectorize the #$%@ out of my morph function, and by using numpy the way its meant to be used, I achieved morphing my images using no for loops at all! Both images and every thing done to either is done at the same time. The same goes for the mean face below. I will talk about it and show some results soon, but keep in mind that the average of each face is done all concurrently.

Warping using Barycentric Coordinates VS Inverse Transform

I implemented two different methods for achieving image morphing. The first was demonstrated above, using barycentric coordinates taken from the mid-image and using them to interpolate each point in it's original image given it's triangle's vertices. The other method was creating a change of basis transformation matrix that transforms a point from the mid-face into a point in the original image. This matrix is composed of the product between the basis of the mid-face (which can be calculated by using the three points to create your two coordinates), lets call that T0, and the basis of the original image, lets call that T1. Then you multiply each pixel-point in the mid-face by inverse(T0) * T1 and you get the corresponding pixel-point in the original image.

From inspection and testing, it turns out that both methods return the exact same results. Also, against my initial intuition, the inverse-transform method is about 10% faster than the barycentric method. Although, I still rather using barycentric coordinates because the code itself is shorter, more clear, and easier to work with. Here are two results comparing the two methods, demonstrating that they give the same results:

Midway face.

Using barycentric coordinates.

Midway face.

Using Inverse Transformations.

Using barycentric coordinates.

Using Inverse Transformations.

Annoying issues:

You might notice in some of the videos some "jumping" effects between frames. I believe that many of these issues are due to the number of points created. For example, in the image below both images have different patterned background but I didn't add corresponding points to it, therefore you see this "jumping" effect happen between some frames due to those point lying on top of a triangulation's edge:

Delaunay Triangulation

based on the points of the mid-face.

Gif demonstrating jump

Note that the black-and-white sandy noise is due to downscaling my video and converting it into a gif. If you'd like to see the high-res video please email me!

The Mean Face

The task for this part of the project was to select a face dataset with attributes and use it to compute a "mean" face. Being Brazilian, I naturally used FEI Face Database. Below you can see the results for the generic mean face, the male mean face, and the female mean face.

General Mean Face

Male Mean Face

Female Mean Face

Just for fun I decided to also calculate the mean face between my siblings and I (all brazilian too).

Myself

My Sister

My Brother

Sibling Mean Face

Playing around with my siblings faces made me wonder how close we are to the mean face between our parents and grandparents. So I took an image of my four grandparents and my parents when they were all near a similar age (around 20) and starting taking the mean between all of them. So I calculated the mean between each pair of my grandparents, then I took the mean between each pair with their child (my mom and my dad). Then I took the mean between those two final means and got my final family mean! Check it out:

What do you think? Are my siblings mean similar to our "expected" mean based on our genes? Either way, this was a fun little investigation.

Mean Face between my sibling and I

Mean Family face

Caricatures

Here I played around a bit to see how "off" my own face is from my people! I used the female brazilian mean face to try and get more accurate images. The results are pretty funny, which just shows how diverse us Brazilians really are. The warping factor below is how much I applied to my face, and (1 - warping) is how much I applied to the mean face.

Image of Myself

Caricature morph!

warping 0.5

Image of Average Female Face.

In the images below you can see me playing around with extrapolating the warping constant to above and below zero.

warping -0.1

warping 1.5

>

warping 1.8

Class Morph

This is a class warp that I participated in! We each worked on a little chunk of it (from the person before us morphed into ourselves).