Project 3: Face Morphing

by: Gavin Fure

Overview

Using pretty simple matrix multiplication operations, we will be able to create smooth morphs between two images. Using the same warp operations, we can explore the feature spaces of populations, look at averages, and perform all kinds of operations.

Part 1: Defining Correspondances

The first thing we'll need to do in order to warp between two images is to define some consistent points between the two images. Next, we can compute a triangluation over those points. After these two steps, we've created a structure that we can use to warp from one face shape to another. We've created a mapping between pixels and points in one image to those in the other. I'll be using 99 points and Delaunay triangulation. I've chosen to warp between an image of Icelandic singer Bjork and Hong Kong singer/actress Faye Wong. I aligned the two images in an image editing application before importing them to my program. The Bjork image was taken from this link, and the Faye Wong image was taken from here.


Bjork, original image

Bjork, with points

Delaunay triangulation of Bjork's face

Faye Wong, original image

Faye Wong, with points

Delaunay triangulation of Faye Wong's face


The triangulations on these points are good to visualize shape, but they aren't useful for the warp. We want to have one triangulation that relates the points on both images. To do this, we'll calculate an average shape and then find the triangluation of this average shape. The average shape will be computed by averaging the x and y coordinates of each corresponding point in both images. Here is a plot of the average shape's points and Delaunay triangulation.


Average shape points (over Bjork)

Triangulation of average shape (over Bjork)

Average shape points (over Faye Wong)

Triangulation of average shape (over Faye Wong)/th>

You can already see that this is working. These points are pretty close for both, but not perfect for either. Look especially closely at the right eye and the shape of the hair. The movements of the points are especially obvious in these positions.

Part 2: Computing the Midway Face

Now that we have computed the average shape between the two faces and a Delaunay triangulation of that average shape, we can compute the 'midway face'. We'll warp both faces into the average shape, then cross-dissolve them into one face that will hopefully look like both image A and B. This requires us to write the warp function.


For each triangle, we'll have to warp the pixels in that triangle from the source image to the corresponding locations mapped to by the triangle of the destination. To do this, we'll need a transformation matrix between the two bases. I computed the affine transformation algebraically. Since we are looking for the inverse warp (instead of the forward warp), we calculate a tranformation matrix that translates from the coordinates of the result image to the coordinates of the input image. To compute this, we just perform a matrix multiplication between the source matrix (containing the coordinates of the points in the triangle of the original image) and the inverse of the destination matrix. Then, we can run each pixel of triangle in the destination image through this transformation matrix to get the corresponding point in the source image. These resulting points will not be integers, so we run them through an interpolation function (RectBivariateSpline in this case) to calculate the best color values to place in the output image.


Once our warp function is complete, we run both images through it, warping them to the midway shape. Then, we can just average (cross-dissolve) the two warped images together to get the midway face.


Bjork warped to midway shape


Faye Wong warped to midway shape


Midway Image

I think this looks pretty good, but it's not perfect. The largest flaw is with their clothes; there aren't good correspondances between them, and I should have cropped them out. I can crop the midway image to just be the face, and it looks much better. This issue especially affects warped Faye's neck: it looks very strange and inhuman.


Midway image, cropped to just face

Other problems: I didn't get the points perfect, so there are some strange-looking features on both. Faye's chin looks unnatural, and the area to the side of Bjork's right eye is extending farther than it should. In the midway image, the hair doesn't quite line up correctly at the bottom, although it looks really good at the top. The eyes, nose, eyebros, and mouth also blended very well. You can faintly see both Bjork's freckles and Faye's blush, which is a win for detail. Also, there is a small black outline on the bottom and right sides of these warped images. I think this has to do with the corner points not being clicked perfectly in the corners. It is a small border, so hopefully it is ok. It's something that could be easily removed by just chopping off the last row and column of the images. Overall, it's not perfect, but it's pretty good.


Part 3: The Morph Sequence

This section is fairly straightforward. Now that we have a warp function, we can use it to create a morph over time. To do this, we just have to vary the amount of warping and crossfading we do at each step. Then we stack all of the images we created into a gif, and voila we are done!


gif!
Link


Because of the way my points lined up, one of the triangles of the average shape corresponds to a straight line of three points in the Bjork image. This creates a singular matrix, erroring out the first warp operation. I got around that by skipping the first and last steps (which are just warping between images at 0% and 100% warp_frac and dissolve_frac) and manually appending the input images to the start and end of the sequence. Since the warped images all have that annoying little black border but the original images do not, it causes a little bit of visual disturbance when the gif loops, but otherwise this looks like a pretty nice smooth warp. Even the clothes warp into each other decently well, much better than I was expecting. Some of the bits of hair weren't properly accounted for in the points, so they don't end up warping properly, but mostly this turned out okay. Now that we have this, we can morph between any two images! If we are willing to click all of those points, at least.

Part 4: Mean Face of a Population

I used the Brazilian face dataset for this section. The subpopulation I selected was just the smiling group, because this was already clearly delineated in the structure of the data. The first thing we have to do is compute the average shape of all of the faces in the dataset. Since the points are already labeled for this dataset, we are pretty good to go. I just had to write a little function to read in the points, and then we can compute the mean shape. We've done this before, now we just have more images. It's not too different, so here is my results:


Average face shape

Now we can compute the average face! To do this, we can warp all of the smiling faces to the average face shape and then just average them all together, using existing functions. Here are some faces warped to the average shape:

Original Warped

21b

21b warped

41b

41b warped

141b

141b warped

There's a pretty noticeable difference, but the warped faces still look fairly natural. Next, we can display the completed average face!


Mean face


Triangulation of mean face

Next, we'll warp Bjork's face into the mean shape, and warp the mean face into the shape of Bjork's face. I reshaped and cropped the photo of Bjork to line up with the mean face as best as I could, then labelled some new matching points on both of them. Bjork is looking slightly downwards, her head is tilted, and the focal lengths of the two are very different (Bjork is farther away from the camera, and that photo likely uses a longer lens). Also, the mean face doesn't really have very much hair, but Bjork has a lot (and I didn't add any points for hair). Because of these differences, I don't anticipate this to translate perfectly, but hopefully it will be good enough.


Bjork resized (with new points)


New points on mean face


Bjork warped to avg shape


Mean face warped to Bjork's shape

Well.... that's terrifying. I suppose this worked how it was intended to. Bjork isn't smiling in her portrait, so he chin is much higher up in the frame. That throws off the alignment for things, making her chin massive in the result. Plus, I think I was a little sloppy with the points, leading to things like that weird downward warp in her left eye/eyebrow area. The hair also got pretty muddled on the left, but the upper-right part of the image down't look half bad. The mean-face-to-bjork looks even weirder, for the same reasons mentioned above. Bjork's face is just smaller here (even though their nose, eyes, and mouths are aligned), meaning there isn't much information to fill in the background of this warp. Also, the smile looks pretty unnatural because of the tiny size of the chin. Also, since we are warping this to a smaller shape, there are some angular corners on there. These warps would have been greatly improved with more compatible faces.


Part 5: Caricature

Now that we have a mean, we define each image in relation to the mean. We can take the difference between any image and the mean. This difference vector represents the way that any image differs from the mean (obviously haha), and we can exploit that by adding it back to the original image in order to accentuate the differences. To caricature properly, we must caricature both shape and color. I caricatured Bjork and a couple images from the dataset. I sincerely apologize to all depicted because these get a little goofy.

Original Difference from mean Caricature
Bjork
21b
43b

Part 6: Making People Smile

The dataset also includes a group of pictures where people are not smiling. If we find the difference between the means of the two subgroups, we can end up with a vector that represents smiling. Then, we can add that vector to the faces of nonsmiling people to make them smile! Here are my results. First, we'll look at the color vectors:


Normalized smile color vector


Normalized not-smiling color vector

And now, onto some results of performing this on actual images.

Original + Smile Color + Smile Shape + Smile Both Color & Shape
Bjork
15a
140a

This seems to work very strangely even on the images within the dataset. The shape warp seems to perform the best, but the color change just looks weird. The big difference is the teeth, but it's difficult to add those without whiting out the rest of the face. We have to keep alpha fairly small or the whole thing will get whited out.


Part 7: Weezer

Using the warping techniques, I made a short music video warp sequence thingy. I computer the mean weezer, then morphed between the weezers and their faces warped into the mean weezer shape. I also had a happy little mistake happen here: I accidentally computed the mean face while including a full picture or the blue album (which I had labelled with points and everything), and the mean shape came out super weird. I liked the way it looked, so used some of those pictures in there too! This is kinda fun and silly, hope you enjoy. I added the music in the Premier Pro free trial (because we lost Berkeley access to it its so sad), but I got a little confused with the settings so I think it rendered the frame rate kinda weird. It also removed some of the pauses in between images, which were intended. I will include both the original version without music and the version with music.

With music:

Link


Without music:

Link



Conclusion

Warping is pretty cool. It's definitely an effect I've seen many times before, and I always wondered how it was done. It was really cool to find out how to do it for myself!