CS 194-26 Project 3: Face Morphing

Adithyan Sujithkumar

Defining Correspondences

It's important to identify correspondences (aka points) on both Image A and Image B such that they are identified in the same pattern/order. Shown below are the correspondences for Image A (me) and Image B (George Clooney). I first manually chose 50 points on each image individually making sure the order was the same. Then I added the 4 corners as correspondences for each image, as well.

Computing Midway Face

To compute the midway face, first I computed the average structure/shape of the midway face. This was done by averaging the correspondences of Image A and Image B. Then I used my affine warp function to identify the transformation matrix for a triangle in Image A's triangular mesh to the same triangle in the average shape. This was also done for Image B. These inverse of these matrices can be used to find the pixel locations in Image A and Image B which warp into the shared average shape triangle. Since these pixel locations are often float values, we can use an interpolation function (such as scipy.interpolate.RectBivariateSpline) to find the color value for a given pixel location, even if the values are floats. We can now create morphs between two images. Below are my original image, me warped to the average shape, the midway image between me and George Clooney, Clooney morphed to the average image, and Clooney's original image. the original images of both me and George Clooney, the images of me and George Clooney after being morphed to our average shape, and the midway morph image between us.

Morph Sequence

The morph function I implemented followed the format morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac). For each frame of the morph I had the warp_frac = dissolve_frac (differing these parameters proved useful for later parts of the project). Below is a morph sequence starting with my face and ending with George Clooney's face.

"Mean face" of a population

For this section I used the FEI Face Database as my dataset. I was able to load all the dataset files quickly because of this stackoverflow post which I modified for my purposes. Specifically, I used the spatially normalized images as they were pre-annotated for facial landmarks. The only downside was that these images were in grayscale. I calculated the average non smiling and smiling face in the dataset and have displayed them below.

The left images below is an original image in the dataset, and the right images are the left images warped to the average shape of the average_non_smiling face in the dataset.

Additionally, below are the results when warping my face to the average non smiling face structure and vice versa. The first two images are my original image and the average non smiling image. The next two images are me warped to the average shape of the dataset and the average dataset image warped to my shape.

Carricatures

Similarly to how I created the morph between me and George Clooney, we can expand the parameters a bit to include carricatures at each end of the morph. I expanded the warp_frac and dissolve_frac to [-1, 2] to allow for heavy weighting towards one image on each end of the morph output. However I clip the dissolve_frac to be in between [0, 1] for any frame the frac parameter is outside of that range. Below are some of the carricature results (carricatures of me ([warp_frac = -0.25, dissolve_frac = 0], [warp_frac = -0.50, dissolve_frac = 0], [warp_frac = -0.75, dissolve_frac = 0], [warp_frac = -1, dissolve_frac = 0]) carricatures of the average no smiling image in the dataset ([warp_frac = 1.25, dissolve_frac = 1], [warp_frac = 1.50, dissolve_frac = 1], [warp_frac = 1.75, dissolve_frac = 1], [warp_frac = 2, dissolve_frac = 1]). Below the images is also a gif of the entire morph.

It was cool to see the morph automatically emphasize features I expected it to emphasize (large nose for me and a smaller nose for the average image) even if my carricatures look cursed lol.

Bells and Whistles

Morph to Smiling Average

I calculated the average smiling face for the Brazilian dataset and morphed my original face with it. Below is the original image of me and the average smiling face in the dataset.

The following images are the results. The first image is the cross-dissolve between the two images, the second image is output after morphing the average image to my structure, the third image is the output after morphing my face to the average face's structure, and the fourth image is the midway image between both images.