Project 3

Define the Correspondences

I used 28 key points to capture the appearance of a face. A lot of points are used to identify eyes and jaw. Besides the 28 points on the face, I also used 4 corner points to track the rest of the picture.

Computing the "Mid-way Face"

To begin with, I used my image and my mom's image. I cropped and rescaled both images accordingly such that both images have the same dimension 600 x 600.

The images (origin vs destination) after cropping and rescaling

The mid-way face

After defining the points, I computed the mid-way face. The left is just the face. The right we have the points over the face.

Triangulation

I get the mid-way face points from averaging all the points from both me and my mom. Then, using Delaunay Triangulation, I form triangles between the average points. I'll keep these sets of indices because all the other frames will follow the same triangulation.

The ComputeAffine function

I used the algebraic way to calculate the affine transformation matrix. For the three points that define each triangle, I stack their coordinates sideways and then add a new row of 1. The coordinate matrix looks like: [[x1, x2, x3],[y1, y2, y3], [1, 1, 1]]. Suppose A is the coordinates of the first points, B is the coordinates of the points of a frame. T is the transformation matrix. B = T A.

$T = B * inv(A)$

I calculated the transformation matrix from original to midway and from second to midway. This gives me information of where in the original (me) or the destination (mom) image should I get pixel values.

Map the pixels

Once we find the transformation matrix and the set of triangles on the original and the destination image, I would use the polygon function to draw a mask on the mid-way face. I called interp2d function to interpolate the pixel value given the (x, y) coordinate. This gives us the estimate pixel value (since the coordinates are almost always decimals) of the original image or the destination image. With the pixel value mapper from the original or the destination, we can cross-dissolve to get the mid-way image.

I struggled with two things. First, the points I got from ginput has a different row-column order than image representation. After consistent failures to align eyes and other parts of the face, I realized I needed to swap the x-y values of the points from ginput to match the image coordinate representation. Second, I struggled with doing pixel calculation (cross-dissolve) through matrices. Once I put all the calculation in a matrix, it speeds things up a lot.

Morphing Sequence

The first frame is just my face while the last frame is my mom. The warp_frac and dissolve_frac are both 1 for the first frame and both 1 for the last frame. As I generated all frames from 0 to 45, I put them together and created a video. Then, I turned the video into a gif for demonstration purposes. Below is the final result.

The "Mean face" of a population

I used the Brazilian dataset to calculate the mean face. I downloaded the 46-point data for the first 200 faces (1a, 1b, 2a, 2b ... 100a, 100b). It took a while to find the right images that match these points.

Similar to Part One, I calculated the average face shape and then morph each face into that shape. I used the same triangulation (the one calculated from the average face) on all 200 faces when calculating the morphing. Each new face has a weight of 1/200. Then I add all the faces together to get the mean face pixel values. Since the original pictures are all grayscale, the mean face is also grayscale.

Here's some examples of a few faces morphed into the shape of mean face

Here's the mean face (shape and appearance) takes into account all gender and all facial expressions

Caricatures: Extrapolating from the mean

I rescaled and cropped my face to match the mean face image dimension (250 x 300). Then I read the image of my face in grayscale and define the 46 points in the right order. To find out the order of the points, I called imshow on each point layering over a sample face for 46 times.

Below shows my face and the caricature of my face morphed into the mean face.