Define the Correspondences
I used 28 key points to capture the appearance of a face. A lot of points are used to identify eyes and jaw. Besides the 28 points on the face, I also used 4 corner points to track the rest of the picture.
Computing the "Mid-way Face"
To begin with, I used my image and my mom's image. I cropped and rescaled both images accordingly such that both images have the same dimension 600 x 600.
The images (origin vs destination) after cropping and rescaling
The mid-way face
After defining the points, I computed the mid-way face. The left is just the face. The right we have the points over the face.
I get the mid-way face points from averaging all the points from both me and my mom. Then, using Delaunay Triangulation, I form triangles between the average points. I'll keep these sets of indices because all the other frames will follow the same triangulation.
The ComputeAffine function
I used the algebraic way to calculate the affine transformation matrix. For the three points that define each triangle, I stack their coordinates sideways and then add a new row of 1. The coordinate matrix looks like: [[x1, x2, x3],[y1, y2, y3], [1, 1, 1]]. Suppose A is the coordinates of the first points, B is the coordinates of the points of a frame. T is the transformation matrix. B = T A.
I calculated the transformation matrix from original to midway and from second to midway. This gives me information of where in the original (me) or the destination (mom) image should I get pixel values.
Map the pixels
Once we find the transformation matrix and the set of triangles on the original and the destination image, I would use the
polygon function to draw a mask on the mid-way face. I called
interp2d function to interpolate the pixel value given the (x, y) coordinate. This gives us the estimate pixel value (since the coordinates are almost always decimals) of the original image or the destination image. With the pixel value mapper from the original or the destination, we can cross-dissolve to get the mid-way image.
I struggled with two things. First, the points I got from
ginput has a different row-column order than image representation. After consistent failures to align eyes and other parts of the face, I realized I needed to swap the x-y values of the points from
ginput to match the image coordinate representation. Second, I struggled with doing pixel calculation (cross-dissolve) through matrices. Once I put all the calculation in a matrix, it speeds things up a lot.
The first frame is just my face while the last frame is my mom. The
dissolve_frac are both 1 for the first frame and both 1 for the last frame. As I generated all frames from 0 to 45, I put them together and created a video. Then, I turned the video into a gif for demonstration purposes. Below is the final result.
The "Mean face" of a population
I used the Brazilian dataset to calculate the mean face. I downloaded the 46-point data for the first 200 faces (1a, 1b, 2a, 2b ... 100a, 100b). It took a while to find the right images that match these points.
Similar to Part One, I calculated the average face shape and then morph each face into that shape. I used the same triangulation (the one calculated from the average face) on all 200 faces when calculating the morphing. Each new face has a weight of 1/200. Then I add all the faces together to get the mean face pixel values. Since the original pictures are all grayscale, the mean face is also grayscale.
Here's some examples of a few faces morphed into the shape of mean face
Here's the mean face (shape and appearance) takes into account all gender and all facial expressions
Caricatures: Extrapolating from the mean
I rescaled and cropped my face to match the mean face image dimension (250 x 300). Then I read the image of my face in grayscale and define the 46 points in the right order. To find out the order of the points, I called
imshow on each point layering over a sample face for 46 times.
Below shows my face and the caricature of my face morphed into the mean face.
Bells and Whistles
A family face-morphing music video!!
Since I already morphed my face into my mom, I decided to change her face into my dad and then finally going back to my face
The three original images: me, mom, dad. All of them are cropped and rescaled so that they have the same dimensions (600 x 600).
The three mid-way faces: me-mom, mom-dad, and dad-me
Once the morphing is complete, I put the frames together and created a video!!
Video link: https://youtu.be/RKHVO7rpX24
Change my ethnicity
I tried to morph my face into an average white male face.
The average white male face was found online. I put the white face at the center of a 600 x 600 canvas to match the dimensions of my face.
shape and colors