# Programming Project #4 (proj4)CS194-26: Image Manipulation and Computational Photography

## Face MorphingProject Due Date: 11:59pm on Thursday, October 18, 2018

### Overview

In this assignment you will produce a "morph" animation of your face into someone else's face, compute the mean of a population of faces and extrapolate from a population mean to create a caricature of yourself.

A morph is a simultaneous warp of the image shape and a cross-dissolve of the image colors. The cross-dissolve is the easy part; controlling and doing the warp is the hard part. The warp is controlled by defining a correspondence between the two pictures. The correspondence should map eyes to eyes, mouth to mouth, chin to chin, ears to ears, etc., to get the smoothest transformations possible.

To start with, you should take a pictures of yourself on a uniform background (for instance, white). Your image should be the same size and aspect ratio as your target face (for instance, this beautiful portrait of George, taken by Martin Schoeller). Treat your target face as a passport photo template -- your face in the picture should be about where their face is. This will make your morphing result more pleasing to the eye.

Use your photo as Picture A and your target person's photo as Picture B. You'll morph still picture A into still picture B and produce 45 frames of animation numbered 0-45, where frame 0 must be identical to picture A and frame 45 must be identical to picture B. In the video, each frame will be displayed for 1/30 of a second. Create a video from your sequence of frames, either a YouTube or an animated gif.

### Defining Correspondences

First, you will need to define pairs of corresponding points on the two images by hand (the more points, the better the morph, generally). The simplest way is probably to use the cpselect (matlab) tool or write your own little tool using ginput (matlab or python) and plot commands (with hold on and hold off ). In order for the morph to work you will need a consistent labeling of the two faces. So label your faces A and B in a consistent manner using the same ordering of keypoints in the two faces. It's strongly recommended that you save the points once you obtain something you are happy with so that you don't have to do all that clicking more than once!

Now, you need to provide a triangulation of these points that will be used for morphing. You can compute a triangulation any way you like, or even define it by hand. A Delaunay triangulation (see dalaunay and related functions) is a good choice since it does not produce overly skinny triangles. You can compute the Delaunay triangulation on either of the point sets (but not both -- the triangulation has to be the same throughout the morph!). But the best approach would probably be to compute the triangulation at midway shape (i.e. mean of the two point sets) to lessen the potential triangle deformations.

### Computing the "Mid-way Face"

Before computing the whole morph sequence, compute the mid-way face of your images A and B. This would involve: 1) computing the average shape (a.k.a the average of each keypoint location in the two faces), 2) warping both faces into that shape, and 3) averaging the colors together. The main task in warping the faces into the average shape is implementing an affine warp for each triangle in the triangulation from the original images into this new shape. This will involve computing an affine transformation matrix A between two triangles:

```A = computeAffine(tri1_pts,tri2_pts)
```

(You will write this function.)

A set of these transformation matrices will then need to be used to implement an inverse warp (as discussed in class) of all pixels. Functions tsearch (matlab and python) and interp2 (matlab) or interp2d (python) can come very handy here. But note that you are not allowed to use any built-in offerings for computing transformations, (e.g. imtransform, cp2tform, maketform (for matlab), and scikit's warp or any of the transformation matrix generating functions (for python). etc). Note, however, that tsearch assumes that your triangulation is always Delaunay. In our case, this might not always be true -- you may start with a Delaunay triangulation, but through the course of the morph it might produce skinny triangles and stop being Delaunay. To do non-Delaunay tsearch in MATLAB, David Martin (who is now at Google) has kindly given access to his versions of tsearch that work on any triangulation: mytsearch.m and mytsearch.c (compile by typing "mex mytsearch.c" in Matlab; if you get compiler errors try replacing mytsearch.c with this one. If you get lots of type errors, try this one.) Python has a similar method called contains_point for Path objects in matplotlib. Here is an example.

An alternative method (which may actually be faster) is to generate the mask directly using draw_polygon in python or roipoly in MATLAB. This method performs the same calculation as mytsearch above, but only on those pixels inside the bounding box of the polygon. Although you will end up testing each pixel more than once, you only need to check it against nearby triangles rather than all the triangles.

Show us the original A and B images as well as the image of the mid-way face that you got.

### The Morph Sequence

You need to write a function:

```morphed_im = morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac);
```

that produces a warp between im1 and im2 using point correspondences defined in im1_pts and im2_pts (which are both n-by-2 matrices of (x,y) locations) and the triangulation structure tri . The parameters warp_frac and dissolve_frac control shape warping and cross-dissolve, respectively. In particular, images im1 and im2 are first warped into an intermediate shape configuration controlled by warp_frac, and then cross-dissolved according to dissolve_frac. For interpolation, both parameters lie in the range [0,1]. They are the only parameters that will vary from frame to frame in the animation. For your starting frame, they will both equal 0, and for your ending frame, they will both equal 1.

The output of this part should be a video sequence of a morph from your image A to image B. Please do NOT put a video file on your website! Either include a link to a YouTube video, or create an animated gif.

### The "Mean face" of a population

Pick a freely available dataset of annotated faces (for instance the Danes or this one or something from here or ask for permission to use this one). Using the keypoints already annotated on the data:

1. Compute the average face shape of the whole population or some subset of the population - say, all the girls or all the old/young/white/asian/blond etc. However, if you pick a subpopulation - make sure it contains enough faces for this to be interesting.
2. Morph each of the faces in the dataset into the average shape. Show us some examples.
3. Compute the average face of the population and display it.

Show the mean image that you got, as well as 1) your face warped into the average geometry, and 2) the average face warped into your geometry.

### caricatures: Extrapolating from the mean

Produce a caricature of your face by extrapolating from the population mean you calculated in the last step. This might work better if you have gender-specific mean or a mean of a subset of the population that look similar to you.

### Bells and Whistles - (you will have to do at least one in order to get full crediT)

• (up to 3 points) Change age/gender/ethnicity/smile/etc of your (or your friend's) face. You can use average images off the web for this, no need to recompute the averages yourself (unless you want to). Show morphing just the shape, just the appearance, and both.
• (up to 3 points) Make a morphing music video on a theme. For instance, you can pick a photo of yourself from different ages and make a movie that shows how your face changed over time. Or you can morph between your friends/dorm mates etc.
• (up to 3 points) Produce a face-morphing music video of the students in the class! Something like this but waaaayyyy cooler. In order to do this successfully you will have to gang up with other students in the class and organize yourselves in one big chain of students. Each student in the chain will have to create a morph video sequence from their face to the next students face - just like you did for the main part of the homework. In order to end up with something visually appealing it is advisable that all the backgrounds for everyone's photographs are the same -- for instance white. Another trick is to get everyone's face roughly in the same place. For this you can use George as a passport photo template -- your image should be the same size and aspect ratio as George's and your face in the picture should be about where his face is. Finally, everyone should label their keypoints in a consistent manner with one another. For instance, you can use the point labeles in the following two images as a guide for everyone's labelling: points, point_labels. At the end, pick a volunteer to put everyone's videos together in order and add some music. Here are a coule of good tunes, but obviously you can come up with your own!: Kate Nash, Smash Mouth, The Flaming Lips, Steelers Wheel, Asian historical period drama etc. Everyone in the movie gets 0.2 points for each student participating.
• (up to 3 points) Use one of the datasets to compute a PCA basis for the face space. Try performing caricatures and other transformations in the new basis. Compare your results to doing this in normal basis. Are the results better?
• (up to 3 points) Do something else that is fun in this space, e.g. a different morphing algorithm, automatic morphing, "visual lip-syncing", whatever else your imagination comes up with.

### Scoring

To get full credit, you will need to implement a warping algorithm and turn in a video morph (submit a link to Youtube or animated gif) as well as the mid-way face. You will also need to compute the mean face of a population and show results of warping faces into the mean face and producing caricatures. This time you are required to also do at least one bells and whistles option. Doing other Bells & Whistles will earn you extra points.

For turn-in, as usual, code goes to bCourses and websites use the upload utility as per the submission instructions.