berkeley logoProgramming Project #3 (proj3)
CS194-26/CS294-26: Image Manipulation and Computational Photography

Face Morph Example
Face Morphing
Project Due Date: 11:59 pm on Monday, October 12, 2020

Overview

In this assignment you will produce a "morph" animation of your face into someone else's face, compute the mean of a population of faces and extrapolate from a population mean to create a caricature of yourself.

A morph is a simultaneous warp of the image shape and a cross-dissolve of the image colors. The cross-dissolve is the easy part; controlling and doing the warp is the hard part. The warp is controlled by defining a correspondence between the two pictures. The correspondence should map eyes to eyes, mouth to mouth, chin to chin, ears to ears, etc., to get the smoothest transformations possible.

To start with, you should take a pictures of yourself on a uniform background (for instance, white). Your image should be the same size and aspect ratio as your target face (for instance, this beautiful portrait of George, taken by Martin Schoeller). Treat your target face as a passport photo template -- your face in the picture should be about where their face is. This will make your morphing result more pleasing to the eye.

Use your photo as Picture A and your target person's photo as Picture B. You'll morph still picture A into still picture B and produce 45 frames of animation numbered 0-45, where frame 0 must be identical to picture A and frame 45 must be identical to picture B. In the video, each frame will be displayed for 1/30 of a second. Create a video from your sequence of frames, either a YouTube or an animated gif.

Defining Correspondences

First, you will need to define pairs of corresponding points on the two images by hand (the more points, the better the morph, generally). The simplest way is probably to use the cpselect (matlab) tool or write your own little tool using ginput (matlab or python) and plot commands (with hold on and hold off ). In order for the morph to work you will need a consistent labeling of the two faces. So label your faces A and B in a consistent manner using the same ordering of keypoints in the two faces. It's strongly recommended that you save the points once you obtain something you are happy with so that you don't have to do all that clicking more than once!

Now, you need to provide a triangulation of these points that will be used for morphing. You can compute a triangulation any way you like, or even define it by hand. A Delaunay triangulation (see dalaunay and related functions) is a good choice since it does not produce overly skinny triangles. You can compute the Delaunay triangulation on either of the point sets (but not both -- the triangulation has to be the same throughout the morph!). But the best approach would probably be to compute the triangulation at midway shape (i.e. mean of the two point sets) to lessen the potential triangle deformations.

Computing the "Mid-way Face"

Before computing the whole morph sequence, compute the mid-way face of your images A and B. This would involve: 1) computing the average shape (a.k.a the average of each keypoint location in the two faces), 2) warping both faces into that shape, and 3) averaging the colors together. The main task in warping the faces into the average shape is implementing an affine warp for each triangle in the triangulation from the original images into this new shape. This will involve computing an affine transformation matrix A between two triangles:

A = computeAffine(tri1_pts,tri2_pts)

(You will write this function.)

A set of these transformation matrices will then need to be used to implement an inverse warp (as discussed in class) of all pixels. One way to do so is to generate a mask directly using polygon in python or roipoly in MATLAB.

Note that you only need to write one loop here, looping over all of the triangles. Don't loop over the pixels!!!

Show us the original A and B images as well as the image of the mid-way face that you got.

The Morph Sequence

You need to write a function:

morphed_im = morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac);

that produces a warp between im1 and im2 using point correspondences defined in im1_pts and im2_pts (which are both n-by-2 matrices of (x,y) locations) and the triangulation structure tri . The parameters warp_frac and dissolve_frac control shape warping and cross-dissolve, respectively. In particular, images im1 and im2 are first warped into an intermediate shape configuration controlled by warp_frac, and then cross-dissolved according to dissolve_frac. For interpolation, both parameters lie in the range [0,1]. They are the only parameters that will vary from frame to frame in the animation. For your starting frame, they will both equal 0, and for your ending frame, they will both equal 1.

The output of this part should be a video sequence of a morph from your image A to image B. Please do NOT put a video file on your website! Either include a link to a YouTube video, or create an animated gif.

The "Mean face" of a population

Pick a freely available dataset of annotated faces (for instance the Danes or this one or something from here or ask for permission to use this one). Using the keypoints already annotated on the data:

  1. Compute the average face shape of the whole population or some subset of the population - say, all the girls or all the old/young/white/asian/blond etc. However, if you pick a subpopulation - make sure it contains enough faces for this to be interesting.
  2. Morph each of the faces in the dataset into the average shape. Show us some examples.
  3. Compute the average face of the population and display it.

Show the mean image that you got, as well as 1) your face warped into the average geometry, and 2) the average face warped into your geometry.

Caricatures: Extrapolating from the mean

Produce a caricature of your face by extrapolating from the population mean you calculated in the last step. This might work better on a gender specific mean or some other characteristics specific mean.

Bells and Whistles - (you will have to do at least one in order to get full credit)

Scoring

To get full credit, you will need to implement a warping algorithm and turn in a video morph (submit a link to Youtube or animated gif) as well as the original images + the mid-way face. You will also need to compute the mean face of a population and show results of warping faces into the mean face and producing caricatures. This time you are required to also do at least one bells and whistles option. Doing other Bells & Whistles will earn you extra points.

Submission

Due to the department server issue, we are now working on the submission details, the procedure to submit the website will be announced soon in piazza.