CS 194-26 Project 3 Report

Defining Correspondences

In this part of the project, I select 25 correspondence points on the source and target images. The source image is my picture, and the target image is George's picture. I take the mean of the two correspondence point sets and then produce a Delaunay triangulation over this mean set. For display purposes, I overlay this triangulation over the two images.

Computing the "Mid-Way Face"

To generate the mid-way face, I compute the average shape between the source and target images by averaging the corresponding keypoints that I've computed in the above step. I then warp both images into the average shape by computing transformation matrices corresponding to inverse warps that map coordinates in the "average" image to coordinates in an original image. Let \(A\) be the inverse warp matrix. We map coordinates \((x', y')\) that will form the average image to \((x, y)\), which are coordinates in the original image using the following system of equations: \[A=\begin{bmatrix} a \text{ } b \text{ } c \\ d \text{ } e \text{ } f \\ 0 \text{ } 0 \text{ } 1 \\ \end{bmatrix}\] \[A \begin{bmatrix} x' \\ y' \\ 1 \\ \end{bmatrix} = \begin{bmatrix} x \\ y \\ 1 \\ \end{bmatrix} \] To solve for \(A\) for a mapping of a particular pair of triangles between the average shape and the original image, we take triangle points from the average shape and corresponding triangle points in the original image and solve: \[X' = \begin{bmatrix} x_{1}' \text{ } x_{2}' \text{ } x_{3}' \\ y_{1}' \text{ } y_{2}' \text{ } y_{3}' \\ 1 \text{ } 1 \text{ } 1 \\ \end{bmatrix}\] \[X = \begin{bmatrix} x_{1} \text{ } x_{2} \text{ } x_{3} \\ y_{1} \text{ } y_{2} \text{ } y_{3} \\ 1 \text{ } 1 \text{ } 1 \\ \end{bmatrix} \] \[AX' = X\] \[A = XX'^{-1}\] In my code, I defined A = computeAffine(tri1_pts, tri2_pts) to do the computation. Then, once I have the matrix A, I can map points within the average shape triangle to points corresponding in the original image. Then, I use bilinear interpolation (scipy interpolate's RectBivariateSpline) to obtain the colors at the source image corresponding to the warped points, and assign the color to corresponding average shape points. I do the aforementioned process of inverse warping for the original source and target images, and then take the average of the two images to get the final midway face. Here's the final result of the midway face between my face and George's face:

The Morph Sequence

I produce a morph sequence between im1=source image (Kristy) and im2=target image (George). I define warp_frac to weigh how much each original image's triangles contribute to the average shape and dissolve_frac to see how much each returned warped original image is used in cross-dissolving to the final image returned. The average shape triangle points is computed by equation \((1-\text{warp_frac}) * \text{warped_source_image} + (\text{warp_frac}) * \text{warped_target_image}\); the cross dissolve is computed by equation \((1-\text{dissolve_frac}) * \text{warped_source_image} + (\text{dissolve_frac}) * \text{warped_target_image}\). I define frames 0 to 45 and let warp_frac = dissolve_frac \(\in [0,1]\), splitting \([0,1]\) evenly between the frames. At frame 0 corresponds to the source image, and frame 45 corresponds to the target image. I create the function morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac) to generate one of the morph's images using a particular dissolve_frac=warp_frac. I call the function 46 times, once for each of the equally spaced values \(\in [0,1]\), and save all the resulting images from the morph function into a .gif that is set with fps=30. Here is the .gif I've created displaying the morph sequence:

The "Mean Face" of a Population

I used 33 male forward grinning faces from the Danes dataset as the population I computed the mean face from. After getting the corresponding points from the .asf files, I computed the average face shape by averaging the corresponding points and creating a triangulation of those resulting points. Then, I morphed each face of the dataset into the average shape by computing the resulting image from inverse warping. Finally, I computed the average face of the population by adding all of the morphs (of each face into the averge shape) and dividing by 33. Here's the resulting average face:

Here are some examples of original Dane images, warped Dane images, and morph of the original Dane images to the mean face:

Here's a video of Dane 0 warped to the average Dane image:

(Link just in case:) https://www.youtube.com/watch?v=4drWIGtEmrA

Here are the images of my face warped to the average Dane face shape and the average Dane face warped to my face shape.

Caricature

I produce a caricature of my face through extrapolating from the Dane population mean face. The formula for cross-dissolving to produce the final image from source image (my face) and target image (the Dane mean face) is \(\text{out} = (1-\alpha) * \text{source_image} + (\alpha) * \text{target_image}\). The point of extrapolation is to define \(\alpha \in (-\infty, 0) \cup (1, \infty)\) to heavily exaggerate weighting on a particular image and giving a negative weight for the other image. Here are a few examples of images (I also included the case \(\alpha=0.5\)):

Bells and Whistles

For Bells and Whistles, I morphed between friends and made a morphing music video. The face morphing video is also part of a morph between students in the class, too. I practiced skills of choosing corresponding points, producing triangulations/inverse warps, etc when producing the video. I used photoshop to add tint to some of the portrait images. Here is the video I created:

(The video link in case): https://www.youtube.com/watch?v=pi5A0DaA8T0. Song: Bruno Mars's "Count on Me".

I also created a gif from me to a male face

(The video link in case): https://www.youtube.com/watch?v=NrIoi02sBas0.

I also created a music video showing age change/growing up from 5th grade me to high school me to summer 2020 me to summer 2021 me.

(The video link in case): https://www.youtube.com/watch?v=18ogfXSBlCI. Song credits to Thomas Rhett's "Growing Up".

Summary

I learned about image morphing and transitions when producing this project. I most enjoyed learning about inverse warping and the interpolation required to get average colors in order to compute a resulting warped image, because of the matrix multiplication behind that. I also enjoyed creating a .gif for the morph sequence.

Acknowledgements

• CS194-26 Project Description
• I've obtained the Danes dataset from the instructions from the project description. Link
• I've obtained portraits of Shaina Chen and Anjali Thakrar for the classmate video.