Project 3: Face Morphing

CS 194-26: Intro. to Computer Vision & Computational Photography, Fa21

Doyoung Kim


Overview

  In this project, we have discovered how matrix multiplication is basically a change of basis, and therefore, can allow smooth transformation between different bases. And in this project, the base is a triangle, which requires 6 points to compute the transformation matrix. This transformation for triangles are also referred as Affine Transformation.

  Defining a triangle is also important. We want all the triangle meshes defining the image to be as equally spaced as possible, rather than having some triangles to be narrow and others to be wide. Equally spaced triangles will allow to well spread transformation throughout all the triangles in the image. And the Delaunay Triangle is well-known to have such property.


Part.1 Defining Correspondence

Kim Do-young
Lee Jeong-jae


  Inspired by the recent successful Netflix series, Squid Game, I decided to morph my picture to Lee Jeong-jae, the main character of the show. Before doing any transformation and morphing, we need to define corresponding points between the images I am trying to morph. I defined 50 feature points, which describes the facial features as jaws, lips, eyes, and etc, of my image and the other image that I am morphing to. Using the points, I can build the Delaunay Triangle mesh which each are the separate bases consisting the morphing module. The following



Part. 2 Computing the "Mid-way Face"

  "Mid-way Face" refers to the number of face images(phases) exist while morphing from my face to Lee Jeoung-jae's face. Each of these faces are created by computing average shape of two faces and average color of two faces. Average shape is obtained by warping one image's Triangular mesh to another Triangular mesh obtained by getting the weighted average of two facial feature points each from both images. After warping both images to the average shape, we average the color by getting the weighted average pixels of both warped images.

Kim Do-young
Mid-way Face
Lee Jeong-jae

  There are two main operations used to perform the warping. It is inverse affine transformation and interpolation. We first compute the forward transformation from the original images' Triangular mesh to the average shape's mesh using our Linear Algebra knowledge. After obtainimng the forward transformation, we compute the inverse of the transformation. Then we can use this inverse trans. to go from average shape mesh's coordinate to the original image's coordinate. After obtaining the original image's coordinate, I used scipy's function, RectBivariateSpline, to interpolate the color of the image at the computed coordinate. Then, I save this color at the coordinate of the average shape's mesh where I started the inverse transformation from.

  The reason why we obtain the color this way, rather than doing forward transformation to start with is because transformations does not gaurantee to land on the integer grid. For example, it can give us coordinates like (95.123, 55.344), and we do not have any information of what that color should be at that coordinate of the average shaped image. However, we do have color information of the original image. This is why we compute the inverse and interpolagte from the original image.

Part. 3 The Morph Sequence

  After implementing the inverser warp and the cross-dissolve, we can now implement the morphing. And having 45 different levels of images morphed, we can create an animation of morphing sequence. Some of the mid-way face images are shown in the following.

1
12
22
32
45


Morphing Sequence


Part. 4 The Mean Face

  Now, we discover what kind of characteristics human face has by looking at the mean of 240 people("Danes" dataset). I first computed the average face shape of the entire dataset by adding up all the facial feature points and dividing it by 240. Then, I morph all the faces from the dataset to this average face shape, and the following are some of the examples:

01-1m
avg. 01-1m

39-5m
avg. 39-5m

40-1m
avg. 40-1m


  Now, I took an average of all the 240 morphed images:



  At last, I morph myself to this average of average morphed images and vice-versa:

myself to avg.
avg. to myself


Part. 5 Caricature

  As all the morphing I did above are considered as interpolating process, now, I want to discover how the image would turn out by extrapolating the image. I first obtain the points that I can create the triangular mesh and extrapolae. The equation I used to obtain the extrapolating points is: cari_points = me_points + (alpha * diff), where diff = me_points - avg_points. And the following is the result by morphing to the extrapolated points:

Caricature


Bells and Whistles

  With all the techniques I have learned throughout this project, I tried to give myself a beard! And this was done by morphing myself with some of the male image from the "Danes" dataset. And following are the final results!

Beard Me Color
Beard Me
Beard Me Shape