The project explored the concepts of face morphing by defining correspondence points that should match between images, forming a triangle mesh using those correspondence points which are efficiently created using Delaunay triangulation, computing an affine transformation for intermediary (or averaged) triangle shapes using the recomputed, intermediary correspondence points which are then used to backwards sample pixels (inverse warping) from the original images to compute the warped image before cross-dissolving the two images by some weight (alpha and 1-alpha) to produce the final morphed image.
The two-step process of image warping and cross-dissolving, in other words morphing the shape and appearance of the images separately, can produce interesting results such as an animated sequence of morphs that show a smooth transformation from one individual’s face to another, finding the average looking face of a population, or even changing the gender, age, or ethnicity of a person’s face among many other interesting results. The image warping is done first prior to cross-dissolving since morphing the geometry of the image is essential to allowing for a linear interpolation of pixel values used in cross dissolving. Without changing the facial geometry, the two images are not guaranteed to exist in a linear subspace which is evidently visible in one of my results in the Bells and Whistles section which should cross-dissolving two images without first changing their geometry. The faces do not align well and so the cross-dissolved image does not look convincing. The details of producing a morphed image are further explored in the following subsections.
A crucial step to warping the geometry of a face is to define correspondence points on each image that we want to warp the geometry. I do this by using pyplot’s ginput method which provides a GUI for each image that is called with the method and correspondence points are selected by clicking on pixels in the image. It is important that the order in which correspondence points are selected in the image remain the same between images otherwise the indices returned after triangulation will not correspond to the correct triangle regions of each image resulting in a warped geometry that fails to align properly.
For the images used in this section, I defined 101 correspondence points on each image and ensured that the correspondence points on each image were selected in the same order. For ease of processing I convert the list of tuple coordinates returned by the ginput method for each image into a list of lists, where each coordinate pair is its own list of the format [x,y]. I then called scipy’s Delaunay method passing in as arguments the correspondence points for each image to form two triangulations, one for each image. Below are the two images with the selected correspondence points and the computed triangulation.
In order to compute the mid-way face, and any intermediary geometry between two images, we follow the general procedure of computing the intermediary correspondence points, finding the affine transformation matrices from the intermediary state to each respective image’s correspondence points space in order to inverse sample the pixel values from the original images and compute the affine warped intermediary image, and finally cross dissolve the pixel values from the warped images to produce the final morph. This general process is applied to each color channel of both images and then the color channels are recombined to produce the final color image.
This section specifically computes the mid-way face, or when alpha is set to equal 0.5. I first begin by computing the average correspondence points which is done by linearly interpolating the x and y positions for each respective correspondence point in both images with an alpha, or weight, of 0.5 (this is one reason why selecting correspondence points in the same order for both images was important). Doing so gives us the correspondence points that are an average between the two images. The averaged correspondence points overlayed onto the original images can be viewed below.
I then compute the Delaunay triangulation on the average correspondence points which will produce the following triangle meshes if overlayed onto the original images.
The next step is to compute two separate sets of affine transformation matrices which convert coordinates for each triangle in the averaged coordinates image to either the first or second image (hence why two sets of transformation matrices is required). Doing so will enable inverse sampling of the original images to generate the average image with the warped geometry for both images. The transformation matrix for each triangle is computed using the following general logic.
T * Imid = Isrc
T * Imid * I-1mid = Isrc * I-1mid
T = Isrc * I-1mid
Where T is the affine transformation matrix we are trying to find, Imid is the matrix that holds the three coordinates forming a triangle in the averaged coordinates, and Isrc is the matrix that holds the three corresponding coordinates from the averaged triangle but that are from the original image correspondence points (another reason why selecting correspondence points in the same order for both images was important). Performing these operations for every triangle in the mesh, we can compute the affine transformation matrices for every triangle in the averaged image and perform inverse sampling on the original images to compute the warped geometry image. Below are what the warped geometries of the faces look like prior to cross-dissolving.
The final step is to cross-dissolve the two warped images. In this case, since we are computing the midway face, I also use an alpha of 0.5 for the cross-dissolve operation. The cross-dissolve operation is really a linear interpolation of pixel values between the two images and follows the following formula.
The results of the computed midway face along with the original and warped images are displayed below. The cross-dissolved midway face is the center image labeled morphed.
To produce the morph sequence, the same process described in the previous section was used for image warping and cross-dissolving; however, the alpha term was varied such that a more linear progression of the morph from one image to the other would be produced. My morph consists of a total of 45 frames which are played over a duration of 1.5 seconds (30 frames per second). The beginning frame uses an alpha of 0 and so displays the original image of one of the portraits and the last frame uses an alpha of 1 and so displays the original image of the other portrait. The intermediary frames are a linear interpolation of alpha between 0 and 1 such that the step of incrementing alpha for each new frame is the same size for every frame. In this case alpha controls both the weight of correspondence points from each image when computing the intermediary coordinates for the affine warp as well as the cross-dissolve weights for both warped images. The produced animated GIF is viewable below.
To compute the mean face of a population, I used the images and correspondence coordinates provided in the Danes IMM Face Database. The mean face of the population was computed using the first pose for each of the 40 photographed subjects. I also wrote a script that would read each picture’s asf file which contained the correspondence points identified by the researchers who created the database. I only added four additional correspondence points to each image which included the four corners of the image. That way when performing the triangulation and image warping, the background would also be displayed in the final image result. A sample of the correspondence coordinates and triangulation on a couple of the images is viewable below.
I then proceeded to find the average correspondence coordinate positions across all the images. I did this by summing the x and y position values for each of the correspondence points that were associated with the same point in each image and then divided by the number of images, which was forty. I then computed the Delaunay triangulation associated with the averaged correspondence points. A set of transformation matrices from the averaged coordinates to the original coordinates for each image was then computed using the same algorithm as previously described. From there, the affine transformation matrices were applied to each triangle within each image to produce the warped image for each of the forty portraits. Below are some examples of the original and average warped output for some of the portraits.
The final step of the process for producing the mean face of this population was to sum the pixel vales across all of the warped images and then divide each of the pixel values by the number of images. This provides the average pixel value across all of the warped images. The result of the computed mean face of the population is viewable below.
I then also proceeded to warp my face to the average face and the average face to may face geometry. I began by defining the correspondence points on the two images which can be seen below.
Afterwards, for warping my face to the average face geometry, I computed the Delaunay triangulation on the average face correspondence points and then found the affine transformation matrices from the average face to my face so that I could sample my original image and produce the warped image of my face into the average facial geometry. Likewise, for warping the average face to my face’s geometry, I computed the Delaunay triangulation on my face’s correspondence points and then found the affine transformation matrices from my face to the average face so that I could sample the average face image and produce the warped image of the average face into my face’s geometry. The results are viewable below. Some of the distortion in the images are likely due to the different pose of our faces in the original images, whereas the average face has the chin and head pointed more downward and it looks like the image was taken from a further away distance, my image was taken very close to my face and I was also looking a bit upward.
In order to produce a caricature of my face I extrapolated information from the population mean that was previously computed. What this means is that instead of using an alpha value within the range [0, 1] like I previously did, I will use values of alpha outside of this range. Intuitively, this means warping my face beyond the geometry of the population mean for alpha values greater than 1 and exacerbating my geometry features for alpha values less than 0. Below you can visualize the extrapolated caricatures on my face for alpha values ranging [-1, 2] with some alpha values within the range [0, 1] for reference of what the warp would look like if we were not extrapolating from the geometry. The most apparent observation is that extrapolating beyond an image’s geometry can result in severe visual distortion which is to be expected since we are making a prediction on what the geometry should look like while not actually having the geometry data to back-it, which is why it is called an extrapolation.
For this section, I decided to experiment with changing my face’s gender to look more like the average female face, changing my ethnicity to appear more Asian, and also make my face look older. After much experimentation, I settled on an alpha value of 0.8 used for warping my face geometry into the reference image (female, Asian, or old). Higher alpha values produced a bit more distortion in my facial geometry which did not look as visually appealing on the final cross-dissolved image. I do not use the same alpha value of 0.8 when cross dissolving. I instead use an alpha value of 0.5. I settled on this alpha value since it best retained my facial characteristics so that when looking at the final morphed image, I could still recognize myself in it but visual appearances from the reference images were also noticeable, such as some wrinkles with the old image and different skin color complexions. Below are the intermediary steps, such as the reference images used, correspondence points selected, triangulation, my warped facial geometry, and the final morphed image after cross-dissolving. Prior to this; however, I applied a cross-dissolve immediately on my image and each reference image to see what the results were without warping the facial geometry which are displayed below.
As mentioned in the overview section, these images demonstrate the necessity of warping the facial geometry in order to make the linear interpolation of pixel values possible in a linear subspace. Otherwise, the facial features rarely align well and the resulting image is not very convincing. Now below are each of the intermediary steps taken towards producing the final morphed images.
The reference images used:
Defining the correspondence points on the images:
Showing the triangulation on the images:
Warping my facial geometry to that of each reference image:
The final morphed image results:
