CS 194-26: Image Manipulation and Computational Photography, Fall 2018
Cody Zeng, CS194-26-AGP
The objective of this project was to complete face morphs, from one image to another. This was achieved by marking correspondence points throughout both images, where sets of points correspond to certain features of each face (for example points for eyes, ears, nose and mouth). Then by taking the midpoint of these points for the two images, the "midpoint" points could be determined. Delaunay triangulation was used to create triangles between these correspondence points, and with the midpoint and original image points in place, all that is left is to execute an affine transformation for each triangle from the original image's triangles to the midpoint image's triangle. This process was the actual shape morphing. However, this only morphed the shape of faces, and color blending also needed to be completed. Color blending involved taking actual weighted average of pixel values and setting each pixel value to these weighted averages so that the morph from one image to another would be smooth in terms of color.
Mid-Way Face
In order to begin morphing, it is easy to think of a morph in terms of going from an image A, to a midway image that is halfway between image A and B, and then morphing from this midway image to image B. Thus, the first exercise was to create such a midway image. This midway image was created by taking the correspondence points for image A and image B, and for each point pair, finding the midpoint of the two. These new points are then triangulated, and for each triangle in the midway image, we will then attempt to morph triangles (with the same point tirangulation) from image A and image B to these midway triangles. These affine transformations were accomplished by creating two matrices, one with points in the original image, and one with points in the midpoint image (with both images having the first row being all the x coordinates and second row being all the y coordinates). Pad both of these matrices with a row of 1s on the bottom. Then invert the original image's matrix, and left multiply the midpoint image's matrix with the inverse to get the affine transformation matrix. To get the inverse affine transformation matrix, simply take the inverse of this resulting matrix. The image below describes this process.
A is the affine transformation matrix, the matrix with s's contains points from the original image, and the matrix with d's contains points for the image we want to tranform to (in our case the mid-way image). We want to inverse map from the mid-way image to the original image, so we take the inverse of A and multiply that by coordinates in the mid-way image to find the pixel coordinates in the original image that correspond to the pixel coordinates in the mid-way image. Interpolation was also used since multiplying by the inverse of A results in float values, and pixel indices have to be integers, so we interpolate (get values from neighboring pixels and add them up to get new pixel values) if we do get indices that are floats.
Stephen Curry
Lebron James
Mid-way between Lebron and Curry!
Triangulation of Curry
Completing the morph
To complete the morph, we just repeat exactly what we did for finding the mid-way image, but with different fractions (not just .5). So instead of adding .5 of image A, and .5 of image B, we also add .1 of image A to .9 of image B and .2 of image A to .8 of image B and so on and so forth. Repeat this for numerous decimal values to get a smooth transition from image A to image B.
Morph from Curry to Lebron
"Mean face" of the danes
The next task was to gather images from a public data set (I used the Danes data set) and find the "mean face" of this data set. I chose all the males in the Dane data set (30 images in total), calculated the midpoint of all these 30 images, and morphed each image to the midpoint. Then I added 1/30 of each image's pixel value to a new picture, and this new picture is now the "mean face" of the danes.
Below are all the pictures in the data set shape morphed to the midpoint.
And here is the "mean face" of the Danes!
Mean face
Morphing my face to the "mean face" for the Danes
Morphing the "mean face" of the Danes to my face
Caricatures
After obtaining the mean face, it was easy to create carcicatures of my face. By finding the difference between the correspondence points of my face and that of the Danes, and then subtracting this difference from the correspondence points of my face, I was able to create a carcicature by making my points even further from the mean! As seen, the carcicature emphasizes my smile and slants my eyes.
Subtracting deviation with an multiplier of .5
Subtracting deviation with an multiplier of .75
Bells and Whistles
For Bells and Whistles, I transformed myself into the "mean face" of an Angola woman. Doing this simply involved labeling correspondence points for both the Angola mean face and myself, and then morphing halfway between my face and the Angola mean face.
Me
Angolan Woman Mean
Me as an Angolan Woman (40% to Angolan Woman)
Morph both appearance and shape into Angolan Woman
Morph just appearance into Angolan Woman
Morph just shape into Angolan Woman
The objective of this project was to complete face morphs, from one image to another. This was achieved by marking correspondence points throughout both images, where sets of points correspond to certain features of each face (for example points for eyes, ears, nose and mouth). Then by taking the midpoint of these points for the two images, the "midpoint" points could be determined. Delaunay triangulation was used to create triangles between these correspondence points, and with the midpoint and original image points in place, all that is left is to execute an affine transformation for each triangle from the original image's triangles to the midpoint image's triangle. This process was the actual shape morphing. However, this only morphed the shape of faces, and color blending also needed to be completed. Color blending involved taking actual weighted average of pixel values and setting each pixel value to these weighted averages so that the morph from one image to another would be smooth in terms of color.
Mid-Way Face
In order to begin morphing, it is easy to think of a morph in terms of going from an image A, to a midway image that is halfway between image A and B, and then morphing from this midway image to image B. Thus, the first exercise was to create such a midway image. This midway image was created by taking the correspondence points for image A and image B, and for each point pair, finding the midpoint of the two. These new points are then triangulated, and for each triangle in the midway image, we will then attempt to morph triangles (with the same point tirangulation) from image A and image B to these midway triangles. These affine transformations were accomplished by creating two matrices, one with points in the original image, and one with points in the midpoint image (with both images having the first row being all the x coordinates and second row being all the y coordinates). Pad both of these matrices with a row of 1s on the bottom. Then invert the original image's matrix, and left multiply the midpoint image's matrix with the inverse to get the affine transformation matrix. To get the inverse affine transformation matrix, simply take the inverse of this resulting matrix. The image below describes this process.
A is the affine transformation matrix, the matrix with s's contains points from the original image, and the matrix with d's contains points for the image we want to tranform to (in our case the mid-way image). We want to inverse map from the mid-way image to the original image, so we take the inverse of A and multiply that by coordinates in the mid-way image to find the pixel coordinates in the original image that correspond to the pixel coordinates in the mid-way image. Interpolation was also used since multiplying by the inverse of A results in float values, and pixel indices have to be integers, so we interpolate (get values from neighboring pixels and add them up to get new pixel values) if we do get indices that are floats.
|
|
|
|
|
|
Completing the morph
To complete the morph, we just repeat exactly what we did for finding the mid-way image, but with different fractions (not just .5). So instead of adding .5 of image A, and .5 of image B, we also add .1 of image A to .9 of image B and .2 of image A to .8 of image B and so on and so forth. Repeat this for numerous decimal values to get a smooth transition from image A to image B.
|
|
"Mean face" of the danes
The next task was to gather images from a public data set (I used the Danes data set) and find the "mean face" of this data set. I chose all the males in the Dane data set (30 images in total), calculated the midpoint of all these 30 images, and morphed each image to the midpoint. Then I added 1/30 of each image's pixel value to a new picture, and this new picture is now the "mean face" of the danes.
Below are all the pictures in the data set shape morphed to the midpoint.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
And here is the "mean face" of the Danes!
|
|
|
|
|
Caricatures
After obtaining the mean face, it was easy to create carcicatures of my face. By finding the difference between the correspondence points of my face and that of the Danes, and then subtracting this difference from the correspondence points of my face, I was able to create a carcicature by making my points even further from the mean! As seen, the carcicature emphasizes my smile and slants my eyes.
|
|
|
Bells and Whistles
For Bells and Whistles, I transformed myself into the "mean face" of an Angola woman. Doing this simply involved labeling correspondence points for both the Angola mean face and myself, and then morphing halfway between my face and the Angola mean face.
|
|
|
|
|
|
|
|