This project was composed of constructing the core functionality behind morphing faces, and then applying to a variety of creative ideas. Our morphs were based upon both warping (on the facial structure) and cross-dissolving (on the facial coloration). I detail the approach I used to generate such functionality, as well as the results found from using it to play with an analyze pictures and datasets.
I used human-generated input to tag face image pairs with correspondences on important features. To prepare for structural warping, I interpolated an intermediate point set structure, and found a suitable triangulation for it.
Finding satisfying "average objects" requires human discretion; I used Matlab's cpselect tool to label correspondences on image pairs, at important features such as the location of eyes. I then found an intermediate shape between these two objects (in preparation for the next section, where I find a mid-way face) by interpolating via averaging on the point sets. I then found a Delaunay triangulation on this intermediate point set, which is optimal for our purposes, as it avoids skinny triangles which are likely to result in unsightly artifacts.
None for this section.
As I found out later in the section, the ultimate feasibility of our approach relies heavily on our human-generated preprocessing. Choosing aligned images, and cleverly picking correspondences is key to getting convincing morphs. Missteps in alignment or lack of important correspondences can often lead to some insightly artifacts in the resulting morphs.
I found the "mid-way" face between two face images by combining both structural face warping together with coloration cross-dissolving.
In order to perform structural warping, I used inverse warping onto an averaged correspondence point set, interpolating from the two source images. To do this, I warped the images one region at a time, using triangles from the suitable Delaunay triangulation found in the last section on an averaged point set. To inverse warp, I needed to be able to map a pixel on the final image to its respective locations on the original source images; thus, I needed to compute the affine transformations from source to target for each triangle region. In class we learned that affine transformations, under a homogenous coordinate system, have 6 parameters. Because I knew the original and final positions of the triangle vertices, I had just enough information to solve for these parameters, if I set up a linear system. This approach ultimately allowed me to inverse map pixels from both source images onto the final target; all that was left to do was to average the color values from these mapped source pixels to set as the pixel value in the target location (cross-dissolving).
Source image 1 (Lonzo Ball):
Source image 2 (Lavar Ball):
Mid-way face:
Of notable character in the mid-way face is that the faces do not quite match up, leading to some ghostly artifacts. The reason for this was touched upon earlier; because the two source images differ reasonably in alignment, a half-way morph from one picture to the other will not fully ameliorate this misalignment. The mid-way face captured is more accurately a depiction of a morphing in motion; our simple linear interpolation technique simply cannot avoid these artifacts. As we will see later, however, these artifacts do not lead to much trouble when we actually animate the morphs, at a reasonable rate of frames per second.
I generated a series of morphs between two images, progressively morphing more heavily towards the second. I used these frames to generate a visualization of a morph through time as an animated GIF.
I applied the functionality constructed in the previous sections to morph from a source image to a target image. I parameterized the process based upon how much to warp the structure from the source towards the target, as well as how much to cross-dissolve the colors, from the source towards the target. By progressively increasing this morphing paramter in a series of frames, I was able to generate an animated GIF from the resulting morph frames.
I used the same morphing fraction for both warping and cross-dissolving, increasingly constantly with time from the source image to the target image.
Lonzo-nope!-Lavar animated GIF:
Note that the ghostly artifacts seen in the individual intermediate morph frames become much more fluid and invisible when animated in the final animation. Nice!
I downloaded a dataset of Danish faces, and found the average face of the dataset using uniform averaging and the warp/cross-dissolve functionality implemented in previous sections. I played around with this average face, warping each face in the dataset as well as my own into the geometry of this average, as well as warping this average into the geometry of my face.
There was minimal additional infrastructure needed to implement this section. I downloaded the annotate face images, and generated an average face using the same functionality described earlier, uniformly weighting structure and color data across the face images. I warped each face (plus my own face) into the geometry of this average face, and detail some intermediate morphs during this process in the results below. Finally, I morphed this average face into the geometry of my face.
Average Danish face (includes mostly males, some females):
A Dane partially warped into the average geometry (original and warp):
Another Dane partially warped into the average geometry (original and warp):
My face:
The average Dane face partially warped into my face's geometry:
My face partially warped into the average Dane's geometry:
This section was a case study in using our face morphing functionality to explore a dataset. There are a few things to note. First of all, the average picture contains many visible ghostly silhouettes - the reason for this is that our dataset of Danish faces contained only 36 images. We can see a shadowy region around where long hair would appear, which corresponds to the fact that some of the Danes were female and had long hair. Nonetheless, a clear, Danish schema emerges from this average face, and our technique was quite successful. Morphing individual Danish faces into this average helps reveal the individual special characteristics for a given individual. Noting my face's geometry compared to that of the Dane's - it can be clearly seen that the Danish facial schema favors a smaller sort of face, in the warps that somehow involve my face. Interesting!
Regarding failure cases: Notice that the images are snapshots of partial warps, and appear to be quite strange. This isn't because of an actual systematic issue. Recalling from earlier, this is because I captured the morph midway through its process, where correspondences don't quite line up so we get some bizarre artifacts. Another note - the sort of annotating provided with the Danish dataset was not quite optimal for our purposes, as it missed some important correspondences, such as many on the upper half of the face. Dane's also were not quite consistently aligned, with some faces being in a lower relative position in the images, or at varying angles. These all lead to the weird sort of oddities that are manifested when we snapshot partial warps.
I used the average Danish facial schema from the previous schema to extrapolate my face warped towards a caricaturized fashion opposing the average Danish face.
I used the same morphing functionality to warp my face to a specific caricaturized geometry, this time subtracting contributions from the average Danish face.
Average Danish face:
My face:
My face partially warped to a geometry opposite of the average Dane:
In the previous sections, we linearly interpolated warping/cross-dissolving contributions from images in question, whereas in this section, we took a more tantalizing jump to extrapolating away from the contributions of one image. As described earlier, the Danish face seems to be more inwardly oriented than my own. In this anti-Danish caricurazation, we can see, as expected, that my face spreads out, among other things. Some oddities that occur in the snapshot of the partial warp have the same explanations as explained earlier.
I download an image of the average female face that I found online, and used the techniques described above to warp, cross-dissolve, and morph my male friend's face towards the image, independently.
This section was just playing and exploring around; I used the same morphing technique earlier to warp, cross-dissolve, and morph my friend's face into the average female face. In a sense, I generated different candidates for faces that represent what he would look like if he was a female.
For the snapshot of the partial warp (only), I used:
For the snapshot of the partial cross-dissolve, I used:
For the snapshot of the partial morph, I used:
Average female face:
Sudeep Dasari:
Sudeep Dasari warped to the average female:
Sudeep Dasari cross-dissolved to the average female:
Sudeep Dasari morphed to the average female:
Combining both shape warping and color cross-dissolving clearly gives the best results, as this section shows. It is clear in this morph that Sudeep has taken on some feminine characteristics! Some oddities that appear in the partial morph snapshot have the same underlying causes as explained in previous sections.
I made a music video where I morph my suitemates...
I used the same approach as the morph sequence section previously discussed, except added more images to morph between.
Watch the video!