Project 3: Fun with Frequencies and Gradients!

Introduction

Blending between two images is a difficult process in image processing. A user can specify feather blends, but these are difficult to tweak because too large of a window can lead to ghosting, and too small of a window can lead to seams. In this project, we investigate two automatic blending options: Multiresolution Blending and Poisson Blending in the Gradient Domain.

Unsharp Mask Filter

Images can be enhanced by applying an unsharp mask filter. This filter adds the high frequencies in an image back into the original, scaled by some amount a. This can be implemented by subtracting a gaussian blurred image from the original image, which is also called the Laplacian of Gaussian.

We can tune the output quality by changing the values a, which determines the intensity of the high frequencies we add back, and sigma which determines which high frequencies we choose. To handle the case for color images, we apply this filter individually to each channel.

Parks and Recreation

Here we have applied the unsharp mask filter to an image of a park, with a variety of a and sigma values.

Hybrid Images

Humans can perceive different frequencies at different ranges. Interesting images can be created by blending the low frequencies of one image with the high freqiencies of another image, and then added together. The high frequencies are visible at close range, and the low frequencies are visible at long range.

When viewing the images, it is best seen by viewing them from close up, then moving far away from your screen and looking at them again!

Dwayne "The Rock" Johnson

We sought to blend an image of Dwayne "The Rock" Johnson as happy and sad. One of the hardest parts of this project was trying to find a picture of Dwayne "The Rock" Johnson sad. He doesn't ever seem to be sad. We settled for perturbed instead.

We can understand what is happening in this blending by examining the frequency domain representations of the image (seen below). From left to right: The first image is the frequency analysis of the happy picture. After applying a low pass filter, we see that only the low frequencies remain (those near the axes). The third image is the perturbed image, and we can see that after applying a high pass filter, only the frequencies far away from the axes remain. The frequency analysis of the hybrid image is a combination of the 2nd and 4th images.

We will now see a few more examples of hybrid images

Shanghai, then and now

We found this fascinating image of Shanghai taken 20 years apart. We created a hybrid image of the two so that you can see the economy of Shanghai bloom before your eyes! Color was used in this image to enhance the effect so you could see how green it used to be.

Doggos

Doggos are the best animal. Therefore, two doggos should produce the best hybrid image ever created. We can see in this example that the hybrid image doesnt work as well, because the bodies of these two doggos are quite a bit different, so you can still see the low frequency doggo clearly behind the high frequency doggo.

Gaussian and Laplacian Stacks

Laplacian stacks are instrumental in performing multiresolution blending.

Laplacian stacks are created by first creating Gaussian stacks. The differences between successive layers of the Gaussian stack are the layers of the Laplacian stack. Each layer represents a range of frequencies in the image (also called bandpass images). The lowest level of the Gaussian stack is added to make sure we can reconstruct the image by having the average intensity of the image that is not represented by the stack.

Laplacian stacks can reveal interesting information about images that contain low and high frequency information. We provide 3 examples.

Gala Contemplating the Mediterranean Sea which at Twenty Meters Becomes the Portrait of Abraham Lincoln

As the name implies, this painting by Salvador Dali intentionally contains imagery at low and high frequencies so that the image changes its appearance depending on where the viewer is standing. This is remarkably similar to our hybrid images! Looking at the Laplacian stack shows Gaia is very clear at high frequencies, but at lower frequencies an image of Abraham Lincoln emerges.

Mona Lisa

The Mona Lisa by Leonardo da Vinci is a peculiar painting of a woman whose expression appears to change, depending on where you view the painting from. Perhaps this painting contains details at low and high frequencies! We analyze this painting using Laplacian stacks to see if we can observe this phenomenon. The analysis of the stacks is left as an exercise to the reader.

Dwayne "The Rock" Johnson, Revisited

We return to our generated hybrid image of Dwayne "The Rock" Johnson. Two of the many expressions of this fine man can be seen by looking at the Laplacian stacks.

Multiresolution Blending

Multiresolution Blending is the process of blending two images by blending them at each layer of their Laplacian stack. Blending boundaries can be specified with a mask, that is blurred appropriately at each level of the stack.

Oraple

An apple and an orange can be blended to appear like one strange fruit.

Nick the Eagle

Irregular masks can be used to blend images into others with more details. Here we replace the face of this bald eagle with that of the great actor Nick Cage. This doesn't work very well, however, because his skin color is different than the feather colors of the eagle. We address this issue later.

Face/Off

Face/Off is a 90's science fiction action film directed by John Woo, where the characters played by John Travolta and Nick Cage switch faces for some reason. I bet they didn't think about blending the two faces together, though! (We did)

We can look at the Laplacian stack of this process to get a better idea of what is happening. Each layer of the stack for each image is blended by the mask, and then added together. These steps are display below.

Gradient Domain Fusion

Gradient-domain processing is a technique for blending images seamlessly by blending images in their gradient space, which is the 2D partial derivatives between neighboring pixels. Merging images by copying their pixels directly creates very noticeable seams, and images blended with Multires blending can sometimes not look quite right (as seen before). Instead we solve this by keep track of the gradient inside of a masked area, while also trying to maintain the intensities at the edge of the masked area. This can be formulated as a linear least squares problem.

The solution to this problem is the blended image, using a technique called Poisson Blending.

Toy Problem

Before we perform full Poisson blending on an image, we examine a simpler problem that has no masks. We load an image and compute the 2D gradients of it, setting up a constraint matrix to relate all of the gradients to a target pixel. Then we lock the top left pixel of the target image to that of the source image. There is only one solution to the equation, which is the original image. The results of this are demonstrated below.

Poisson Blending

The same technique is now applied to images with masked areas. At the borders, we sample the source image so that the colors of the masked area are changed to match the borders, creating seamless blends. As an optimization, we implement this using sparse matrices, and only compute the image for the area around the mask, as most of the image is just copied directly (identity operation). This speeds up computation significantly.

Mona Nicholas

We return to our friends, the Mona Lisa and Nick Cage. Copying the pixels directly create a noticeable seam, despite the fact that they have relatively similar skin color. We attempt to blend Nick's face onto Mona Lisa's using Poisson blending techniques. This works fairly well, although it is creepy looking.

Sea-attle

Some say that stingrays are the birds of the sea. Seattle is near the sea. Lets see if we can imagine what Seattle might look like with a flying stringray population. This new city would be known as Sea-attle.

Pudelhund

Poodles are the best doggos, especially baby red poodles. Poodle comes from the German word Pudelhund, which basically means water dog. I don't know of any poodles that live under water, but lets see if we can create an image of how majestic that would be.

This is an example of Poisson blending not working. The green color around the dog gets changed to blue, which ends up changing the color of the dog to purple. Also the borders dont blend in very well, because the grass around the dog is quite noisy, and the water is very smooth. It struggles to create a smooth blend.

The Return of Nick the Eagle

Earlier, we attempted to place the face of Nick Cage onto a bald eagle, with dubious results. We try again with Poisson blending, and find that the result is much more believable.

Here we compare the results of Multires blending and Poisson blending. In this case, Poisson blending works better because of the difference in color. However, the difference between Nick's skin and the eagle's feathers is not so strong that it messes up the colors in the rest of the image, like it did with the poodle.

At one point in the process of developing this, we accidentally switched the source images, We leave you with the hideous monster that this produced.

Reflections

This project was quite difficult, although once it started working, it was rather fun coming up with werid combinations of images to blend. The key insight to this project came during the lecture that our TA Jun-Yan Zhu gave us, showing how to properly setup the least squares problem. Once that was figured out, proper usage of sparse matrix libraries were also critical to the success of the project.