Ajay Ramesh, October 2, 2018

Fun with Frequencies and Gradients

Overview

In this project we explored how the frequency domain and gradient domain can help us manipulate images. We implemented parts of the papers Oliva, Torralba, Schyns, 2006, Burt, Adelson, 1983, and Perez, Gangnet, Blake, 2003.

1.1 Warmup

We can sharpen an image by adding back a scale factor of its high frequencies, back to the original image. In this example, I arbitrarily chose the scale factor of 1.5.

1.2 Hybrid Images

The Hybrid Images paper uses the fact that the human eye perceives high frequencies at close distances and low frequencies at long distances to create images that appear to change when viewed at different distances. In order to construct these images, we apply a low pass filter to one image, a high pass filter to another image, and display the average of the two images. For these examples, I've applied the algorithm to each of the R, G, B color planes to construct color hybrid images. The results are heavily dependent on how much patience the "artist" has to create a parameterize a good blend. As you can see, I'm no artist.

Derek & Nutmeg Young Anakin & Darth Vader Failure Case I believe this failed because the interior of the flowers are very low frequency. So, when we apply a high pass filter to the same flower, at a later point in time, only the edges are preserved - making the second flower almost invisible. Fourier Analysis of Derek & Cat

1.3 Gaussian and Laplacian Stacks

In this project we try to decompose a hybrid image into its low and high frequencies by constructing the Gaussian and Laplacian stacks respectively.

Painting by Salvador Dali
Abraham Lincoln appears in low frequencies and Salvador Dali's wife appears in high frequencies. Mona Lisa
Historians believe that Mona Lisa's smile changes depending on how you look at her. Could it be that Da Vinci painted one smile with low frequencies, and another smile with high frequencies? Derek & Nutmeg
Derek appears in low frequencies and Nutmeg appears in high frequencies.

1.4 Multiresolution Blending (a.k.a. the oraple!)

In the Spline paper, we learn how to create "seamless" image blending by computing weighted sums of two image's Laplacian stacks. Each image is weighted by the Guassian of a binary mask image, which chooses which "elements" of one image should appear in another image.




Room and Chair from another Room
This is an example of how the algorithm performed with an irregular mask of a sofa from a different room. We will later see how gradient domain blending does much better on the same problem.

2.1 Gradient Domain Fushion

In Part 2, we are concerned with reconstructing images by solving for their gradients alone. Let's take a look at the governing optimization problem.

We are solving for the vector v which minimizes the above cost function. v contains the pixels of the image we are trying to reconstruct. We can find v by solving the least squares equation Ax = b. Our "known" vector b contains the known gradients of the image we are trying to reconstruct. The coefficient matrix A defines what it means to take a discrete gradient in the x and y dimensions. In part 2.1, we are concerned with the first term of the summation, where we are trying to reconstruct an input image in its entirety. Here's my reconstruction of the example image.

2.2 Poisson Blending

In this part we are solving the same optimization problem, in a different context. We are trying to cut out a mask of one image called the "source", and insert it into another image called the "target". However, simply cutting and pasting a rough mask of one image, onto another, leads to a pretty obvious "shop-job". In the second summation in the cost function, we are trying to ensure that the gradient of a pixel in the source image mask and a bordering pixel in the target image, is similar to the gradient of a pixel in the source image mask and a bordering pixel in the source image, outside of the mask. This technique allows us to cut a rather coarse mask of the source image, and still see a convincing result.

Fish + Open Water

Buzzfeed Automation

Flying Space Squirrel

Penguin + Mountain

Failure Case
Poisson blending attempts to preserve gradients but not necessarily intensities. This means that the color of the source image will change in an attempt to minimize the least squares cost function. As a result, this Star Destroyer looks a lot less threatening.

Poisson Blending vs. Laplacian Stacks

In this example, we compare how the Laplacian stack algorithm blended the irregular mask around the sofa in the source image and how the Poisson Blending algorithm blended the irregular mask around the sofa in the source image. Poisson Blending Result

Laplacian Stack Blending Result

Clearly, the poisson blending did a lot better. The Laplacian blending is very subjective to the quality of the user defined mask. The Poisson blending algorithm takes this into account because of the second term in the cost function, the one that minimizes the difference in border gradients. Additionally, the Laplacian stack algorithm distorted the colors of the source image severely, while the Poisson blending algorithm distorted the source colors only slightly. This is likely due to how I parameterized the Laplacian stack, since only the high frequencies of the sofa seem to be accentuated.