Fun with Frequencies and Gradients!

Student Name: Divyansh Agarwal

Course: CS 194-26 (Computational Photography)

Frequency Domain

Part 1.1: Warmup

Here, I use the unsharp masking technique to sharpen images. The main idea behind this is to use a low pass filter such as a Gaussian Filter to get a blurry version of an image. Then I subtract this filtered image from the original image, to accentuate the edges and thus make the image sharper.

I used a Gaussian filter with a sigma of 10, and alpha of 2 (i.e., I took a linear combination of the original and filtered image, where I gave the original image a weight of 2, and the filtered image a weight of -1). This gave me a much sharper image, as shown below:

Snow Original Image

Forest Sharpened Image

Part 1.2: Hybrid Images

Here, the aim is to create hybrids of two images, by low pass filtering and high pass filtering one image respectively, such that high frequency dominates when the hybrid is viewed from a short distance, and the low frequency dominates when the hybrid is viewed from a long distance. I used the Gaussian Filter, a low pass filter, to extract low frequencies from one of the images. To high pass filter the other image, I passed it through the Gaussian Filter, and subtract it from the Impulse Filter. I experiment with values of Sigma for the Gaussian Filter used for each image, to determine the best combination for each pair of images.

Snow Cheetah (Highpass, Sigma = 10)

Forest Messi (Lowpass, Sigma = 5)

Mountains Blended Cheetah and Messi

The FFT are as follows:

Snow FFT of Cheetah

Forest FFT of Messi

Forest FFT of Highpass Filtered Image

Snow FFT of Lowpass Filtered Image

Forest FFT of Hybrid Image

Additional Images

Snow Joker (Highpass, Sigma = 25)

Forest Nadal (Lowpass, Sigma = 7)

Mountains Blended Joker and Nadal

Snow Adult Harry Potter (Highpass, Sigma = 5)

Forest Child Harry Potter (Lowpass, Sigma = 5)

Mountains Blended Child and Adult Harry Potter

Bells and Whistles (Part 1.2)

Snow Blended Cheetah and Messi

Forest Blended Joker and Nadal

Mountains Blended Child and Adult Harry

Part 1.3: Laplacian and Gaussian Stacks

Here, the goal is to create Laplacian and Gaussian Stacks of classic paintings and one of my hybrid images. A Gaussian Stack is obtained by repeatedly applying a Gaussian Filter to our image. As we progress further along the stack, we increase the sigma, and we obtain the image at level i of the stack by applying the filter to the image at level i - 1 of the stack (with the image at level 0 being the original image itself). A Laplacian Stack is obtained by taking at each level the difference between the image at the corresponding level in the Gaussian Stack and the image at the next level in the Gaussian Stack.

Salvador Dali Painting

Gaussian Stack

Snow Original Image

Snow Height = 1, Sigma = 1

Forest Height = 2, Sigma = 2

Mountains Height = 3, Sigma = 4

MountainsHeight = 4, Sigma = 8

Placeholder

Laplacian Stack

Snow Original Image

Snow Height = 1 (Laplacian)

Forest Height = 2 (Laplacian)

Mountains Height = 3 (Laplacian)

Mountains Height = 4 (Laplacian)

Mona Lisa Painting

Mona Lisa Painting

Gaussian Stack

Snow Original Image

Snow Height = 1, Sigma = 1

Forest Height = 2, Sigma = 2

Mountains Height = 3, Sigma = 4

Mountains Height = 4, Sigma = 8

Laplacian Stack

Snow Original Image

Snow Height = 1 (Laplacian)

Forest Height = 2 (Laplacian)

Mountains Height = 3 (Laplacian)

Mountains Height = 4 (Laplacian)

Cheetah and Messi Hybrid

Gaussian Stack

Snow Original Image

Snow Height = 1, Sigma = 1

Forest Height = 2, Sigma = 2

Mountains Height = 3, Sigma = 4

Mountains Height = 4, Sigma = 8

Laplacian Stack

Snow Original Image

Snow Height = 1 (Laplacian)

Forest Height = 2 (Laplacian)

Mountains Height = 3 (Laplacian)

Mountains Height = 4 (Laplacian)

Part 1.4: Multi-Resolution Blending

Part 1.4: Multi-Resolution Blending

We want to seamlessly blend two pictures together, and to achieve this goal, we can use an approach proposed by Burt and Adelson in 1983. In this approach, we first define a mask, and compute the Gaussian Stack of the mask. We then construct a Laplacian stack for our blended image by first computing the Laplacian Stack of both of our images, and then at each level, we perform a linear interpolation of the corresponding images in Laplacian stacks of both images, with the weight in the interpolation being the corresponding image in the Gaussian Stack of the mask. The final blended image is obtained by adding together all images in our Laplacian Stack for the blended image.

I found that this approached worked well when at each level, the sigma for the Gaussian Stack of the source mask was much higher than the corresponding sigma for Laplacian stacks of each images. I created Gaussian and Laplacian stacks of height 4 over here. For Gaussian Stack of the Mask, I used sigmas of [1, 4, 9, 16] for levels [1,2, 3, 4] respectively, and for the Gaussian Stack of each image (used to get Laplacian Stack of each image), I used sigmas of 0.5, 1, 2, 4] for levels [1,2, 3, 4] (I set the last level of the Laplacian stack to be the same as the last level of the Gaussian stack and found this to give me good results in this case).

Snow Apple

Forest Orange

Mountains Blended Orange and Apple

Additional Examples: Irregular Masks

Snow Eye

Forest Forehead with Hair

Mountains Blended Eye and Hair

Snow Serena Williams

Forest Serena's Coach

Mountains Serena on Coach's Forehead

Bells and Whistles (Part 1.4)

Snow Blended Eye and Hair

Forest Serena on Coach's Forehead

Mountains Blended Orange and Apple

Part 2: Gradient Domain Processing

The goal of this part was to use gradient domain processing techniques to seamlessly blend an object from one source image into another target image. This is done via Poisson Blending, which maintains the the gradients of the portion of the source image being pasted onto the target image. At the same time, we force the gradient of a pixel at the border of the source image mask and a bordering pixel in the target image outside the mask to be similar to the gradient of the corresponding pixel at the border of the source image mask and a bordering pixel outside of the mask in the source image.

Part 2.1: Toy Problem

In the quest to solve the Poisson Blending problem, we first solve a simpler Toy Problem. In this problem, we try to reconstruct a given image, by making sure that the left most pixels of the reconstructed image and the original image are the same, and by also preserving the x and y gradients of the source image. Thus, we create a linear system of equations, and use Least Squares to obtain the reconstructed image, by minimizing the difference in the x and y gradients of the source and reconstructed image.

I was able to successfully reconstruct the image, as shown below:

Snow

Original Image

Forest

Reconstructed Image

Part 2.2: Poisson Blending

In this part, I implement Poisson Blending. As described earlier, this approach maintains the the gradients of the portion of the source image being pasted onto the target image, and at the same time, we force the gradient of a pixel at the border of the source image mask and a bordering pixel in the target image outside the mask to be similar to the gradient of the corresponding pixel at the border of the source image mask and a bordering pixel outside of the mask in the source image - doing so helps us blend the images seamlessly. Thus, it is an extension of the Toy Problem above.

The results look very natural, as shown below:

Snow Source Image: Helicopter

Forest Target Image: Sky

Snow Naive Pasting

Forest Poisson Blended Image

Additional Examples

Snow Source Image: Red Star

Forest Target Image: New York

Snow Naive Pasting

Forest Poisson Blended Image

Failure Image

The following image failed because the texture of the grass field in the target image was very different from the texture of the grass field in the source image, as the texture of the grass changes very rapidly in the target image, compared to the source image which has a more consistent texture throughout. In other words, the backgrounds of the source image and target image have very different frequencies, and this affects the ability of Poisson Blending to produce natural looking images.

Snow Source Image: Cristiano Ronaldo

Forest Target Image: Textured Grass Field

Snow Naive Pasting

Forest Poisson Blended Image

Laplacian vs Poisson Blending

I tried Poisson Blending to blend the eye and the hair like I attempted in Part 1.4. Below, we can see that Poisson Blending works better in blending the eye with the hair as the part surrounding the eye from the source image matches the color and texture of the surrounding the hair, whereas with Laplacian Stack Blending, the blend does not seem natural as the part surrounding the eye from the source image does not match the color and texture of the surrounding the hair. However, Poisson Blending changes the color of the eye. Laplacian Blending might work better if we blended two objects with more similar backgrounds, as it would not affect the intensity of the pixels in the part of the source image copied over, and would thus retain the original color.

Snow Source Image: Eye

Forest Target Image: Forehead with Hair

Snow Laplacian Blending

Forest Poisson Blending