Project 3

Fun with Frequencies and Gradients!

Part 1.1: Warmup

I picked this blurry picture of my brother and "sharpened" it using the unsharp masking technique covered in class, where the high frequences of an image are added back to the original image scaled by a constant alpha.

Blurry, original image

a = 0.25, sigma = 5

a = 0.5, sigma = 5

Part 1.2: Hybrid Images

In this part of the assignment, I created hybrid images that change in interpretation with viewing distance. Generally speaking, I extracted the high-frequency portion of one image and the low-frequency portion of another so that in the hybrid image the high-frequency image shows up when viewing the image closely while the low-frequency image appears when the viewer steps farther away.

First I low-pass filter one image using the standard 2D Gaussian filter, then I high-pass filter the second image by subtracting the Gaussian-filtered image from the original image. I filter the low pass image with a Gaussian where sigma = 3, and for the high pass image, I use sigma = 5 in the Gaussian.

The Aslan/Jesus image was my favorite hybrid image because it's the most seamless. In the Trump/Hillary image, the details of hair from the high frequency image are distracting. In the failure image of Umbridge and the toad, the proportions of the toad’s features don’t align well with the human face. The distance between each of the toad's eyes is much larger in comparison to a human face, and the distance between the eyes and the mouth is much smaller than in a human. As a result, the images don't align well.

Hybrid Jesus and Aslan

Hybrid Trump and Hillary

Failure Umbridge and toad

Aslan

FFT applied to Aslan image

Jesus

FFT applied to Jesus image

Aslan with high-pass filter

FFT applied to filtered Aslan

Jesus with low-pass filter

FFT applied to filtered Jesus

Final hybrid Jesus and Aslan image

FFT of the final image

Part 1.2: Bells and Whistles

I used color to enhance the hybrid effect. On the left, the Aslan image (high frequencies) is kept in color, while on the right, the Jesus image (low frequencies) is kept in color. The implementation is similar to 1.2, but I copied the values of the greyscale image into a 3D matrix so that the dimensions would match the colored image for computation purposes. The effect, where Aslan is present up close and Jesus appears from far away, is still present in both pictures. However, when color is used in high frequncy image, it is slightly more difficult to see the low frequency image. Coloring the low frequency image produces better results since both images are more equally visible.

Aslan (high frequency) color

Jesus (low frequency) color

Part 1.3: Gaussian and Laplacian Stacks

In this section, I implemented a Gaussian and a Laplacian Stack. The Gaussian stack is created by applying the Gaussian filter to the image successively until the specified number of levels is reached (N = 5 for my images). As the successive Gaussian level is created, the Laplacian stack can be created simultaneously by subtracting the new Gaussian image from the image at the previous level. The result is the band of frequencies that are blurred at each Gaussian level, essentially the results of a high-pass filter between successive Gaussian images.

Below are the results of creating the Gaussian and Laplacian stacks of Salvador Dali's painting of Lincoln and Gala. We can see the top row's resemblance to Lincoln, and in the bottom row, we can see that Gala clearly in the first and second high frequency bands.

Top row: Gaussian stack, Middle row: Laplacian stack, Bottom row: Histogram equalization applied to Laplacian stack for clearer viewing

I created the Gaussian and Laplacian stacks of my Aslan/Jesus hybrid image. In the Gaussian stack, we can see that the removal of high frequencies results in an image that looks like Jesus, and in the Laplacian, we can see the high frequencies of Aslan being extracted from the image.

Top row: Gaussian stack, Bottom row: Histogram equalization applied to Laplacian stack for clearer viewing

Part 1.4 Multiresolution Blending

For two input images A and B, I first created Gaussian and Laplacian stacks for each. Then I build a Gaussian stack for the mask. In the Orapple case, I built the mask in python by creating a matrix with half zero columns and half one columns in the same dimensions of the input images.

After the necessary stacks are created, at each level, I reconstruct the original image. At level n, I add together the Gaussian image from the stack at level n with the Laplacian image from level n-1. I do this for A and B and add it to the reconstructed images at previous levels. Then, at each level, I construct the blended image by applying the mask to A and adding it with (1-mask) applied to B.

Trump and Hillary

Orange and Apple

Hand and Eye (irregular mask) - for this image, the sigma on the mask Guassian is much less (0.5 on this one vs. 10 on the others)

Me and Yudi

Illustrated process for blending Trump and Hillary. The first image shows the gaussian and laplacian stacks for Trump. The second image shows the gaussian and laplacian stacks for Hillary. In the last image, the left two columns show the reconstructed images at each level after being masked, and the final column shows the blended image at each level.

Part 1.4 Bells and Whistles

See above for colored multiresolution blending

Part 2: Gradient Domain Fusion

The goal of this part of the assignment is to blend an image (the source image) seamlessly into another image (the target image). We explore doing this with poisson blending. The main idea is to create a new image v, solve for the values of the pixels within the "s" region from the source image while keeping the same gradients as the source image and keeping the background pixels from the target image.

Part 2.1: Toy Problem

For the toy problem, I reconstruct an image using its top left pixel value along will the gradients in the image. In order to reconstruct the image, we want to make sure the x-gradients of v (the new image) should closely match the x-gradients of s (the source image) and the y-gradients of v should closely match the y-gradients of s.

To implement this, I created a sparse A matrix that held the coefficients necessary to compute the gradient for a particular pixel. For example, if we were computing the gradients for pixel x, then the equation would be 4(pixel value at x) – pixel value of each of x’s neighbors, so the A matrix would hold 4 at the pixel and -1 at each of its neighbors. The 4 coefficient changes if the pixel of interest is on the border of the image. At the same time I construct a vector b which contains the gradients for all the pixels (calculated with the same equation used for A). Then I used a sparse matrix solver to recover the pixel values of the image.

Original Image

Reconstructed image

Part 2.2: Poisson Blending

In this part, I implemented poisson blending. In order to blend the source image area into the target image, we want to make sure the x-gradients of the desired region “s” in v (the new image) should closely match the x-gradients of the source image and the y-gradients of the region in v should closely match the y-gradients of from the source image. At the same time, we set the boundary pixels to this region to be the same as those of the target background image. At the high level, I construct an A matrix with the proper coefficients to calculate the gradients and a vector b that contains the gradient values from the source image. Solving Ax = b for x will give me the pixel values of the desired region.

First, I split the RGB image into its color channels. Then, I create a new image v that copies all the pixel values from the target image in the non-mask area. After, I compute the gradient (np.gradient) of the mask to help me find the boundaries of the mask (anywhere the gradient isn’t 0 means that it’s either part of the mask or a boundary to the mask). Given the indicies where the gradient is nonzero, I find the minimum and maximum row and column indicies, which provides the area of the pixels that I will solve for in the matrix equation.

Next, similar to part 2.1, I loop through the target area I found and construct an sparse A matrix with the coefficients from the Laplacian matrix. If the pixel I am currently looping through is not part of the mask area, then I set its row to be the identity in A and the value in b to be the pixel value in the target image. This takes care of the boundary condition. If the pixel is part of the mask area, then like 2.1, I compute the coefficients for a given pixel and its neighbors (i.e. 4, -1, -1, -1, -1 or 3, -1, -1, -1, or 2, -1, -1) and insert it into A. In b, I insert the gradients of these pixels in the source image. Thus, solving for x gives the pixel values in the desired area.

In the end, I reshape the x vector to fit into the image. I add together the x vectors computed from each color channel and insert that into v to produce the final image.

Final Image - Trumpkin

Trump copied onto pumpkin

Target Image

Source Image

Mask

More Results

Here, I blended a boy on a floaty thing into a pool, but because of the lighting and color changes, the boy looks yellow and unnatural

Pool target image

Boy on floaty source image (pre-aligned in photoshop)

Mask used

Here, I blended the example of a penguin into a snow scene.

Snow target image

Penguin source image (pre-aligned in photoshop)

Mask used

Comparing multiresolution blending with poisson blending

Blending is better with the poisson method because it maintains the gradients of the source image even though it recalculates the pixel values. As a result, the eye fits into the image better even though the the pixels are different colors. In the multiresolution blending, the result shows a clear distinction between the area surrounding the eye and the hand, whereas in the poisson one, the area is less noticeable. In addition, in multiresolution blending, a gaussian stack for the mask needs to be created, and as a result, more of the image is blurry.

Multiresolution blending

Poisson blending