Homework 3: Frequencies and Gradients

Jacob Green | cs194-26-afl

Part 1: Frequency Domain

1.1 - Warmup

For the warmup, I simply calculated a 2d gaussian filter (based on input of filter size and sigma), convoluted it with an image, subtracted this blurred image from the original value, and finally added this high passed-image to the original image. I did this with different weightings of the addition, and the results are shown below.

Sharpened Images with Different Edge Weights

Edge Weighting: 1
Edge Weighting: 2
Edge Weighting: 3
Edge Weighting: 4
Edge Weighting: 5

1.2 - Hybrid Images

To create a hybrid image, I used a combination of the starter code and the code from part 1.1. I started by aligning the two images based on two user-input points. After that, I simply blur the low frequency image by convoluting it with a 2d gaussian filter, and high pass the high frequency image by convoluting it with an impulse filter minus a 2d gaussian filter. My resulting images and fourier analysis is below.

Input low Frequency Image: Derek
Input high Frequency Image: Nutmeg
Blurred Low Freqeuncy Image: Derek
HiPassed High Frequency Image: Nutmeg
Hybrid Image: Derek and Nutmeg
Input low Frequency Image: Drake
Input high Frequency Image: Snake
Blurred Low Freqeuncy Image: Drake
HiPassed High Frequency Image: Snake
Hybrid Image: Drake and Snake

The Drake and Snake hybrid image was defintiely an example of a fail case, which I think stems from the fact that there wasn't much overlap between the images in the spatial domain, so it simply looks like a gradient image superimposed onto a blurred image.

Input low Frequency Image: Bill
Input high Frequency Image: White Walker
Blurred Low Freqeuncy Image: Bill
HiPassed High Frequency Image: White Walker
Hybrid Image: Bill and White Walker
Fourier low Frequency Image: Bill
Fourier high Frequency Image: White Walker
Fourier Blurred Low Freqeuncy Image: Bill
Foureir HiPassed High Frequency Image: White Walker
Fourier Hybrid Image: Bill and White Walker

1.3 - Gaussian and Laplacian Stacks

To split an image into a gaussian and laplacian stack, I begin by creating a 2d gaussian filter, convoluting it with the original image, and subtracting this blurred image from the original image. The blurred image becomes the first layer of the gaussian stack, and the original image minus this blurred image becomes the first of the laplacian stack. Then, recursively, I convolute the gaussian filter with the previous layer of the gaussian stack to calculate the current layer of the gaussian stack. Additionally, I calculate the next layer of the laplacian stack by taking the difference of the two most recent layers of the gaussian stack. Finally, I set the last layer of the laplacian stack to be equal to the last layer of the gaussian stack. Below are some laplacian and gaussian stacks I calculated.

Gaussian Stack: Drake
Laplacian Stack: Drake
Gaussian Stack: Mona Lisa
Laplacian Stack: Mona Lisa
Gaussian Stack: Bill and White Walker Hybrid Image
Laplacian Stack: Bill and White Walker Hybrid Image

1.4 - Multiresolution Blending

To calculate a laplacian blend of two images, I begin by calculating the laplacian stack of both input images. Additionally, I take an input mask from the user (or, in the case of the apple and orange, manually generate a mask covering half of the image) and generate its gaussian stack. Then, for each level i of the pyramid, I add the element-wise multiplication of the mask at level i with the first stack at level i to the element-wise multiplication of the inverse of the mask at level i with the second stack at level i. Finally, I collapse the resulting pyramid into a single image by adding all of the layers together, which results in the blended image. To do this with color, I simply iterated over all three color channels and calculated the results separately.

Multiresolution Blending: Bill and the White Walker
Multiresolution Blending: Bill and the White Walker, with color!
Multiresolution Blending: Drake and the Snake
Multiresolution Blending: Drake and the Snake, with color!

DIfferent Layers of Oraple Blending

Apple Layer 1
Combined Layer 1
Orange Layer 1
Apple Layer 2
Combined Layer 2
Orange Layer 2
Apple Layer 3
Combined Layer 3
Orange Layer 3
Apple Layer 4
Combined Layer 4
Orange Layer 4
Apple Layer 5
Combined Layer 5
Orange Layer 5
The Collapsed Final Image

Part 2: Gradient Domain Fusion

The main idea behind this project is to superimpose a specific region of one image into another image. This technique differs from laplacian or multiresolution blending (Part 1) in that it focuses not or direct pixel intensities, but rather pixel gradients, or the difference between pixels and their neighbors in the surrounding image. So, in laplacian blending, the goal was to simply smoothly transition by a fixed amount between two images, or at least a fixed amount at a given frequency band. In Poisson blending, each pixel in the target region is thought of as a variable that must be optimized. Then, we constrain this optimization by trying to minimize the error between gradients in the source and target image. For any given pixel, four constraints are generated. Each constraint corresponds to the gradient between the center pixel, and one of its four neighbors. The constraint attempts to minimize the difference between the gradient in the outputted pixels and the gradient in the source image. In the case that the pixel is on the edge of the image, the gradient is then measured between the output pixel and the neighboring pixel in the target image instead. Then, once all of these constraints have been amalgamated, they can be solved using a least squares solver to calculate the optimal values of each pixel, and then copy these solved values into the target image. This can be extended to the color space by simply calculating the poisson blend for each channel separately, and combinng the three.

2.1 - Toy Problen

Input Image
Output Image

2.2 - Poisson Blending

Penguin(Multiple Times)
People with Skis again
Together!
Bill
White Walker
Blended!

I think this blend of Bill and the White Walker illustrates how effective poisson blending is at dealing with variance in color. Despite the white walker being a completely different hue, the algorithm is able to basically port the features over onto Bill's face without bringing any of the blue color. To implement poisson blending, I simply iterate through all pixels in the target image, and if that pixel is within the provided mask, I append the four constraints from that pixel to my list of equations to be solved (taking into account edge cases, where the pixel is either on the edge of the mask or on the edge of the image). Then, I amalgamate these equations into a sparse matrix and vector, and use an interative least squares solver to find the pixel values. From there, I simply paste these values back into the original image at their specified locations, and get some really nice images like the ones above.

Direct Copy: Drake and Snake
Laplacian Blend: Drake and Snake
Poisson Blend: Drake and Snake
Direct Copy: Mona Lisa and Penguin
Laplacian Blend: Mona Lisa and Penguin
Poisson Blend: Mona Lisa and Penquin

Sometimes Poisson Blending has some real issues. In the Drake and Snake example above, it seems to struggle to blend the edges between Drake's lips and the Snake's head, and seems to lose a bit of detail in the snake compared to the Laplacian blend which seems slightly more effective in this case. More generally, one issue Poisson Blending fails to deal with is any attempt at intensity and/or color invariance. Since the restrictions only hold it to differences between pixel values, the base values of the pixels can all increase or decrease almost arbitrarily, except that they are slightly constrained to match the border pixels of the target image. In the case of Drake and Snake, it seems that the snake is much brighter than it should be and it ends up looking somewhat too bright, as Drake's lips just happened to be a very bright color. However, in the case of the Penguin on Mona Lisa, the Poisson Blending is the clear winner in my book. I think that Poisson Blending is at its worst when the masked area of the target area contains very large variances that are not captured by the neighboring pixels outside the target area. For example, with the Drake and Snake image, the inside of Drake's mouth is very dark, but his lips are all very bright, so the Poisson Blending algorithm thinks it should make the Snake bright enough to match his lips without taking into account the darkness that is within the masked area. In the case of the Mona Lisa, however, her face is relatively constant in terms of gradient and color magnitude, so the constraints accurately match the relevant variables in the equation.