CS194-26 Project 3: Fun with Frequences and Gradients

Josh Zeitsoff, cs194-26-abi

1 - Frequency Domain

1.1 - Warmup

To sharpen my image, I convolved a Gaussian filter on the image to blur it with sigma = 2. To get the detail, I subtracted this convolved image from the original. To determine the level of sharpness, I added the blur back into the image times some constant alpha, which I varied to achieve sharper or less sharp images. The final result here uses alpha = 10.

Original Image and Sharpened Image

1.2 - Hybrid Images

To make a hybrid image of Derek and Nutmeg, I applied a lowpass filter to Derek by convolving it with a Gaussian and a highpass filter to Nutmeg by subtracting a Gaussian filtered image from the original. I tried applying the inverse of this (lowpass to Nutmeg and highpass to Derek) but my images did not turn out as well. I messed with the cutoff frequencies of each until I settled upon 2 sigmas that worked - 8 for Derek and 15 for Nutmeg.

Original Derek and Nutmeg

Hybrid Derek Nutmeg

Fourier Analysis

This image turned out similar to Derek and Nutmeg, and I chose the same sigma values and ordering of the images (lowpass on the image of myself and a highpass filter on the image of the alpaca.

Original Josh and Alpaca

Hybrid Josh Alpaca

s

1.3 - Gaussian and Laplacian Stacks

Taking the Dali painting "Gala Contemplating the Mediterranean Sea which at Twenty Meters Becomes the Portrait of Abraham Lincoln-Homage to Rothko (Second Version)," I made separate Gaussian and Laplacian stacks of the image. For the Gaussian stack of 5 levels, at each level, I applied a Gaussian filter to the previous level. The sigma I used for my Gaussian filter was sigma = 8, and a window size of 20x20. For my Laplacian stack of 5 levels, at each level I took in a Gaussian filtered image from the level before, applied a Gaussian filtered image to ths image, and then subtracted the image I applied the Gaussian to from the original (L(i) = G(i) - G(G(i))). For my Laplacian stack, I found sigma = 15 and window size of 30x30 to work well.

Original Abraham Lincoln Dali

Laplacian Levels 4-0

Gaussian Levels 4-0

Applying Laplacian and Gaussian stacks to hybrid images

Similar to Figure 7 in the Oliva et al. paper, I attempted to deconstruct my hybrid image into the two original images. It seems like this process worked well to extract the image of Derek from the hybrid image, as you can see below in the Gaussian Level images. However, I expected the Laplacian stack to extract out the image of Nutmeg, but it did not. I made sure to use the same sigma values for making the hybrid image and making the Laplacian stack, so I am unsure why I was unable to extract out the image of Nutmeg in the Laplacian Level images.

Original Derek Nutmeg

Gaussian Levels 4-0

Laplacian Levels 4-0

1.4 - Multiresolution Blending

In this section, I made a Laplacian Stack of 2 input images (the apple and the orange) and used a mask to represent the location where I wanted the two images to meet (in the middle of the image). For my Laplacian stack, I used sigma = 10 and window size = 20x20. I applied a Gaussian to this mask to better blur the boundary between the 2 images. For the Gaussian blur on the mask, I used sigma = 20, and a window size = 40x40. I then combined each level of these stacks according to the formula (G(i) * A(i) + (1 - G(i)) * B(i)), where A is a layer in the apple laplacian stack and B is a level in the orange laplacian stack. Finally, I summed up all of the layers of the stack to achieve my final image.

Original Apple and Orange

Orapple

In addition to the orapple, I attempted to use multiresolution blending to blend 2 different boba drinks. I took one image of a milk tea, one image of a fruit tea, and tried to make a milk fruit tea. The original images themselves did not align as well as the apple and the orange did, which I think is why the straws are not aligned perfectly. However, if you take a look at the anteater, I think it looks like it blended fairly well between the images.

Original Milk Tea and Fruit Tea

Milk Fruit Tea ( Zot Zot Zot)

In this example, I tried to use an irregular mask compared to the mask used in the first two examples. The mask captures part of the sun and blends it with the cotton candy on a stick I was holding. I think the mask could've been rounded a bit more, and if I knew about Nikhil's starter code for part 2.2 earlier, I probably could've used it to make a mask for this section as well, better aligning the sun onto my cotton candy on a stick.

Original Fiery Sun and Josh holding a balloon

Josh holding a fiery balloon

2 - Gradient Domain Fusion

Our goal for this section was to better blend an image from one source into a different target image. With Laplacian blending, there was a clear distinction where the source image and target image met. Our plan was to choose new pixel values that minimized the difference between them and the gradient of the original source pixels, formulating it as least-squares problem. We did need a different approach for the border pixels, where instead looking at the difference in gradients between new pixels and the source pixels, we looked at the gradient between a new pixel in the target region and a target pixel from outside the region. Thus, we are ensuring that along the border, our final pixel values will take into account the background of the image we are trying to blend them into.

2.1 - Toy Problem

For the toy problem, I constructed an A matrix and B vector by going through all of the x and y gradients, setting the appropriate columns in the A matrix based on whichever pixels I was looking at, and assigning values in B based on the values in the original image at this gradient. My toy problem did not come out exactly as the original toy problem, I think because I might've shifted my values somehow. I tried just using the result of the least squares solver, scaling my values to between 0 and 1, and dividing by 255 to get all values between -1 and 1, but none of these approaches allowed me to achieve the same color as the original.

Toy Story Original

My Toy Problem

2.2 - Poisson Blending

For this section, I used Nikhil's starter code to obtain a mask on the source image from which to select pixels of, and a mask on the target image to tell me where new pixel values should be placed (essentially overlaying the mask from the source image minus the background from the source image on the target image). Now, it was a matter of solving for pixel values in the target image inside of the mask. My overall approach to Poisson blending was to construct a sparse matrix A, a vector B, use a sparse linear algebra matrix solver, and then repopulate these pixels into the mask in the target image. I had a list of nonzero pixel indices in both the target mask and source mask, since these were the pixel values I wanted to solve for and use as source pixels, respectively. Going through this list of indices, I looked at the neighbor of each pixel in the x,y direction as well as each diagonal pixel. If the neighbor pixel was also in the mask, I used values from the source image and added the difference between them to the B vector. If the neighbor pixel was not in the mask, I used the same pixel values as above, but also subtracted a pixel from the target image at the same location of the neighbor pixel (according to the formula in the spec). This was done to ensure that the pixel values on the edge blended with the pixels adjacent to it from the target image. I then reassembled the pixels in the target mask within the target image.

Best Blending Result

Original Josh

Original Half Dome

Unblended

Mt. Joshmore

Here, I attempted to blend myself onto Half Dome. I think this worked well because my face and Half Dome are of a similar color, so it doesn't look weird to the humam mind. Also, my face fits entirely within the section of Half Dome that the sun is hitting, so I only need to blend my face with one color (an oranish-yellow). We'll see in later examples that images that require blending with multiple bacckground image colors didn't look as well.

Poisson Blending 2

Original Arctic

Original Dragonite

A Wild Dragonite was spotted

Here, I attempted to blend a wild Dragonite onto a picture of the Arctic. Similar to my Half Dome image, this worked well because the background Dragonite is being blended into is roughly the same color all around it, a bluish white.

Poisson Blending 3

Original Sun

Original Josh in Sunglasses

Sunglasses Emoji

Here, I attempted to blend myself onto the sun, al la the sun in Teletubbies or the Sunglasses Emoji. I'm happy with how black my sunglasses remained, though there is a tint of yellow in them, perhaps an artifact of trying to blend them with the surrounding yellow.

Poisson Blending Unsuccessful

Josh and Voodoo Donut

Josh Donut

This image was unsuccessful because I attempted to blend my face into 2 different background colors. My forehead was on the background of a pink wall, so the algorithm tried to blend my forehead in with the pink. However, the head of the donut is brown, so the lower part of my face was blended in with the brown. This might've been resolved with a better mask, since if my face fit entirely into the brown face of the donut and did not touch the pink background at all, it would've blended with the pink. However, I found it hard to have the shape of my mask match completely with the donut head. Finally, I think the image is slightly jarring because my face does not share the same color as the original donut's head, so it is distinctly different from the donut body.

Laplacian Blending vs Poisson Blending

Image 1

Image 2

Laplacian Pyramid Blending

Poisson Image Blending

Looking at the Laplacian blending example and the Poisson blending example, I think the Poisson blending algorithm did a better job of blending the edges between the sun and the cotton candy I was holding. I think one reason why is that I had a better mask for the Poisson blending example, and the irregular mask I used for Laplacian blending wasn't as exact as the mask I obtained from usign Nikhil's starter code. However, I think the Laplacian blending example does a better job of maintaining the color of the original source image, as you can more clearly tell it is the sun in the Laplacian blending example rather than the Poisson example even though the former is in gray. It seems like Poisson blending is more appropriate when you really want the source image pixels to blend well into the target image, but Laplacian blending might do a better job if you don't mind the border being a bit more obvious and prefer to maintain the color of the original source image.

What I learned

A collection of thoughts during this project. A lot of imaging is really just linear algebra and solving matrices. Bluring the boundaries between 2 images can be done several ways (Laplacian Blending vs Poisson Blending) with different tradeoffs for eahc depending how well we want our border to be blended. Masks are necessary to obtain certain pixels from source or target images. This project was actually a lot of fun once my 2.2 code worked and I was able to try it on different images. Since I never learned how to use Photoshop, I think I would probably come back to this code if I was to blend someone's face onto something rather than learn Photoshop because I understand what this code does better than clicking a button on Photoshop. Also, minimizing the difference in gradients produces images that are more pleasing to the eye than simple copy paste, even if you lose some of the color in the source image. Finally, don't start this project the Friday before it's due. s