Fun with Frequencies and Gradients!

Nerissa Lin CS194-26-adv

Overview

In this project we aim to blend pictures together seamlessly if possible. We explore a number of methods to approach blending images. A lot of the image manipulation comes from filters taken on the images. We take a look at blending on the frequency domain before also trying gradient blending to achieve a better product.

Warmup: Sharpening the Image (1.1)




We apply unsharp masking techniques we learned in class to uncover a sharpened image. We are taking a high pass filter of the image (by taking the a Gaussian filtered of the image away from the original) and adding it back to the original. This takes the blur out and the high frequency images and adds it back in to refortify the image. Above we can see the difference this makes - although it might wash out the image a little, the details seem little clearer. Especially in the second llama photo, in the background the tree that was slightly blurry in the background becomes a more clearly defined tree. But the Llamas are still the cutest part of this photo!

Hybrid Images (1.2)



Grumpy Llama


FFT Process for Grumpy Llama

We have the frequency in order of - grumpy cat, happy llama, their respective filters, and the hybrid


Doggo Llama


Real Llama


Failed Trump Llama


We aim to create a hybrid function that seems to change as a function of view distance - up close the high frequency image pops out significantly and at a distance, its the low frequency image that is visible. We can use this property to emultate image blending, by letting one image come through as the low frequency and the other as the high frequency. Above we demonstrated this in a few scenarios. It seems that the images have to have some sort of compatibility to really be able to merge - in the case of trump llama, the face sizes are too different resulting in it the overlay being extremely noticeable, even at different level of sigmas and changing the order of the images.

Gaussian and Laplacian Stacks



Laplacian Lincoln Galleria

Guassian Lincoln Galleria

Laplacian to Guassian CatLlama



Pyramids were constructed to a depth of 5 but only displaying 3 levels out of need to save space

Multi-Resolution Blending (1.4)

Taking advantage of the pyramids that we implemented, we can do blending at every level to make the blending feel less harsh. We add these all up to get a result that is a rather smooth blend. This has to be done in conjunction with a mask where we first experiment with linear masks that are half half. Then we can try to use masks that are not linear for example, using a circular mask.






Gradient Domain Fusion

Copying parts of an image over directly can cause extremely harsh looking blends - we can correct for this by understanding that humans comprehend images on the intensity gradients as opposed to the actual image. Its more importance to mimic the difference between pixels than it is to mimic the color exactly. Using this concept we can take advantage of this and blend more seamlessly, slightly altering the color of a region to make it seem native to the image. We can see that this might help with out catllama image where cat's fur is a different color from llama's fur making the blending jarring. We use poissons blending as described above to attempt to fix for this. Below is work that makes progress on this idea.

Toy Problem (2.1)

We can reconstruct an image simply by saving the gradients of each pixel in relation to its x and y direction, as well as saving a constant (we choose the top left pixel to have to be matched accurately). If the top left is successfully matched, then we have sufficient relations to reconstruct the rest of the image. We can notice that there is some definition loss but almost full reconstruction is possible.

Our equations of interest:
1. Minimize ( v(x+1,y)-v(x,y) - (s(x+1,y)-s(x,y)) )^2
2. Minimize ( v(x,y+1)-v(x,y) - (s(x,y+1)-s(x,y)) )^2
3. Minimize (v(0,0)-s(0,0))^2
Equation 1 being our dx differential, 2 being our dy and 3 being out adjustment to find the constant after reintegration.



Poisson Blending (2.2)

Given a source and a target image, we can try to make the source object appear in the target image through masks and poisson blending. Our goal is to make the blending minimally noticable taking advantage of the gradients that we played with in the Toy Problem. The key difference here is that the pixels outside of the mask are kept the same and only the ones inside the mask are included in an equation we would like to solve for. While solving for these, we have to consider the relation of these pixels to those outside the mask, altering the pixel values returned to be dependant upon the edge pixels of the background outside of the mask.

Our equation of concern here is the following


Here the cat blends in quite nicely but actually you can see that theres a hue about the cat that seems to glow eerily. Maybe thats the true color of a moon cat.


The first toyllama is actually quite well blended, theres some darker backgrounding occuring to meet the gradients of the llama face but generally next to seamless blending. The failed sample we have is the one right under. We have a very poorly blended llama scene here and it seems that the blue from the background has bled through very severly into the llama. We get this green hue as well around the llama that is cutting into the tree even and all in all the photo is a lot darker since it tries to balance for the distinct tones.



Here we have the example of the catnosed - llama and when we tried to blend using pyramids, we got a really nice placement of the catnose onto the llama with a next to seamless blend except for the color of the image where there seems to be something distincly different. The middle image is a direct paste of the nose onto the llama before we blend it with gradients and retrieve the right most image. In this image we see that the color and the nose blend in almost perfectly. However theres some residue lines, likely from the process where the gradients are ill defined. Best approach here was the gradient blending

Generally it seems that blending is a good method when the immages are greatly different - although the color change might be jarring like in the case of our koalallama we can accept this difference since our perception of it already somewhat expects it. Then you get some nice blends that are actually quite subtle across the mask lines. However when you have regions being blended that need to match in color and most items are in fact in that same color palette, then the gradient blending works great as in our cat llama where we can see that the benefit of the furtone matching is far more important the the region necessarily feeling seamless.