CS194 Project 3: Fun with Frequencies and Gradients
Matthew Waliman, cs194-26-afq
An image can be thought of in terms of its constituent frequencies and gradients. In this project we take advantage of these by modifying the image in either the frequency or gradient domain and the recovering the image to great effect.
I. (un)Sharpening
We'll "sharpen" these images by adding successively more high frequency information into the image. In order to get the high pass of the image, we apply a Gaussian blur, giving us the low passed filtered version of the image. We then subtract this from the original image, leaving just the high frequency information which we can add back into the image. Here alpha referrs to the width of the gaussian, which determines how much high frequency information you add back to the image.
II. Hybrid Images
These hybrid images are assembled by merging the high pass information of one image, with low pass information of another. This works because your eyes can only discern an upmost bound of frequencies at different focuses, so you see different levels of detail, or frequency.
Failure Case
I tried to combine two people with similar head shapes and expressions. In this case it was unsuccesful simply because their coloring was so different.
I found that adding color to was very effective for the low pass filtered image, oftening overpowering the high pass filtered detail entirely.
III. Gaussian and Laplacian Stacks
By repeatedly applying, and subtracting off the gaussian from multiple images, we can isolate different bands of frequency in the image. The application of the gaussion filter gives us the gaussian stack, while subtractions between these gaussian levels gives us the laplacian stack which gives us different bands of frequency. Each image here was made by increasing the sigma by a power of 2.
We clearly see the bands of frequency here we can see the higher frequency Brad Pitt detail give way to the lower frequency Angelina image data.
IV. Multiresolution Blending
Given a Laplacian stack, we can add each level of the stack on top of each other until we recover the original image. If we apply a mask to one stack, and the opposite of that mask to another stack, we can blend the two images along the seam of the mask. This works by adding each of the laplacians at each level, using the gaussian of the mask as weights.
V: Gradient Domain Fushion
The Laplacian blending method performs blending by combining pixel values of two different images according to a mask. However, the human visual system is far more sensitive to changes in gradient than changes in intensity. We can take advantage of this fact to create more natural looking images.
Toy Problem
In this toy problem we reconstruct the original image (with minimal error) by solving a system of constraint equations that specify that the reconstructed image match the gradients of the original image, then by setting an initial condition as a single pixel value from the original image.
Poisson Blending
In order to construct hybrid images using gradient blending, we use largely the same method from the toy problem, but now use a mask to determine the constraint equations. If a pixel is not within the mask, we want the new pixel to simply match the target region. If it is within the mask we have two cases. a. All of its neighbors are contained within the mask or b. it is in edge pixel. In the interior case we simply want the gradients to match the gradients of the source image. In the boundary case, we want the gradients aross the boundary to blend smoothly from the target region to source region. We hold the target pixel value constant, and choose an edge pixel value that as closely matches the source gradient across these pixels as possible. In effect we are constraining the derivative to begin at the exterior edge of the source region so pixel values inside the source region will differ from the original image but the gradient across the boundary will stay relatively unchanged.
Because the algorithm works by matching gradients, I wanted to try a target image with many varying gradients, like the interior of a stained glass window for example.
Poisson Blending tends to work best on light colored source images and image with plenty of texture to hide the mismatched regions around the source object. This image of Mariah Carey in a snowglobe fulfills the color constraint beautifully, and the resulting lighting looks natural. However, because the target image has a fairly smooth background, you'll notice that the region around her body reflects the texture of the fringe behind her and not the smooth background of the target image.
This image was significantly less successful than the others due to the choice of image. Because the target image is so dark, the edge constraints force the resulting image to become much darker as well. You'll notice that around the feet and head, the images actually match the background color quite faithfully. It's only adjacent to the black regions of the target image do we lose all definition in the source.
A little repositining "fixes" the output image, if you can call that fixing.
Blending Comparison
Both the multiresolution and Poisson methods have different strengths. The Poisson blend method faithfully reflects the colors of the target painting, but suffers because the mask is situated close to the hairline. Because Possion attempts to match the gradients at the edges of the mask, some of the darker colors of the hair begin to bleed into the face. The Multresolution blend is less realistic, suffering from the inconsistencies in light fixed by the Poisson blend. However, we don't have the same gradient matching/color bleeding problem because the two hair regions of the target and source simply blend through each other, creating quite a nice hairline for the resulting image.
Mixed Gradients
When Poisson blending a source image with a smooth background onto a textured target image results in the source background standing out very clearly from the surrounding target background. This effect is particularly obvious in images that require transparency. We fix this by selecting the gradient with the largest magnitude between the source and target images.
VI: Conclusion
I really enjoyed this project! I learned a lot about Gaussians, the frequency domain, gradients, constraint equations and the importance of modularizing your code.
