Project 3: Fun With Frequencies and Gradients

James Lin, CS194-26-abr

Part 1: Frequency Domain


Part 1.1: Warmup

To sharpen the image, I first smoothed it with a gaussian filter of std 1, then subtracted the result from the original to get the high frequencies. After rebalancing values, I multipled the high frequencies by an alpha of 0.5, added it to the original image, and rebalanced. This was the result.


For rebalancing, I took the min and max intensity values of the image and rescaled everything such that the min ended up at 0 and the max ended up at 1.

Part 1.2: Hybrid Images

I created hybrid images by smoothing one image and extracting the high frequencies from another, then combining the two via an average. In the following example, I used Marilyn Monroe as the low frequencies (std of 5) and Albert Einstein as the high frequencies (std of 3).



Here are the Fourier transforms of the two (aligned) input images. Marilyn's is on the left and Einstein's is on the right.


Here are the Fourier transforms of the processed images. On the left is Marilyn's smoothed, in the center is Einstein's with the high frequencies extracted, and on the right is the final hybrid image.


As you can see Marilyn's picture maintains only the green on the centers (the low frequencies), while Einstein's picture loses the intensity in the center with the loss of lower frequencies. As the final image averages the two, each image makes up for what the other lacks and the result is similar to that of the input images.

Here are two more hybrid images.


On the left I combined the character Doomfist from Overwatch with actor Terry Crews. You'll notice that the picture of Doomfist is actually from a 2D animation. I wanted to see if this technique would work on images from different mediums, and it seemed to work out well.

On the right I combined a cat with a dog. While previously I always attempted to find similar head shapes and aligned on the eyes, here I experimented with forgoing that. The result is that the transition from high to low frequencies is nowhere near smooth and it looks more like the two images were overlayed. This was a failure.

Part 1.3: Gaussian and Laplacian Stacks

The gaussian and laplacian stacks were created with a standard deviation of 3. Below you can see the gaussian (top) and laplacian (bottom) stack for the Lincoln/Gala painting.



On the top row, you can see how the image turns into a blocky portrait of Lincoln as the higher frequencies are taken away. For the lower row the effects of taking away the lower frequencies are less evident, but the outline of the female remains visible throughout the stack.

Here is another gaussian/laplacian stack on the Marilyn/Einstein hybrid image.



Once again the effect is much clearer on the gaussian stack than the laplacian one. In particular, Einstein seems to retain Marilyn's mouth as the lower frequencies are removed.

Part 1.4: Multiresolution Blending

I followed the algorithm given in the slides and the paper to generate the following vertical seam multiresolution blended images. I used 8 layers and varied the STD from 1-5 to see what worked the best.



The blending seems to easier to pull off in busy/messy scenes. Similar perspectives are a must, and similar lighting helps as well. Here are some blends that don't use a vertical or horizontal seam, using the same parameterizations as before.


Some workflow notes: I created an alpha mask manually in Gimp to trace out the contours of the beach and the moon. In the case of the moon from Majora's Mask (left), I manually moved the moon to its position so that I wouldn't have write additional alignment code.


Part 2: Gradient Domain Fusion


In the previous section, I blended images together by linearly interpolating between pixel intensities, and changed the weights gradually to smoothen out the transition. But as you can see there are often still disctinct seams between the two images.

I now take a different approach to the problem. Instead of trying to preverve an image's pixel intensities, I instead preserve their gradients. This solution relies on the fact that the human eye perceives relative changes in intensity far more than the absolute value.

More specifically, given a target image, a source image that we want to insert onto the target, and a mask that marks out the area of the source we want to insert, we wish to do two things. The first is to match the gradients within the mask to that of the source image as closely as possible. The second is to match the gradients on the boundary of the mask to that of the source image (assuming anything outside of the mask uses the target image's pixel values.

Part 2.1: Toy Problem

As a warm up, I followed the instructions and implementation tips to complete the toy problem. On the left is the original image, and on the right the output.


Part 2.2: Poisson Blending

For this section, I followed the algorithm given in the slides. Choosing to preserve the original pixel intensities of the target image, I clipped any values in the resulting image that were below 0 or over 1. Mask construction was done by selecting a polygon given 15 points around a target.

I tried putting a whale in the sky. Below are the original images.


And here were the results. The left was with direct replacement, and the right with poisson blending.


Since the two backgrounds were fairly different in color, the color of the whale had to be modified greatly in order for the gradients to be maintained. I guess Moby Dick can fly now.

Below are some other blends. The penguin turned out well as the backgrounds matched, but the raptor was forced to turn red in order to compensate for the background. I actually liked the aethestic the white whale gave in the sky, but the bright red makes it difficult to sell this blend as a success.


Finally, I retried the majora mask moon picture from part 1 using poisson blending. Here were the original images:


For convenience, the Laplacian pyramid blend again:


And finally, the two blends from poisson blending.


The poisson blending of course is much more seamless, although at the cost of giving the rest of the moon a washed out appearance. It's hard to objectively say which one is better for this situation, but I would go with the poisson blended one because the lack of a clear seam makes it look more natural.

In general, I'd say that whenever background colors match between two images, poisson blending will perform far better in removing the seam. In pictures w/ noisy backgrounds or not matching ones, however, it would likely be better to use Laplacian pyramid blending to try and hide the mis-match with noise.