CS 194-26 - Project 2 (Fun with Filters and Frequencies!) [Andrew Ke]

Part 1: Fun With Filters

Part 1.1 Finite Difference Operator

Here is the partial derivative in x and y, created by convolving the finite difference operators with the image. The finite difference operators are [1,-1] and [1, -1]^T and they approximate the derivative along the x and y axis.

The gradient magnitude is computed by combining the partial derivatives elementwise (magnitude = sqrt(dx^2 + dy^2)). The magnitude represents the edge strength at a given pixel. Here is the gradient magnitude image:

I then binarized the image with a threshold of 0.25 (chosen by hand):

Part 1.2 Derivative of Gaussian (DoG) Filter

Since the result of the binarized image is noisy, we can try using a gaussian smoothing operator to remove the noise. I convolved the image with a gaussian, and then applied the finite difference operators again.

After computing the magnitude, here is the binarized result (threshold of 0.065):

Observations: In the partial derivatives image, there is less noise in the flat gray parts of the image, and the edges are smoother and less jagged. In the binarized image, we get thicker edges that are less jagged and broken up than before (and the magnitude image is also smoother). The overall result is much better.

Now I combine the two convolutions (gaussian/finite difference) together in to a single filter. Here is what the derivative of gaussian filters look like:

And my resulting image looks the same as before.

Part 2: Fun With Frequencies

Part 2.1 Image "Sharpening"

To sharpen the image of the Taj Mahal, I created a low pass on the image using a gaussian blur, and then subtract it from the original image to get the high pass.

We then add the high frequencies to the original image to "sharpen" it. I used alpha=0.6 here (alpha controls how much of the sharpening we want).

You can see that the sharpened image is more contrasty and there is also some aliasing around the trees.

I combined this all in to one filter, and the results look the same.

Below is an oversharped image with alpha=1.4.

Some more examples:

My own image of the international house:

Some lions (source):

In the image below, I picked a sharp image (a durian), blurred it, and tried to sharpen it back.

As expected, the re-sharpened version doesn't look very sharp since once you remove the high-frequencies with the blur (a low pass), you can't bring them back.

Part 2.2: Hybrid Images

Hybrid Images are created by combining a low pass of one image with a high pass of another image. From close up, you see the image that was high passed, but when you step back, you see the image that was low passed.

Here is my hybrid image of Derek and his cat. I darkened it a bit to match the reference in the project spec.

Bells And Whistles: I played around with color combinations, and ended up using color for the high pass, but no color for the low pass. This works best because the color of the cat helps make the cat look more convincing from close-up, but when you look at it from far away, the colors become indistinguishable and you can still see Derek clearly.

I also experimented with full color and no color, but the results were not as convincing.

Full Color (Derek too obvious)
No Color (Cat not as convinving)

FFT Analysis

After the high pass, the low frequencies (around the center) of the cat are removed:

After the low pass, the high frequencies (far from the center) of Derek are removed:

Finally, this is the FFT of the combined hybrid image (You can see both the horizontal/vertical bars of Derek, and the X shape explosion from the cat):

More Fun Hybrid Images

I didn't include input images here because they give away the illusion, but I have uploaded the inputs to Gradescope.

For the next two examples, I got the input images from here and recreated the effect.

Close: Old Shanghai, Far: Modern-Day Shanghai

Close: Durian, Far: Guitar

These next two examples were my idea:

Close: Prof. Anant Sahai, Far: Prof. Jonathan Shewchuck

This one is quite cool since they have similar hairstyles and expressions in the photo, and the eyes are lined up well. In this image, when you step back, the direction that the head points changes, which I think is pretty neat.

Close: Real Bear, Far: Oski Bear

This one is a failiure since Oski is too obvious in the background, especially with the big C on his jacket. I couldn't find any pictures of real bears posing like Oski, so I had to settle for this.

Part 2.3: Gaussian and Laplacian Stacks

A gaussian stack is a series of same size images that are progressively blurred (I doubled my blur size for each image). The laplacian stack represents the deltas between consecutive images in the gaussian stack. The final image of the laplacian stack is the same as the final image of the gaussian stack (since we want the laplacian stack to sum up to the original image).

I applied a Gaussian and Laplacian stack to my finished oraple image from the next section.

Linked are the G and B channels.

This is what is looks like on the apple (on its own):

Part 2.4: Multiresolution Blending (a.k.a. the oraple!)

To blend, the two images, I use a mask that is progressively blurred. For the high frequencies, I combine them using a very sharp mask, while for the low frequencies, I use a much more gradual mask. This is the mask stack I used for blending:

Here are my visualizations for how the masks are used on the oraple:

Grayscale:

In Color (Bells & Whistles)

To add color, I had to normalize each color channel so the values ranged from 0 to 1.

Halves

Final Result

In the color result, you can see more clearly that the fruits being blended are an apple and an orange, and overall it is a lot more convincing.

My own blends:

Winter tree blended with autumn tree (vertical seam):

Baby face blended with the Moon:

I created this mask in Photoshop:

The process:

The result:

I like this result because you can see how the top of his right eye lid is darker than the top of his left eye lid because it is mixed with the black patch on the moon (a good example of multi-resolution blending at work).

Reflection

The coolest thing I learned was the sharpening, since I have used sharpening before in tools like Photoshop, but now I know how it is actually implemented and the reason it works.