Fun with Filters and Frequencies
& Multi-resolution Blending and the Oraple Journey

Ron Wang

CS 180, Fall 2023, UC Berkeley




Part 1: Fun with Filters

Part 1.1: Finite Difference Operator

Partial derivative in \( x \)

Partial derivative in \( y \)

Gradient Magnitude Image

Original

Gradient Magnitude Computation

The gradient of the function \(f\) is given by the vector:
\( \nabla f = \left( \frac{\partial f}{\partial x}, \frac{\partial f}{\partial y} \right) \)
The gradient magnitude, is therefore given by:
\( |\nabla f| = \sqrt{ \left( \frac{\partial f}{\partial x} \right)^2 + \left( \frac{\partial f}{\partial y} \right)^2 } \)
Because this measures how quickly the intensity of the image changes at any given time, it provides a helpful measure for us to detect edges in images, as illustrated in the above example.




Part 1.2: Derivative of Gaussian (DoG) Filter

In this part, we apply a smoothing operator, i.e. the Gaussian filter, to create a blurred version of the original image. This allows us to compute the partial derivatives and gradient magnitude on less noisy images.

Partial derivative in \( x \)

Computed from blurred image

Partial derivative in \( y \)

Computed from blurred image

Gradient Magnitude Image

Computed from blurred image

Gradient Magnitude Image

Previous version

What differences can we observe?

When we process the image with a Gaussian filter, the noisy pixels have less influence on the overall image. Thus, all of the images we computed in this section are less noisy. On line 2, we display the gradient magnitude image computed using the Gaussian filter (left) and the previous version where no processing is done (right). There are very noticeable differences in the quality of the edges. By using a Gaussian filter, we are able to focus on the most prominent features of the image, and we retrieve smoother and more consistent edges.

Alternative approach

We can do the same thing with a single convolution instead of two by creating a derivative of gaussian filters.

Derivative of Gaussian filter \( D_x \)

Derivative of Gaussian Filter \( D_y \)
The final outputs are exactly the same as above.






Part 2: Fun with Frequencies!

Part 2.1: Image "Sharpening"

Taj Mahal

In this subpart, we start with the blurry image of Taj Mahal. By subtracting the blurred version from the original version, we repeatedly gain the high frequencies of the image. Because an image often looks sharper if it has stronger high frequencies, we are able to "sharpen" the blurry image.

Blurred Taj Mahal image

Sharpened 0.5x

Sharpened 1x

Sharpened 1.5x

Sharpened 2x

Sharpened 2.5x



Mt. Tamalpais

We also explore the results using the following image, which I took on a hiking trip in the Berkeley Hills.

Mt. Tamalpais

Mt. Tamalpais (Blurred)

Now, to sharpen the image again:

Blurred Mt. Tam image

Sharpened 0.5x

Sharpened 1x

Sharpened 1.5x

Sharpened 2x

Sharpened 2.5x



Part 2.2: Hybrid Images

In this part, we create hybrid images using the approach described in the SIGGRAPH 2006 paper by Oliva et al. Hybrid images are static images that change in interpretation as a function of the viewing distance.

Sample images (of Derek and his former cat Nutmeg)

Derek Picture

Nutmeg Picture

Hybrid Picture
Nutmeg (the cat) is clearly visible in the hybrid picture. When we view the image from a distance, however, we should be able to (only) see Derek.

More examples - Ed Sheeran & Squirrel

I had a great time at Ed Sheeran's The Mathematics Tour, and I study at Berkeley, so I came up with the idea of mixing two things I like - Ed Sheeran and squirrels. When you are close to the image, you should be able to see the squirrel. However, if you move further away from the screen, you should only be able to see Ed Sheeran. If you don't see Ed, you are not far enough from the screen.

Ed Sheeran

Nutmeg Picture

Hybrid Picture



Fourier Frequency Analysis

This part displays the fourier frequency analysis results for the original images.

Input Image: Ed Sheeran

Input Image: Squirrel

Filtered Image: Ed Sheeran

Filtered Image: Squirrel

Hybrid Image

Bells & Whistles: Using Colors to Enhance the Effect

To enhance the effect of our hybrid image, we can retain the colors of the images in either the low-pass or the high-pass filter. The results are displayed below:

Uncolored output

Colored low-pass filter

Colored high-pass filter
In this particular example, using colors yielded very promising results. When I am close to the image, I can discern even the slightest details of the squirrel. This is especially evident in the second example, where we used color in the high-pass filter (i.e., high-frequency components). It renders the image more vivid and captivating.

Now, what if we used colors in both high-frequency and low-frequency components?

In this example, this is also a viable approach. However, I like the results that used colored high-pass filter better.

I think for different hybrid images, the visual output will be different each time you try different combinations of weights as well as choices of filters to retain color. It's quite challenging to definitively say which approach will always work the best.


More examples - Porsche and Frog

Porsche 911

Frog

Hybrid Picture



Failed Example - Joker and Batman

There are several reasons this example doesn't work as intended. One of the most important reasons is that the Joker has very prominent facial features, including the markings on his face, which are easily visible even from a distance. This does create a very interesting combination, though - the upper half of the image looks like the Joker, but the lower half reminds us of the Batman.

Joker

Frog

Hybrid Picture




Part 2.3: Gaussian and Laplacian Stacks

In this subpart, we implement Gaussian and Laplacian stacks, these are similar to pyramids but without the downsampling. This is a preparatory step for multi-resolution blending. The Gaussian stack contains low-pass images, and the Laplacian stack contains sub-band images. To construct the Laplacian stack, we subtract the Gaussian low-pass image from the previous layer.

Apple

Orange

Apple (Masked)

Orange (Masked)

Blended



Part 2.4: Multiresolution Blending (a.k.a. the oraple!)

Now, let's proceed to create the Oraple, a blended version of an apple and an orange, discussed in the 1983 paper by Burt and Adelson. An image spline is a smooth seam joining two image together by gently distorting them. Multiresolution blending computes a gentle seam between the two images seperately at each band of image frequencies, resulting in a much smoother seam.

To deliver the final result, we blend the corresponding levels of the Laplacian stacks of the two images using a Gaussian mask, and then collapse the final image.

Result

More examples - Venice Beach

The left side of the input image is from the popular video game, Grand Theft Auto 5, which is set in Los Santos—a locale inspired by the real-world city of Los Angeles. On the right side is the same location as in the game, Santa Monica Beach, but captured in real life. Through this composition, I aim to blend the two halves to explore the boundary between fiction and reality. While the skies merge seamlessly, the grounds don't blend as effectively due to differences in texture. I think more advanced techniques might help the two halves blend better, but still I really enjoyed the output.

Input

Output

More examples - Irregular Mask

In this example, I attempted to depict a Corgi running through a baseball field. It's uncommon to see a Corgi running across a baseball field, especially during a major game, so I thought this would be an intriguing scenario. I created the mask for the Corgi using Meta AI's 'segment-anything' and employed the previously described approaches for the multi-resolution blending.

Input 1

Input 2

Irregular mask

Result