Project 2: Fun with Filters and Frequencies!

Part 1: Fun with Filters

Overview

In this part, we explore the 2D convolutions and Gaussian filtering.

Part 1.1: Finite Difference Operator

First we use the finite difference filters $D_x = \begin{bmatrix} 1 & -1\end{bmatrix}$ and $D_y = \begin{bmatrix} 1 \ -1 \end{bmatrix}$. By convolving the image with the filters separately, we get the partial derivative in x and y of our image. Below is the result.

partial derivative in x
partial derivative in y

Then, to get the gradient magnitude, we use the following formula:

$\lVert \nabla f \rVert = \sqrt{(\frac{\partial f}{\partial x})^2 + (\frac{\partial f}{\partial y})^2}$ The image is below.

gradient magnitude image

To supress the noise in the image, we binarize the image by choosing the threshold to be 0.08 (in grayscale). Then, we get the supressed edge image below.

supressed edge image

Although we include some noise, we include as many edges as possible.

Part 1.2: Derivative of Gaussian (DoG) Filter

This time we use a smooth filter first -- Gaussian filter. We first blur the image with the Gaussian filter (using appropriate size and sigma value), then we repeat the procedure in part 1.1 to get the result shown below.

partial derivative in gauss x
partial derivative in gauss y
gauss gradient magnitude image
guass supressed edge image

You can see that the edges in our new image are much thicker and more smooth, since there is much less noise if we first blur the image with gaussian filter.

Now we try the order of operation in a different way. We first convolve Gaussian filter with our finite difference operators. The result is shown below.

DoG in x
DoG in y

Using these derivatives of Gaussian filters, we can use only one convolution with our image. And the result is below.

DoG supressed edge image

Yes, the result is the same! (I also find the difference between the two matrices, and the difference is 0)

Part 2: Fun with Frequencies!

Overview

In this part, we utilize the high and low frequencies of the image and play with the frequencies to get sharp images, hybrid images, and blended images.

Part 2.1: Image "Sharpening"

To sharp the image, we first subtract the low frequency (blurred image) from the original image to get the high frequency of the image, and then we add the high frequency back to our original image in order to sharpen it. Here, I combine these two steps into one! I use what is called the unsharp mask filter. It's defined in the lecture as below.

formula of the unsharp mask filter

To convolve the color image with the filter, I convolve three color plates (r,g,b) with the filter separately and stack them together to get the convolved color image.

Below is the result of apply unsharp filter on the original images. I use $\alpha = 3$.

original image: taj
sharp image
original image: captain america
sharp image

Here is the comparison between the original sharp image and the sharpened blurred image.

sharp image
sharp image after blurred

The sharpened blurred image is slightly more blurred and slightly less sharp than the original sharp image. This is because after the blurring, some high frequencies of the sharp image is removed, so when we try to sharpen it again, the resulting image may not be as good as the original sharp image.

Part 2.2: Hybrid Images

The first step is to align the two images. Then, we apply lowpass filter on one image and highpass filter on the other image. Finally, we combine the two filtered images together to get our hybrid image. You can see the first image (low frequency) when you are far away and see the second image (high frequency) when you are near it.

Below are some interesting hybrid images.

first image: Derek
second image: cat
hybrid image: catman
first image: da vinci
second image: monalisa
hybrid image: monavinci
first image: prof efros
second image: mr bean
hybrid image: mr efros

Below is the Fourier transform analysis of my favorite hybrid image -- Mr. Efros.

Frequency domains of the input images:

And the rest:

Finally, below is the frequency domain for our hybrid image:

Learn from failure

At first, I did not choose the size and sigma value of my gaussian filter carefully, resulting a bad hybrid image where high frequency image still shows up and low frequency image can not be seen at a very far distance. The bad example is below.

After experimenting and adjusting the size and sigma value parameters, I successfully get a good hybrid image shown several images above.

Bells & Whistles

I use color for convolution, and I get the result below.

first image: color for both freqs
second image: color for low freq
third image: color for high freq

In my opinion, using color for both frequencies gives the best result, since colors give more information of the elements in the image. You can see that using color for low frequency gives very similar result to the one for both frequencies, and using color for high frequency gives very similar result to the one using only grayscale images.

Part 2.3: Gaussian and Laplacian Stacks

Before blending the two images, we need to create Gaussian and Laplacian stacks for help. Because it is a stack, we don't need to downsample the image in every level.

Basically, for Gaussian stacks, I convolve the original image with a Gaussian filter first, and then convolve the resulting image with a Gaussian filter again. I keep doing this to get the five layers of my Gaussian stacks. As for Laplacian stacks, I subtract the next lower layer of Gaussian stacks from the current layer of Gaussian stacks to get every layer of Laplacian stacks. The lowest layer of the Laplacian stacks is the lowest layer of the corresponding Gaussian stacks in order to recreate the original image.

My Gaussian stacks have 5 layers, and each lower layer has size 10 greater and sigma value 2 greater than the previous layer. My starting size is 30 and sigma value is 2. Because of this property, the lower level of the image is more blurred and gives better overall result.

My Laplacian stacks also have 5 layers in total and below are the ones for the apple and the orange. Levels from high to low are in the order from left to right and top to bottom in the figure. Notice that I use min-max normalization to better display the result.

apple Laplacian stacks
orange Laplacian stacks

Bells & Whistles

The colored ones are also included below.

apple colored Laplacian stacks
orange colored Laplacian stacks

Part 2.4: Multiresolution Blending (a.k.a. the oraple!)

Now it's time to blend the two images (apple and orange) together to get the oraple!

We reference the steps in the paper (Spline83, P230) below:

For this blended image, we use a regular half and half mask (as R in the step 1b) below.

Following the steps above, let's recreate the 12 figures of our oraple similar to the book.

the 12 figures

And we get our oraple!

oraple

Bells & Whistles

The colored ones are also included below.

colored version: the 12 figures

And also the colored oraple!

colored oraple

my own selections

I use irregular mask to create my own blended images. The irregular circle mask is below (two different size sof the circles for the following two blended images).

circle mask

Soccer in the sky

soccer
sky
soccer in the sky

And the 12 figures:

the 12 figures

Basoc

basketball
soccer
basoc

Takeaways

From this project, I learn the importance of the Gaussian filter. It is useful for smoothing the edge image, sharpening the image, and you can use it to derive another important filter -- Laplacian filter!

Captain America from https://www.marvel.com/characters/captain-america-steve-rogers

Da Vinci from https://www.biography.com/artist/leonardo-da-vinci

Monalisa from https://en.wikipedia.org/wiki/Mona_Lisa

Mr Bean from https://www.mentalfloss.com/article/62455/15-things-you-might-not-know-about-mr-bean

Professor Efros from https://www2.eecs.berkeley.edu/Faculty/Homepages/efros.html

Sky from https://theconversation.com/curious-kids-what-is-the-sky-157834

Soccer 1 from https://www.latimes.com/sports/highschool/story/2021-05-08/high-school-boys-soccer-southern-section-playoff-pairings

Soccer 2 from https://www.amazon.com/Yanen-Traditional-Recreation-Performance-Different/dp/B07HVBC7PB

Basketball from https://www.poolweb.com/products/pool-shot-full-size-basketball-8-5-inches?variant=34932989526154&gclid=CjwKCAjw4qCKBhAVEiwAkTYsPNFJK01-N2NkBWzzaNdwhG0cUx8EdF6e67X6ntshI41HyzviO2R0LBoC3bQQAvD_BwE