CS 194-26 Project 2: Fun With Filters and Frequencies

1.1 Finite Difference Operator

Using "gradient" filters to find edges:

Original image $I$ :

Filters:

Applying $D_x$ and $D_y$ to create partial derivative images $\frac{\partial I}{\partial x}$ and $\frac{\partial I}{\partial y}$ :

We can see that edges exist where the magnitude of the derivative is large. However, these derivative images only help us find edges for one dimension. To combine the magnitude information of both X and Y axes, we can use the magnitude of the gradient:

|| \nabla I || = \sqrt{\bigg(\frac{\partial I}{\partial x}\bigg)^2 + \bigg(\frac{\partial I}{\partial y}\bigg)^2}

Which gives us the following image $M$ :

We can create a threshold image $T$ that thresholds each pixel at $i, j$ in the following manner:

T[i,j] = \begin{cases} 1 & M[i, j]>t \\ 0 & \text{otherwise} \end{cases}

Using $t = 0.22$ we get the following image $T$ :

The edges are very clear, but they look broken up and don't quite cover all the important details of the man and the camera. Perhaps we can try to clean up the image first before finding the edges. We can apply a gaussian filter to blur as a low pass filter to block out noise, which usually has high frequency.

Gaussian filter $G$ with with $\sigma = 2$ and kernel size of 6 (from now on I will use a kernel size of $3\sigma$ for every gaussian filter):

Convolving the original image with the filter ( $I * G$ ), we get the blurred image $I'$ :

Applying the same steps (derivative filter, gradient magnitude, thresholding) on $I'$ we get the following results:

Image order: $\frac{\partial I'}{\partial x}$ $\frac{\partial I'}{\partial y}$ $|| \nabla I ||$ and final threshold image with $t=0.05$ :

The edges look much more well-defined and the extra edges from the grass can be removed without losing details on the camera man!

1.2 Derivative of Gaussian Filter

We can combine the blurring and partial derivative steps into a single filter using the associative properties of convolutions.

Before, to get $\frac{\partial I'}{\partial x}$ and $\frac{\partial I'}{\partial y}$ , we had first blurred the image and then applied the derivative filter:

\frac{\partial I'}{\partial x} = (I * G) * D_x

Using the associative property, we can rearrange the steps:

\frac{\partial I'}{\partial x} = (I * G) * D_x = I * (G * D_x)

We can first apply the derivatives to the gaussian in both x and y axes and then apply these gaussian derivatives to the image to get partial derivatives of the blurred image. This way, we only need to apply a convolution to the image (which is expensive) once rather than twice.

Derivative of gaussians $G_x$ and $G_y$ , $\sigma=2$ :

Applying $G_x$ and $G_y$ to the original image, then finding the gradient magnitude and applying a threshold (0.05), we get the exact same results as the end of 1.1!

2.1 Image Sharpening

If we capture the high frequencies of an image and add more of it back to the image, we can achieve a "sharpening" effect.

An easy way to apply a high pass filter to an image is to first apply a low-pass filter (e.g. with a gaussian filter $G$ ) and then subtract the result from the original image

H = I - I * G

We then add $H$ back to the original image:

I + \alpha H

Where $\alpha$ can be set to a custom value. We can derive a sharpening filter by rearranging the equation:

\begin{align*} I + \alpha H &= I - \alpha(I - I * G) \\ &= (1 + \alpha)I - \alpha I * G\\ &= I * ((1 + \alpha)e - \alpha G)\end{align*}

Results on taj.jpg : Original, using subtracted high pass filter, using sharpening filter.

$\sigma=2, \alpha=5$

We see that the subtle details are emphasized by making the colors more extreme. If we look at the bars on the left side of the Taj Mahal, we see that the sharpened images turn the bar from a brownish gray to black, so they become more noticeable. Similarly, the shadows also become much darker.

As a test, we can apply the sharpening filter to a pre-blurred version of taj.jpg. We see the same effect where the colors become more extreme, giving the illusion that the image is less blurry, but looking closely, it is still lacking in actual resolution.

Trying the sharpening filter on other images (original, after):

$\sigma=2, \alpha=1$

$\sigma=3, \alpha=3$

$\sigma=4, \alpha=2$

For the Campanile and Wheeler Hall, adding the sharpening filter does not add much benefit to the image, but in the case of the Big C, where the picture is blurry, applying the filter does add some benefit.

2.2 Hybrid Images

Hybrid images are images that look different at various distances. When up close, you would see one image, but when far away (i.e. meters away), you would see a complete different image. The way these images are constructed is by:

Aligning the image (e.g. the eyes of two people)

Applying a high pass filter for the image you want to see close up and a low pass filter on the image you want to see far away

Averaging the filtered images together

Normalizing the scale of the final image.

To apply a low-pass filter, we just convolve a gaussian, while for a high-pass filter, we subtract the low-passed image from the original, just like in 2.1. Note that the gaussian filter is parameterized by $\sigma$ instead of frequency. A larger $\sigma$ leads to a larger blur, so a low-pass filter with a lower cutoff frequency, so $\sigma$ and frequency are inversely related.

An important detail to mention is that the cutoff frequency for the high-pass filter $f_h$ and the low-pass filter $f_l$ need to be different. In particular, $f_h > f_l$ . Thus, in my case $\sigma_h < \sigma_l$ . I found that $\sigma_l$ is about 2x larger than $\sigma_h$ for the best effect.

Results (hybrid image, low-passed image, high-passed image):

$\sigma_l=15, \sigma_h=7$

Fourier transformed derek.jpg and nutmeg.jpg

$\sigma_l=8, \sigma_h=5$

$\sigma_l=12, \sigma_h=3$

2.3 Gaussian and Laplacian Stacks

Suppose we want to blend two images together, e.g. an apple and an orange. Instead of just taking an interpolation between two images, we can do a "smarter" blend between various frequencies of the images. To create bands of frequencies we need to create a laplacian stack.

A laplacian stack can be created from a gaussian stack. At each layer, a stronger gaussian filter is applied to the image. This can be achieved by repeatedly applying a gaussian filter to the same image and saving out each iteration:

I * G * G * G * ...

This would lead to a linear increase in $\sigma^2$ of the final gaussian filter.

To make the increase in blur faster than linear, I also tried convolving the original gaussian filter with itself before applying it to the next layer, essentially doubling $\sigma^2$ at each layer. I call the former way of creating the gaussian stack as linear and the latter square.

The obtain the laplacian stack, we subtract the each layer by the layer under (so the one with more blur). If we think in terms of frequency, the lower layers contain less higher frequencies, so subtracting a layer by the one below produces an image with a specific band of frequencies.

To blend images we use the following procedure:

Create laplacian stacks of the two images

Create a gaussian stack of the mask

For each layer of the laplacian stack, apply the corresponding layer of the gaussian stack of the mask to the images to create a blended laplacian stack.