Computational Photography, Fall 2020

Fun with Filters and Frequencies

Avni Prasad, CS194-26-aej



Overview

This project explores many different transformations on images. From sharpening and straightening images to blending images together, we begin to build an intuition for what convolutions, frequencies, transformations, and filtering means for images.

Part 1.1: Finite Difference Operator

We begin by using the finite differences operator on the image above as a filter to help us understand the changes in x and y directions. By convolving the cameraman image with dx = [1, -1] and dy = [[1], [-1]], we are able to get the partial derivatives of the image. The way I like to think about this is suppose this image was instead braille-like. Getting the partial derivatives with respect to x would be like brushing your hand across the image horizontally versus with respect to y, it would be brushing your hand across the braille-like image vertical. The output of these derivatives are the textures you would expect to feel. After calculating the partial derivatives of the image, I calculated the gradient magnitude (sum of the squares of the derivatives), and binarized this result to get a clear picture of the edges at play in the image.

X Gradient Image
Y Gradient Image
Magnitude Gradient Image
Binarized Gradient Image

Part 1.2: Derivative of Gaussian (DoG) Filter

While in the images above, we do begin to see the edge detection we are looking for, we also notice that there is a lot of noise being picked up. To combat this, we look to the Gaussian filter, which can provide a smoothening effect on the images. Continuing with the braille-like image methaphor, we can think of the Gaussian filter as a sandpaper, it sands out the random bumps that you may have originally felt in your braille-like image, so that the textures that you do feel are smoother and more intentional. There are two ways to use the Gaussian filter.

Method 1

We can convolve the original image with the Gaussian filter and then compute the derivatives, magnitude gradient, and binarized the gradients.

Method 2

We can compute the derivatives of the Gaussian filter and using these to convolve with the original image to compute the magnitude gradient and binarized the gradients. I include what the derivatives of the Gaussian filter would look like below.

X Gradient Filter
Y Gradient Filter

Both of these methods as I show below will result in the same output. At times, one method may be computationally more effective than the other. Below I have also included the original edge detection (without the Gaussian filter) to demostrate the massive improvement by introducing the Gaussian filter.

Without Gaussian filter
Method 1
Method 2

Part 1.3: Image Straightening

In this section, we use many of the same concepts as in part 1 and 2 for automatic rotation. Similarly how we used derivatives in the last part to calculate the gradient magnitude, here we use those derivatives to calculate the gradient angles of the image. We assume that images that are straighter are likely to have more right and straight angles. Using this assumption, we calculate the distribution of gradient angles at every rotation angle between [-10, 10] to determine what the best angle for the image in.

Facade

Original Image
Straightened Image
Original Gradient Angles
Straightened Gradient Angles

Dessert

Original Image
Straightened Image
Original Gradient Angles
Straightened Gradient Angles

Village

Original Image
Straightened Image
Original Gradient Angles
Straightened Gradient Angles

Bridge

Original Image
Straightened Image
Original Gradient Angles
Straightened Gradient Angles

Sunlight

Failure Case: Unforuntely, the algorithm for this photo of the sunlight refracting off the tree into various angles. The image itself was fairly straight before, relative to the tree. However, my straightening function rotated it to align with the most horizontal and vertical edges/lines it could find. In this case, that was not a good assumption to make, as the light is going in all different directions

Original Image
Straightened Image
Original Gradient Angles
Straightened Gradient Angles

Part 2.1: Image "Sharpening"

In this section, given an image, we want to mimic sharpening the image as best as we can, without any additional information about the images. To do this, we use the same Gaussian filter as a way to blur the image. Convolving the original image with the Gaussian filter gives us a smoothed image which we then can subtract from the original image to get an image the represents the "details" of the original image. Using this details image, we can create a sharpened image = original image + alpha*details_image

Taj Mahal

Original Image
Smooth Image
"Sharpened" Image

Music

Original Image
"Sharpened" Image

Art

For this example, I start with an already sharp image of the art work on walls. The sharpened version of the image actually brings out some of the noise we might not want to see.

Original Image
"Sharpened" Image

Part 2.2: Hybrid Images (with Bells and Whistles)

These hybrid images look like one image from up close but look like another from afar. In order to achieve this effect, I took two images, aligned them based on the eyes, and then combined the high frequency image of one and the low frequency image of the other. The high frequency image is more visible from up close, and the low frequency image is more visible from afar. I implemented my hybrid images with color from the start (Bells and Whistles).

Dr. Watson meets Joker

Monroe meets Einstein

For this hybrid example, I found the log magnitude of the Fourier transform for the input images and the hybrid image (displayed respective to the images above).

Part 2.3: Gaussian and Laplacian Stacks

Finally, I implemented a Gaussian and Laplacian stack to detect the underlying images functioning at different frequencies in a given photo. I iterated through 5 levels. At each level, I applied a Gaussian filter to the image and calculated the respective Laplacian by comparing the image at level i-1 in the Gaussian stack and level i

Lincoln

Original Image
Original Image

Jatson or Woker

Original Image
Original Image

Part 2.4: Multiresolution Blending

Using the Gaussian and Laplacian stacks function from the last part, we are able to leverage these stacks to produce well-blended images. Following techniques illustrated in Burt and Adelson's 1983 paper on Multiresolution Spline, we can use Laplacian stacks for both images and a Gaussian stack for a mask image to create a blended image.

Orapple

Image 1
Image 2
Blended Image
Mask

Jump over cameraman

Image 1
Image 2
Blended Image
Mask

Bells and Whistles

I implemented color for these images the parts above. My favorite part of this project was being able to visually see the transformations introduced. In the past, I have used FFTs, low and high-pass filters, rotations, but I really feel that a project like this helps someone truly understand the meaning behind these operations. With visually seeing each computation, the operations truly come to life!