Fun with Frequencies!

Ian Markowitz

Overview

For this project, we used filtering and the frequency domain in order to analyze, combine, and blend images. All parts of this project required a gaussian filter; I used a Gaussian with size 21, and a default sigma of 3.3. I wrote a function to generate a Gaussian given a certain size and sigma value, and a second function in order to convolve two images.

In order to convolve images efficiently in python, I use the numpy functions "as_strided" and "einsum". As strided provides a new "view" into an image, in which I can essentially add a new dimension to the image in order to properly align it with the image it is being convolved with. The "einsum" function implements an einstein summation, where two or more arrays are multiplied along specified dimensions and summed along others. With this approach, I can convolve two images in two lines of code, without using any loops.

The coolest part of this assignment was being able to blend images together seamlessly. Its interesting that the simple technique we use here can have such powerful results.

Part 0: Warmup

Part 0 of this assignment was image sharpening. A blurred version of an image can be created by convolving the image with a Gaussian. Let "image" be the original image and "blurred" be the image convolved with a Gaussian. To get a sharpened image, we perform

image + 0.5 * (image - blurred)
The effect is illustrated on Oski below:

Our beloved mascot, Oski the bear
Oski blurred with a Gaussian filter
Oski sharpened by subtracting the Gaussian filter from the original image

Part 1: Hybrid Images

To create hybrid images, we lowpass filter one image and highpass filter the other. The low-frequency image is created by convolution with a Guassian of sigma 8. The high-frequency image is created by making a low-frequency image (convolve with a Gaussian of sigma 4), and subtracting it from the original image.

Only the low-frequency image is visible from a far distance, while the high frequencies are more noticable from up close.

For this part of the project, I combined Oski, a mountain lion, and Bernie Sanders with an image of Chancellor Dirks. I used the provided script in order to align images. The Oski image was not very successful, because the two head shapes do not align very well. However, the other two images were quite effective. With Sanders, I applied a highpass filter to Dirks and with the mountain lion I applied a lowpass filter to Dirks and a highpass filter to the mountain lion.

The fourier transform of the low-frequnecy image is bright along the diagonals and in the center, while the high-frequency fourier transform has a noticable dark spot in the center.

An aligned image of chancellor Dirks
An aligned image of Senator Bernie Sanders
Sanders with a Gaussian filter applied
Dirks with the low frequencies removed
The Dirks-Sanders hybrid
Fourier transform of the original Dirks image
Fourier transform of the original Sanders image
Fourier transform of the high-frequency Dirks image
Fourier transform of the low-frequency Sanders image
Fourier transform of the hybrid image
A mountain lion
Mountain lion and Dirks hybrid
Oski the bear
Oski and Dirks hybrid

Bells and Whistles: Color

I discovered that it did not look so good to use color for the high-frequency edges, but that color worked very well in the low-frequency background image. Convolution is not so different in color than in black-and-white; An identical Gaussian is convolved with each color channel.

Low-frequency Bernie Sanders in color
Hybrid of Dirks and Bernie, with Bernie in color

Part 2: Gaussian and Laplacian Stacks

I created Gaussian and Laplacian stacks of two images: the Dali painting "Gala Contemplating the Mediterranean Sea," and the hybrid image of Dirks and Sanders from above. In order to create a Gaussian stack, we convolve an image with a larger and larger Gaussian. I started with a sigma of 2, and doubled the sigma at each level of the stack. A laplacian stack is created by taking the difference between adjacent images in the Gaussian stack. The final image of the laplacian stack is merely the final image of the Gaussian stack. In the Dali painting, moving down the Gaussian stack, we see Gala slowly disappear and Lincoln take her place. In the Laplacian stack, Gala is clearly visible in the first two levels, and Lincoln can only be clearly scene on the last level.

For the hybrid image, the first four levels of the Laplacian stack show Dirks and the final level shows Sanders. The fifth level is empty, because those intermediate frequencies have been filtered out of the images.

Gala contemplating the Mediterranean sea, by Salvador Dali
Image 1 of the Gaussian stack for the Dali painting
Image 2 of the Gaussian stack for the Dali painting
Image 3 of the Gaussian stack for the Dali painting
Image 4 of the Gaussian stack for the Dali painting
Image 5 of the Gaussian stack for the Dali painting
Image 6 of the Gaussian stack for the Dali painting
Image 1 of the Laplacian stack for the Dali painting
Image 2 of the Laplacian stack for the Dali painting
Image 3 of the Laplacian stack for the Dali painting
Image 4 of the Laplacian stack for the Dali painting
Image 5 of the Laplacian stack for the Dali painting
Image 6 of the Laplacian stack for the Dali painting
Image 1 of the Gaussian stack for the Bernie-Dirks hybrid image
Image 2 of the Gaussian stack for the Bernie-Dirks hybrid image
Image 3 of the Gaussian stack for the Bernie-Dirks hybrid image
Image 4 of the Gaussian stack for the Bernie-Dirks hybrid image
Image 5 of the Gaussian stack for the Bernie-Dirks hybrid image
Image 6 of the Gaussian stack for the Bernie-Dirks hybrid image
Image 1 of the Laplacian stack for the Bernie-Dirks hybrid image
Image 2 of the Laplacian stack for the Bernie-Dirks hybrid image
Image 3 of the Laplacian stack for the Bernie-Dirks hybrid image
Image 4 of the Laplacian stack for the Bernie-Dirks hybrid image
Image 5 of the Laplacian stack for the Bernie-Dirks hybrid image
Image 6 of the Laplacian stack for the Bernie-Dirks hybrid image

Part 3: Multiresolution blending

Multi-resolution blending is implemented by creating a laplacian stack of two images and a Gaussian stack of a mask image. At each level, we take corresponding images from each stack. We multiply one image by the mask at that level of the stack, and the other image by one minus the mask (the mask's inverse). Then, we sum up the values from each level of the stack.

Since the mask is Gaussian blurred, lower levels of the stack will have a higher amount of blending between images.

I attempted three blendings. The first is of the rappers Gucci Mane and Lil Wayne. The second is of Senator Bernie Sanders and Chancellor Dirks. The final one uses an irregular mask to give me a third eye on my forehead. The blending of the two rappers is a partial success, and the blending of the eye was highly successful. Results were best when colors in the images lined up and there was no conflict between large-sized features of the images.

I sought to recreate the effect on this mixtape cover, except using the rapper Lil Wayne rather than Young Thug
This is the result image. I cropped the bottom half of the image because the nose and mouth did not line up so well.
The original image of Gucci Mane (right half)
The original image of Lil Wayne (left half)
The full result image.
I used a simple mask image with one half black and the other half white.
The first level of the laplacian pyramid of the Gucci Mane image
The second level of the laplacian pyramid of the Gucci Mane image
The third level of the laplacian pyramid of the Gucci Mane image
The fourth level of the laplacian pyramid of the Gucci Mane image
The ffifth level of the laplacian pyramid of the Gucci Mane image
The sixth level of the laplacian pyramid of the Gucci Mane image
Level 1 of the laplacian pyramid of the Wayne image
Level 2 of the laplacian pyramid of the Wayne image
Level 3 of the laplacian pyramid of the Wayne image
Level 4 of the laplacian pyramid of the Wayne image
Level 5 of the laplacian pyramid of the Wayne image
Level 6 of the laplacian pyramid of the Wayne image
This image is a blend of Bernie Sanders and Chancellor Dirks. Some parts of the image blended together better than others. Overall, I would consider this image a failure because there is a clear seam between the two halves. The skin tone was very different for the two images and some of the features did not line up very well (hair, mustache, glasses)
Finally, I did a blend on this image of my face, using a mask that covers up the eye.
The mask that I used
The second input image. This is a shifted version of the other image, with the left eye positioned over the other image's forehead.
This is the result of combining the images using the mask without doing any blending. An oval border around the eye is clearly visible.
A multiresolution blending of the two images
By simply inverting the mask, the blend removes one of my eyes rather than adding one.

Bells and Whistles: Multiresolution blending in color

Because of the way I wrote my code, it was very easy to work with images in either black and white or color. In order to do an Einstein summation on a color image, we just add another dimension to each image and preserve this dimension through the summation. Thus, all of the blends for part 2 of this project are done in color.