Project 3: Frequencies and Gradients

Brandon Huang, Fall 2017


Part 1

The first part of this project uses frequency-based methods to enhance and blend images.

1.1: Unsharp mask

In this part I implemented an image sharpening technique using the "unsharp mask", which is a high-pass filtering technique obtained by subtracting a low-pass filter from an identity filter. This sharpens the image by accentuating small details in the image, which represent high frequency content.

Sharpening filter shape in 1 dimension. Source: Wikipedia
Prior to sharpening
After sharpening. Seems like a dark shadow tends to form, especially around bright areas of the bird's body. This makes sense given the filter's shape; i.e. the filter responds to high values by making values around them darker.

1.2: Hybrid images

In this part, I implemented hybrid images as described in a 2006 paper by blending low frequencies from one image with high frequencies from another. This relies on the insight that perception is dominated by high frequency signal when possible, but only low frequencies are available when the image is small or far away. This insight can be used to create images which are perceived differently at different viewing distances.

Example 1: Wolf person (with frequency analysis)

Image to low-pass. My friend who graciously decided to volunteer.
Frequency spectrum. A vertical bar seems to indicate lots of vertical edges (hair?)
Low-passed image.
Much of the high frequency information is lost.
Image to high-pass. I chose an animal with a lot of fur (wolf) to get a lot of high-frequency information.
Less concentrated on low frequencies than the first image (as expected for this furry animal).
High-passed image.
Biased much more toward high frequencies.
Hybrid image. High and low frequencies separated very cleanly.
This frequency spectrum looks like that of the initial images.
Seems like it has similar characteristics to unhybridized images.

Example 2: Sending mixed signals

I'm happy! :)
Blurry happiness.
I'm sad. :(
Sadness in the high frequencies. :(
Hybrid image. Unlike the first example, the frequency content is
similar between the two starting images.
Nevertheless, this method was able to combine them successfully.

Note on color vs greyscale

The main effect I observed of adding color was that the high frequencies stood out significantly more. Below is a side by side comparison of color vs greyscale hybrid images. My guess is that color provides additional information, and perhaps our perception of fine detail benefits a lot from increased information in the image, and our perception of course detail does not.

1.3: Gaussian and Laplacian stacks

I implemented Gaussian and Laplacian stacks to more finely examine images at various spatial frequencies.

Example 1: Lincoln and Gala

Gaussian stack

Original image. Gala is clearly visible.
Gala is barely distinguishable.
The image now looks just like Lincoln.

Example 2: Optical illusion

Gaussian stack

I tried out a different optical illusion to see if the same principle would apply. This illusion has less low-frequency information (solid color patches) to support it and relies mostly on edges and object shapes. I found that while the Gaussian stack still works, the illusion begins to fade as we go deeper in the stack. This is in contrast to Lincoln who stays quite visible as we keep low-passing the image. I'm guessing this is because, since hard edges have signal in all frequencies, the illusion begins to lose strength as we low-pass the image, whereas Lincoln is strong particularly in low frequencies.

Original image. Both the scenery and the hidden face are visible.
The scenery fades. The hidden face is very clear
Only the "hidden" face is now visible, though less than before.

Laplacian stack

The laplacian stack helps to confirm my guess that the scenery is mostly lost in the first few low-pass filtering steps.

First Laplacian layer. There is a lot of scenery information being lost here, as we can see the man, the woman, the swan, and the grass.
The scenery detail appears to already be gone, as we are only losing information about the hidden face now.
Same as the previous.

Example 3: Hybrid image from Part 1.2

Gaussian stack

When I worry about the fine details, I'm pretty sad :(
But as my papa used to say...
If I take a step back and look at the big picture, I'll definitely be happy! :)

1.4: Multi-resolution blending

In this part, I used an alpha channel with varying smoothness of transition, blending the low frequencies more strongly than the high frequencies. This reduces ghosting of fine details while also avoiding hard seams between colors and coarse details.

For all of these examples, I included a baseline analysis with a hard seam (using the original mask) and with uniform blending; i.e. blending all frequencies with the same smoothed mask. Interestingly, the uniform blending performed just as well or even slightly better in all examples. I may have chosen my examples poorly but I expected uniform blending to either exhibit more ghosting, or have harder seams.

Example 1: Oraple

Originals

A nice apple. We should find an orange to blend this with.
An even nicer orange. If only we had an apple to blend this with!

Blending

Hard seam.
Uniform blending.
Multi-resolution blending.

Example 2: Texture blending

Originals

A brick texture with more horizontal than vertical content.
A wood texture with lots of vertical content.

Blending

Hard seam.
Uniform blending. Significant ghosting.
Multi-resolution blending.
A little less ghosting is visible.
Introduced a slight blue pallor at the blended area.

Example 3: Hand eye coordination

Originals

Whatchu lookin' at??
Would be a real shame if an eye sprouted out of this hand...

Blending

Hard seam.
Uniform blending.
Multi-resolution blending.
In this and the uniform blend, a bit of black is visible due to part of an eyebrow being blended in by the filtering operation.

Part 2

In this part, we switch from the frequency domain to the gradient domain. We spend most of our time looking at a technique called Poisson blending, in which a least squares solver is used to find pixel values which smoothly blend the constant value of a patch while preserving the relevant features in the patch.

Example 1: Friends and a flying cat

This example worked pretty well, though a slight seam occurred. Likely this would be resolved by mixed gradients; the source image doesn't account for the slight gradient in the blue of the target image.

What a nice picture. Could really use a flying cat though.
What luck!
No blending.
Blended image. Every picture is better with flying cats. (There is a small seam on the top of the patch)

Example 2: Energy ball

Again, a slight seam due to the nonzero gradient in the target image.

Sad hand holding no magic stuff.
Magic stuff!!
No blending.
Blended image.

Example 3: Magic spell

This one didn't one too well because the source image content didn't have enough buffer on the sides, and so part of the source image's edge contained the glowing light pattern and not the black background. The least squares solver sees a black target image and a white/blue source image, and tries to distribute the error throughout by blacking out parts of the light pattern. It did lead to a cool effect, as if it was not a glowing light but rather an ancient carving or painting on the front of the notebook.

I procrastinated on studying and now I need to memorize 4 month's worth of notes in 1 hour. Is there a magic spell I can use to absorb everything instantly?.
Hmm...
No blending.
Blended image.

Example 4: Frequency vs Gradient Blend

I tried the Poisson blend against one of the examples from the frequency domain section. The frequency domain technique worked better, though I imagine that if I had used mixed gradient blending, the output image would have picked up on the hand texture and actually done better.

No blending.
Frequency domain blending.
Gradient domain blending.