Fun with Frequencies & Gradients

Nancy Li



Overview

This project explores frequency domain and gradient domain image processing tools to blend together some fun images!


Warmup

To sharpen an image, the high frequency components of an image is extracted and added back to the original: original + c*high frequency. High frequencies are found by subtracting the low frequencies (found with the Gaussian filter).


Card image cap

Taylor Swift from her Fearless era.

Card image cap

Taylor, sharpened.

Card image cap

Highly sharpened Taylor.


Hybrid Images

Hybrid images are strange images that seem to change as a viewer gets farther and farther away in distance. High frequency will dominate perception when a viewer is close, but at a distance, the low frequency part of the signal dominates. We can create a hybrid image by blending the high frequency portion of one image with the low frequency portion of another. To achieve the best results, pairs of images are aligned before averaging is done.


Card image cap

A stern former President Obama.

Card image cap

A happy Obama.

Card image cap

Smiling up close, frowning from afar

Card image cap

FFT of high-freq

Card image cap

FFT of low-freq

Card image cap

FFT of low pass

Card image cap

FFT of high pass

Card image cap

FFT of hybrid

Card image cap

Peter from Narnia

Card image cap

Aslan

Card image cap

Success!

Card image cap

FFT of high-freq

Card image cap

FFT of low-freq

Card image cap

FFT of low pass

Card image cap

FFT of high pass

Card image cap

FFT of hybrid

Card image cap

Gandalf from Lord of the Rings

Card image cap

An eagle!

Card image cap

Blending failure b/c of bad alignment

Card image cap

FFT of high-freq

Card image cap

FFT of low-freq

Card image cap

FFT of low pass

Card image cap

FFT of high pass

Card image cap

FFT of hybrid


Gaussian and Laplacian Stacks

Gaussian stacks are made by repeatedly applying a gaussian filter over the source image. The Laplacian stack is built in the same way using the Gaussian filter. These stacks are applied to interesting images such as Davinci's Mona Lisa and Salvador Dali's Lincoln and Gala that contain multiple resolutions. We see Lincoln/Gala and Mona Lisa's smile/frown emerge in different stacks!


Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap
Card image cap


Multiresolution Blending

Multiresolution blending computes a seam between the two images at each band of image frequencies, resulting in a much smoother seam. A mask is used to select the area of one image to blend with the other.

In this section, we use the Laplacian and Gaussian stacks from the previous section. For both images A and B, Laplacian and Gaussian stacks are necessary. The mask needs a Gaussian stack as well.

First, A and B's Laplacian stacks are added together. Then, using a single combined Gaussian image of both A and B, we can rebuild the original combined image, successively adding the nth Gaussian blend with the n-1st Laplacian blend. Below are the results.


Card image cap

Original orange image

Card image cap

Apple

Card image cap

Naive cut and paste (see below section)

Card image cap

Multiresolution blending

Card image cap

A blend of Harry Potter and his nemesis

Card image cap

A failed blend, due to bad alignment

Card image cap

A horse with an eagle head? (A hippogriff)


Project Part 2

This project explores gradient-domain processing, aiming to blend an object from a source image into a target image. The simplest method to do that would be to copy and paste pixels from one image onto the other, but this creates very noticeable differences around the seams, no matter how well the backgrounds match. How can we get rid of the seams without changing up the source region copied into the new image?

Because people often notice the gradient of an image more so than the overall intensity, we set up this problem as finding values for the target pixels that maximally preserve the gradient of the source region.

Toy Problem

Gradients work like 2D derivatives, and in this section we solve for best fitting values to pixel gradients to recreate a tiny Toy Story image. The left image is the original, and the right image is the copy.

Card image cap Card image cap

The result requires solving a system of matrix equations, Ax = b. The first matrix A is a sparse height*width by height*width matrix, ensuring each pixel index has its own variable. The second matrix b is a column vector of length height*width. Solving for x will give us the pixel values we want for the best blend. The matrices are filled by iterating through the entire image; if the mask is 0 at a pixel, matrix A gets a 1 at that pixel's index, and b gets the value of the target image. If the mask is a 1, then we must look at the source image's gradient in 4 directions.

Matrix A gets a 4 at that pixel index, and -1, -1, -1, -1 at the four neighbor pixel indices x-1,y; x+1,y; x,y-1; x,y+1. Matrix b gets 4*source(x, y) - source(x+1, y) - source(x-1, y) - source(x, y+1) - source(x, y-1), where x, y are indices.

Using the sparse matrix solver, this algorithm can successfully run under a minute even on large 2000 by 2000 pixel images.


Poisson Blending

Below are the results that the algorithm produced! Included are source and target images, as well as the naive cut & paste result, and the result of the blending algorithm.


Card image cap
Card image cap
Card image cap
Card image cap
Card image cap

Card image cap
Card image cap
Card image cap

Card image cap
Card image cap
Card image cap
Card image cap



Also shown is the same Harry, Voldemort image used for multiresolution blending. The Poisson blend changes Voldemort to warm colors, while the MR blend (shown earlier) lets Voldemort stay bluish.

I think Poisson blending is useful when you want the colors of the target and source images to match in the blend, while multiresolution blending is more useful in cases when the transition of the seam matters.


Card image cap
Card image cap
Card image cap
Card image cap

The failure case is the image with Dobby the Elf in our CS 194 classroom--the algorithm changed Dobby's gradients too much to match the eclectic background of the chairs and doors. So he's a little seethrough, and doesn't appear to be solid.


Card image cap
Card image cap
Card image cap
Card image cap


HTML/CSS for this page was taken from https://v4-alpha.getbootstrap.com/components/card/