Project 3: Fun with Frequencies and Gradients!

About

In this project, we explore different ways of analyzing an image, namely from the frequency and gradient domains, through image-blending.

Part 1.1: Image Sharpening

Shown is a blurry dog and the result after applying the unsharp masking technique:
img + alpha * (img - Gaussian_blur(img)).
A Gaussian with sigma = 1.5 was used. The best observed value of alpha was 20, in which the image is sharpened, but with added noise.

Original

Sharpened

Part 1.2: Hybrid Images

In this part, two images were taken, aligned, and then averaged together after applying a low-pass filter for one image (essentially blurring it), and a high-pass filter for the other (keeping high frequencies). The last row of images shows the images of John Denero, Paul Hilfinger, John filtered, Paul filtered, and their combination, respectively, in frequency space.

John

Paul

John filtered

Paul filtered

Combined

Part 1.3: Gaussian and Laplacian Stacks

In this part, a Gaussian and Laplacian stack were created for two images. A Gaussian stack is a stack of images created by repeated applying a Gaussian filter over the previous image in the stack. A Laplacian stack is created by taking a Gaussian stack and taking the difference between each adjacent image in the stack. the last Gaussian is kept a the bottom of the Laplacian stack.

Part 1.4: Multiresolution Blending

The idea in this part is to create a seamless blending to combine two pieces of two images. The algorithm is simple:

  1. Create a Laplacian stack for both images.
  2. Create a Gaussian stack for the mask.
  3. For each level of the Laplacian and Gaussian stacks, combine the layer: GM * LA + (1 - GM) * LB, where LA and LB are the Laplaicans of image A and B, respectively, and GM is the Gaussian of the mask.
  4. Combine the resulting stack by adding up the layers.
  5. To do this for a color image, do each color channel individually.

Part 2.1: Image Reconstruction

In this toy problem, we recreate an image by constructing and solving a system of equations using information about it's x- and y-gradients and an intensity constraint.
We can construct a matrix system, Ax = b, as follows:

  • For x-gradients:
    A[e, coord_to_index(row, col)] = -1 A[e, coord_to_index(row, col + 1)] = 1
    b[e] = source[row, col] - source[row, col + 1]
  • For y-gradients:
    A[e, coord_to_index(row, col)] = 1
    A[e, coord_to_index(row + 1, col)] = -1 b[e] = source[row, col] - source[row + 1, col]
  • For intensity constraint: A[0, coord_to_index(0, 0)] = source[0, 0]

where e is the equation counter, coord_to_index is a function that maps a row and column to the corresponding flattened index.

Original

Reconstructed

Part 2.2: Gradient-Domain Image Blending

Finally, the best part of the project, gradient-domain blending. In this part, a source image is blended into a target image in a mask area. Inside the mask, we want the gradient of output image to be as close as possible to the source image. In addition, we impose the constraint that the pixels on the edge (and everywhere else) has to be the same as the target image. This can be expressed as a system of linear equations, which we can solve. The following equation is minimized using least-squares linear regression:

Gallery

Shown below are multiple examples of gradient-domain blending. Included are the source, target, naive copy-and-pasted, and gradient-domain blended images.

Seahorse

Moderate Candidate

They Return

Dreams of Oski

Hiking Companion

Dog in Bliss

San Francisco

Comments

As we can see from "Moderate Candidate", gradient-domain blending works much better than multiresolution blending. This is because when swapping faces, one of the most important aspects for a seamless blend is a consistent skin color, something that gradient-domain blending does really well with. Instead of transitioning colors, it completely changes the color of the source image to match the target. Instances like this, when we want the color of the source to directly match the target, is where gradient-domain blending will work much better than multiresolution.

Hilary

Trump

Naive

Multiresolution

Gradient-Domain

So when does gradient-domain blending not work so well? Looking at "Dreams of Oski", we see that perhaps simply performing the naive copy-and-paste would have yielded better results. This is a bad situation for using gradient-domain blending because this time, we do not want the colors to bleed over from the target image to the source. It is because of this, due to the sky background, that we end up with a ghost-like image of Oski. In addition, there is a large contrast between Oski's original background and the target background.

Naive

Gradient-Domain