CS 194-26: Image Manipulation and Computational Photography

Image Frequencies and Gradients

Florin Langer, Fall 2018


Overview

I explore various frequency- and gradient-manipulation techniques to modify images.


Frequency Domain

Unsharp Masking

I perform a Gaussian implementation of a low-pass filter over the image and then subtract a weighted version of that from the original image to extract the highest frequencies. I then add that to a weighted version of the original image to get a sharpened version. This can be expressed as beta*image + (image - alpha*gaussian_blur(image, kernel, sigma)).

Images Being Processed with [kernel], [alpha], [beta], [sigma] (Where Applicable)
Blurred (9, 9), NA, NA, 30
Blurred (9, 9), NA, NA, 100
High-Frequency (9, 9), .5, NA, 30
High-Frequency (9, 9), 1, NA, 30
Blurred (27, 27), NA, NA, 30
Blurred (27, 27), NA, NA, 100
High-Frequency (27, 27), .5, NA, 30
High-Frequency (27, 27), 1, NA, 30
Sharpened Images with [kernel], [alpha], [beta], [sigma]
Unsharpened NA, NA, NA, NA
Sharpened (27, 27), .5, .5, 30
Sharpened (9, 9), .5, .5, 30
Unsharpened NA, NA, NA, NA
Sharpened (27, 27), .5, .5, 30
Sharpened (9, 9), .5, .5, 30

Hybrid Images

I align two images using two corresponding points on each, apply a Gaussian implementation of a low-pass filter to one and the complement as a high-pass filter to the other, adding the results and clipping the resulting image's pixels to values between 0 and 1. As follows, I perform frequency analysis by showing the logarithmic magnitude of the Fourier transform of the two input images, the filtered images, and the hybrid image, all converted to black and white first, per the Python formula np.log(np.abs(np.fft.fftshift(np.fft.fft2(gray_image)))).

Hybrid Results with [First Image Pass Type, First Image sigma, Second Image Pass Type, Second Image sigma]
Derek
Nutmeg
[Low, 2, High, 3] (Aligned on Pupils)
[Low, 5, High, 10]
[High, 3, Low, 2]
Deadpool
Deadpool Unmasked
[Low, 2, High, 3] (Aligned on Eyes) Failure
Farmer
Trump
[Low, 2, High, 3] (Aligned on Nose and Waistline)

The Deadpool hybrid is a failure due to mismatched scale and angle of the face.

Experimenting with color for the low- and high-frequency components, the best result in my samples comes from using more color for the high-frequency component than for the low-frequency one since the fully colored background can overpower the foreground. This can be seen in the Farmer-Trump Hybrid above.

Frequency Analysis
Farmer
Farmer (Low-Pass) sigma=2
Trump
Trump (High-Pass) sigma=3
Hybrid

Gaussian and Laplacian Stacks

I implement a Gaussian and a Laplacian stack, reapplying the respective filter at each level. For the Gaussian stack, this means applying the Gaussian filter to the previous level with sigma as 2. For the Laplacian stack, this means taking the difference of every pair of Gaussian images. I proceed to apply those stacks to images that contain structure in multiple resolutions.

Stacks
Gaussian Layer 1
Gaussian Layer 2
Gaussian Layer 3
Gaussian Layer 4
Gaussian Layer 5
Laplacian Layer 1
Laplacian Layer 2
Laplacian Layer 3
Laplacian Layer 4
Laplacian Layer 5
Gaussian Layer 1
Gaussian Layer 2
Gaussian Layer 3
Gaussian Layer 4
Gaussian Layer 5
Laplacian Layer 1
Laplacian Layer 2
Laplacian Layer 3
Laplacian Layer 4
Laplacian Layer 5

Multiresolution Blending

I first create Laplacian stacks L1 and L2 of the two images to be blended. It is important to note that the first layer is the original image minus the first layer of the corresponding Gaussian stack and the last layer is just the last element of the corresponding Gaussian stack so that adding all layers results in the original image. I then create a gradient mask and the Gaussian stack LG of that. I use the formula LS[i] = LG[i]*L1[i] + (1 - LG[i])*L2[i] on each layer i and sum them up to get the result.

Stacks
Laplacian Layer 1
Laplacian Layer 2
Laplacian Layer 3
Laplacian Layer 4
Laplacian Layer 5
Laplacian Layer 6
Laplacian Layer 1
Laplacian Layer 2
Laplacian Layer 3
Laplacian Layer 4
Laplacian Layer 5
Laplacian Layer 6
Gaussian Layer 1
Gaussian Layer 2
Gaussian Layer 3
Gaussian Layer 4
Gaussian Layer 5
Gaussian Layer 6
Blended Images
Apple
Orange
Apporange
Heath Ledger
Joseph Gordon-Levitt
Heatheph Gordon-Ledger
Kavanaugh
Rubber Duck
Rubber Kavanaugh
It is apparent that the irregular mask above does not perform as well as the vertical seam due to the superimposed image background not matching the other image. The following section should noticeably rectify this.

Gradient Domain Fusion

In this section, I aim to seamlessly blend an object or texture from a source image onto a target image. Naively copying and pasting raw pixels from one image onto another will show stark boundaries between the two. Instead, I search for values for the pixels in the target (background) image that maximally preserve the gradient of the source region without changing any of the background pixels, ignoring the overall intensity since gradient is much more noticeable.

Toy Problem

Image Reconstruction

I compute the horizontal and vertical gradients of an image and then use all gradients plus one pixel intensity value (that of the top left corner) to reconstruct the image by a least-squares approximation. I pay special attention to data types, ensuring all numbers being operated on are of type np.float32.

Original
Reconstructed

Poisson Blending

I use Adobe Photoshop to create a scaled and positioned version of the source image to correspond to the target image. I then create an appropriate mask. I then perform Poisson Blending on each image band separately (but concurrently, using Python's multiprocessing.Pool.starmap function) and solve the blending constraints that take into account the four pixels surrounding every pixel in the source region that falls within the mask region in the target using a least-squares approximation. I lastly copy the solved values into the target image.

I previously tried to programatically create a positioned and scaled version of the source image and the corresponding mask, but bugs prevented me from doing so successfully in an appropriate amount of time.
Blending Process
Source Overlaid on Target
Positioned and Scaled Source
Mask
Blended
Source Overlaid on Target
Positioned and Scaled Source
Mask
Blended
Source Overlaid on Target
Positioned and Scaled Source
Mask
Blended Failure
Source Overlaid on Target
Positioned and Scaled Source
Mask
Blended
Source
Target
Mask
Blended Failure

The Trump-Kim meeting and parrot salad are failed examples because the intensities are actually important in these cases. The parrots look like ghosts in the salad, not so much a part of it. The conveniently placed tomatoes in the salad also make it look like you can see the top two parrots' brains.

Blending Comparison
Source
Target
Mask
Multiresolution-Blended
Poisson-Blended

The Poisson-blending technique works best for the above set of images due to the edges of the source region matching the gradient of the target image. It is way more obvious that the Multiresolution-blended image is superimposed due to that. However, multi-resolution blending could be useful in a situation in which the source image has a solid background or in the example in which the source and target images correspond at the seam such as in the case of the orange and apple and the faces I blend above.

This project has shown me important techniques in computational photography that I feel would actually be useful in many instances, such as combining shots from different scenes to create an image for an article.