Image Manipulation and Computational Photography

Project 3: Frequencies and Gradients

Jianglai Zhang

Overview

An important goal of image manipulation is to blend multiple images onto one picture. Here we will experiment with several methods to naturally blend images including applying low-/high-pass filters, stacks of laplacian and gaussian masks, and using gradient.

Image Sharpening

First let us try sharpen an image using an 'unsharp masking filter', which contrary to its name, really does the job of sharpening an image. The method works by subtracting a gaussian blurred version of the image from the original image, which generates highlights on high frequencies, or details, of the image, and then adding it with some coefficient alpha to the original image. Thus it will create an image with details strenghthened and looking 'sharpened'. Here are some results with alpha = 0.5 and sigma = 10:

Hybrid Images

Now let us experiment with a fun image hybrid technique similar to the masking we just did. The idea is to first find a pair of images that we find interesting to hybrid, then align them using two-point matching with, for example, two eyes in Derek's and Nutmeg's images. Then we apply a low-pass filter, for which here we use a gaussian filter, to one image, and apply a high-pass filter, for which we subtract the gaussian-filtered image from the original, to another image. Finally we combine the two images by adding them together, which produces the result that will change in interpretation as you change your viewing distance to it. This works because when looked upon closely, the details or high frequency parts of the image will be more obvious to our perception while if looking from a farther distance, only the low frequency parts will get more attention. Here I used sigma = 7 and sometimes sigma = 3 for the filters. I also experimented with colors and the result comes out pretty good:

Failure Hybrid Example

Note that hybrid can fail or does not look pleasing if the images chosen do not have similar patterns so that it doesn't even make sense to hybrid, or the low-pass filtered one simply have too strong intensity so that the high-pass filtered one isn't quite noticeable.

Fourier Analysis

Here is the analysis for the process illustrated with the log magnitude of the Fourier transform of the two original images, the two filtered images, and the hybrid images, respectively:

Gaussian and Laplacian Stacks

A stack is similar to an image pyramid, but only that the image is not downsampled as we repeatedly apply filters to it. Here we construct both a gaussian and a laplacian stack for the source image, where the first one in each layer gets a gaussian blur from its previous layer and the second one in each layer the difference between its corresponding gaussian layer and the previous gaussian layer is recorded. The first level of the laplacian stack is simply the difference between the original image and the first-layer gaussian-filtered image. With the two stacks, we can observe the structure at different level of frequencies and see clearly what the filters do at each layer. Here I used sigma = 2 for each iteration of filtering:

Multiresolution Blending

An image spline is a method to combine/blend two images seamlessly by creating filtered stacks of each of the images and combine them appropriately. The process is also called multiresolution blending, as we create filtered images and masks in different levels of resolutions. Here we apply the algorithm designed by Burt and Adelson in their 1983 paper titled 'A Multiresolution Spline With Application to Image Mosaics'. In summary, we first build Laplacian stacks for both images, denoted as LA and LB respectively; then we create a binary-valued mask M to represent the desired blending region. Now we create a gaussian stack GM for the mask for later use. Then we create a combined stack LS from LA, LB, and GM using formula: (at each level) LS[i, j] = GM[i, j] * LA[i, j] + (1 - GM[i, j]) * LB[i, j]. Finally we combine the images in the combined stack by summing the levels appropriately with weights. Here are some results using spline blending:

Note that sometimes the object in the images can have a negative effect on the combined image since the two sources simply cannot align:

Laplacian Stacks

Here we display the Laplacian stacks for both the original images with masks:

Toy Problem

To get a feeling how gradient works, we give ourselves the task of reconstructing an image given a single pixel value of the source image and the gradient over the source image. The process is to set up a system of linear equations representing essentially three sets of least square problems: the horizontal gradients of each pair of horizontally adjacent pixels in the reconstructed image and each such pair in the source image, the vertical gradients of each pair of vertically adjacent pixels in the reconstructed image and each such pair in the source, and one corner pixel value of the reconstructed image with the value of the same pixel in the original image. Specifically, we set up a sparse matrix A with dimensions (x * (y - 1) + y * (x - 1) + 1, x * y), vector b with dimensions (x * y, 1) where x is the height of the image and y is the width of it. Finally we solve for argmin v of (Av - b) ^ 2, which with some reshaping, gives us the reconstructed image:

Poisson Blending

Now we use graident-domain processing to combine pictures. Different from Multiresolution blending, we will use gradient of the images. The idea is to solve a set of least square problems that aim to minimize a combined difference between both within the selected region of the image to blend in and over the edge between the blended-in and blended-into images while considering such difference with the difference of same pixels in the original image. Specifically, we try to minimize this function:

Here i represents each pixel, and in the first half of the equation j represents the neighbors of i while j is inside of the selected mask region S. The first half means that we hope to create the image where gradients inside S are similar to gradients of the cutout we want to paste in. The least square solver will then take hard edges of the cutout at the boundary and smooth them by spreading the error over the gradients inside S region. In the seccond half, j now represents the neighbor pixels of i while j is not inside of S anymore. And the second half just handles the boundary of S, and we pluck the intensity value right out of the target image.

Credits

Inspiration taken from friends, past projects, python libraries, papers and other online resources.