Project 3: Fun with Frequencies and Gradients

In this project, we explore the various ways of combining images through frequency and gradient domain decomposition. We gain a deeper understanding into the nature of image decomposition and use clever relationships between the pixels in our image to perform manipulations to capture various qualities of images when blending.

Part 1.1: Warmup

In this step, we sharpen an image by convolving a gaussian filter with the image, and subtracting that blurred image from the original to get the sharp edges of the image. We can take those edges and add it to our original image to accentuate the edges and make the image seem "sharper". Here's an example:

Part 1.2: Hybrid Images

In this part, we generate hybrid images by taking the higher frequencies of an image and combining our result with the lower frequencies of another image. This gives the illusion of seeing one image while close up, and seeing a different image from farther away. We achieve this by using the same gaussian filter to blur (and get the low frequencies), and subtract it from the original image as we did in the previous step (and get the high frequencies). Here's what it looks like on our sample images:

Here are the results of some other images:

If you look closely at Mr T, he seems pretty serious, but if you look from afar, it seem like he's grinning! Here's a failed version that didn't go so well:

This failed image was most due to overtuning of the frequencies; I kept too many of the edge frequencies and too little of the blurred frequencies. Let's take a look at the frequency decomposition of the images:

Part 1.3: Gaussian and Laplacian Stacks

In this section, we create gaussian stacks by repeatedly applying our gaussian filter to an image. We create a laplacian stack by taking the gaussian stack at level i, and subtracting it from the gaussian stack at level i - 1 (where level 0 is our original image). We can do some neat things with these stacks (as I'll show in the next part). Here are the gaussian stacks from some of the images in the previous part.

And the corresponding laplacian stacks:

Part 1.4: Multiresolution Blending

Using these gaussian and laplacian stacks, we can achieve a smooth blending effect between images. We start with a mask (1's where one image's pixels should be, 0's where the other image's pixels should be), and produce a gaussian stack of our mask. We multiply each level of our mask's gaussian with the gaussian of our two images (by a factor gaussian(mask)[i] for one image and 1-gaussian(mask)[i] for the other image) and in this way, match the "blur" of our mask with the "blur" of the contents we are trying to blend. Here are some examples:

Part 2.1: Toy Example

In this section, we think about images in terms of their gradient domain. In blending images, we try to preserve the difference in values between pixels while trying to "match" the pixels where one image meets the other. This technique works well because humans are good at perceiving the differences between pixels, rather than the absolute pixel values themselves. In doing this, we try to optimize the following:

argmin_v(sum(((v_i - v_j) - (s_i - s_j))**2) + sum(((v_i - t_j) - (s_i - s_j))**2))

where i are all pixels in our mask, j is each adjacent neighbor pixel to x, s is our source image, t is our target image. To solve this complicated series of constraints, we can set up a least squares problem and "solve" for v to get a good optimization for our image.

As a quick proof of concept, we run this optimization on a single small "toy" image, and ensure that we are able to get back the same image after our algorithm runs. Since we aren't mixing the image with anything, we should expect to get back the same image.

Part 2.2: Poisson Blending

Now, we can apply this algorithm to blend images, here are the two images we'll be blending:

Here's how it looks if we simply copied the images from one to the other:

And here's the result of our poisson blending:

Other nifty poisson blends:

Here's an image that I tried to blend originally with multiresolution blending. It didn't look good before, and I got startlingly better results with poisson blending. It seems that in this situation, poisson blending worked a lot better, but it didn't work well at all in other scenarios (that I forgot to save). I would commonly run into situations where the source image would take on the color characteristics of the target image, and the entire algorithm seems to only work well in situations where the boundary between the source and target are of somewhat similar colors, where multi-resolution blending doesn't run into that problem. However, in examples such as the one below, poisson blending is able to achieve much better results when operating within those conditions.

Bells and Whistles

Each of the parts have been implemented in color.