CS 194-26 Project 3 - Fun with Frequencies and Gradients

Austin Luong

Part 1: Frequency Domain

Warmup: Unsharp Masking

The warmup involving a blurry photo and sharpening using the unsharp masking technique. This involves first creating a "detail" image taking the difference between an image and a slightly Gaussian blurred version of the image. Then, the detail image time some constant alpha is added back to original image to enhance the detail and effectively sharpen the image. Below are results for 4 different alpha values. As you increase alpha, the sharpenning effect increases but also creates an odd stencil effect when it is too high.

Hybrid Images

Most people are likely familiar with Einstein / Marilyn Monroe illusion that look like one or the other based on how far you are from the image. It is possible to create similar illusions to that using the technique described in a SIGGRAPH 2006 paper. The basic idea is to mix a high frequency version of one image with a low frequency version of another image. The high frequency image is the "detail" image created as described previously. The low frequency image is a Gaussian blurred image. Additionally, a litle bit of the high frequency image is added to the hybrid image so that the low frequency image does not completely dominate in texture and color. Now when you look at these hybrid images from up close, you should be able to only clearly see the high frequency image. Similary, when looking from afar, only the low frequency can be seen. Note: it may be helpful to open the image in new tab and zoom in and out.

The above results are pretty good when the original images already have a decent amount of similarity. However, when the images are bit too different in color, alignment, shape, or size the results are not as convicining as shown in the elephant-cow hybrid. From close up, the color of the cow is too noticeable and distracting from the high frequency elephant. In black and white, these issues are moderately alleviated.

Process

The process for can be visualized through looking at the frequency domain of the images being merged. In the frequency images (lower row), a white dot represents a present frequency and angle. Its distance from the origin is its frequency (farther is higher) and its radial position is its angle. The process is shown below for the shiba-cat hybrid.

Gaussian and Laplacian Stacks

A Gaussian stack is simply a stack of images with increasing levels (doubling in my case) of Gaussian blurring applied to some source image at each level. A Laplacian stack is stack created by taking the differences between the levels in the Gaussian stack. That is, the i-th layer of the Laplacian stack is the difference between the i-th and (i + 1)-th layer of the Gaussian stack. This creates bandpass images. Additionally, the last layer of the Laplacian stack is the last Gaussian layer used to create the difference image. This is so we can recover the original image by summing the Laplacian stack. Below are the results of applying the Gaussian and Laplacian stacks to two images.

Multiresolution Blending

Using the techniques described in a 1983 paper bt Burt and Adelson, it is possible to seamlessly blend two images together using a technique called multi resolution blending. The technique first creates Laplacian stacks of two images you want to blend and a Gaussian stack of an image mask specifying where to blend the images. The stacks are all combined into a new stack such that each layer i of the new stack is:

Blend[i] = Laplacian_A[i] * Gaussian_Mask[i] + (1 - Gaussian_Mask[i]) ** Laplacian_B[i]

The blended stack is then collapsed or summed to create the final blended image. Some examples are shown below (only some inputs and masks are shown).

Penguin goes on vacation

A mutant dog owl yawning

Some familiar faces

Process

The process for can be visualized by looking at the contributions of the Laplacian stack of the first image and the Laplacian stack of the second image. This visualization is shown below for the penguin-pyramid example. It may help to turn up your monitor brightness or open the images in a new tab.

Part 2: Gradient Domain

Overview

Now we turn our attention to the gradient domain instead of the frequency domain. It turns out that in most image blending tasks, what we really care about are that the gradients between the images match well rather than the actual pixels. In this part, we explore a way to seamlessly blend a source image into a target image by having the output iamge satisfy a set of constraints involving the gradients of the source and output images.

Toy Problem

First we consider a toy problem using an image. We use all the gradients of the image plus one pixel intensity to reconstruct an image. The constraints are that the image and output image have the same gradients and that they have the same top left pixel. Logically, this should result in the same image. This optimization can be solved as a least squares problem. Below, input is shown on the left and output and shown on the right.

Poisson Blending

Using the idea of gradients, we can specify a least squares problem to seamlessly blend a source image into a target image. The specification that follows results in a technique known as Poisson Blending. The gist of the idea is that inside an input mask, the gradients of the output (blended) image should match the gradients of the source image. That is, the object we paste into a new image should have gradients in the new image that match the object in its original image. Additionally, the pixels not in the mask should all be set to the that of the target image (the image we are pasting into to). We take the gradient as a 3x3 Laplacian kernel at each point. Doing so results in a set of equations that we can use to fill in a sparse matrix and solve for our output pixels. The results are shown below.

Jaws

The input images and the naive blending is show in comparison to the Poisson blending (lower right image). This result turned out quite well and seamless but the shark is colored a bit too dark because I placed it too low halfway into a dark part of an image which messed with the gradients a bit.

Nixle Alert: Lion Spotted on Memorial Galde

This result did not turn out as well as hoped. The lion turned much too green because of the strong green color of Memorial Glade and there is also blurring around the edges of the lion caused by a different and lower resolution texture of grass in the source lion image. The higher resolution of the Memorial Glade also makes the lion look blurrier and out of place.

A Dragon and the Campanile

Another fairly good result. The dragon's color is blended very realistically with the background image but there is some artifacts due to the source image having clouds while the target did not.

Multiresoltuion vs Poisson Blending: Trump + Hillary Revisited

The results Poisson blending exceeded that of Laplacian pyramid blending from part 1 in this case. The face colors blend much more seamlessly and the lighting of the resulting image is more realistic looking. For blends where gradients, lighting, and colors matter a lot more, Poisson blending is the better choice. These cases are most prevalent when you want to add one object into another and have it look realistic. However, Laplacian pyramid blending may work better for cases where you want to perserve the color of the original image more and just want the blend to be seamless. For example, running Poisson on the penguin-pyramid image from part 1 results in a brown colored penguin towards the bottom and makes the snow brown which was not desired (this result is not show because of time).