CS194-26: Image Manipulation and Computational Photography
Programming Project #3: Fun with Frequencies and Gradients!
Allyson Koo
Project Overview
This assignment involved manipulating frequencies and gradients of images to produce various image manipulations.
Implementation
Part 1: Unsharp Masking
This portion of the project involved "sharpening" a blurry image by adding the higher frequencies back into the image. The higher frequencies can be found by running a low-pass filter across the original image, i.e. a Gaussian filter, and then subtracting this from the original image, which should leave the high frequencies. By adding these high frequencies to the original image, we create a "sharpening" effect, as illustrated below.
Input image: blurry red panda
Gray scaled sharpened image
Colored sharpened image
Part 1.2: Hybrid Images
The purpose of this portion of the project was to create "hybrid images", where when you look closely at a picture you see a different image than when you view the same picture from afar. This effect is achieved by combining high frequency and low frequency images into the same picture. When looked at up close, only the high frequncies are visible, but viewed from afar, the low frequencies dominate, allowing you to create a hybrid image that changes when viewed at different distances.
Examples
Example merging of Derek and Nutmeg
Original picture: Derek
Original picture: Nutmeg
Example merging of Grumpy Cat and Boo the dog
Original picture: Grumpy Cat
Original picture: Boo the Dog
Example merging of Tom and Jerry
Original image: Tom
Original image: Jerry
Fourier analysis:
Fourier analysis of grumpy cat picture
Fourier analysis of boo the dog picture
Fourier analysis of the blended picture
Fourier analysis of Gaussian filtered image (Boo)
Fourier analysis of Laplacian portion of image (Grumpy Cat)
An example failure case is the image of trying to merge Tom and Jerry. In this case, the original images weren't aligned very well to begin with, so attempting to overlay them produced a jarring result and makes it easier to see the low-frequency image up close and the high-frequency image from afar due to the non-optimal alignment.
Part 1.3: Gaussian and Laplacian Stacks
The purpose of this portion of the project was to create "stacks" of images, where Gaussian filters are applied at each level. These filters are subtracted from the original image to create the corresponding Laplacian stack.
Gaussian and Laplacian stacks of the Mona Lisa
Gaussian and Laplacian stacks of the boo-grumpy cat hybrid from earlier
Part 1.4: Multiresolution Splining
Example merging of apple and orange
Original image: apple
Original image: orange
Example merging of angry and laughing reacts from Facebook
Original image: angry react
Original image: laugh react
Example merging of the Golden Gate bridge and the Eiffel tower
Original image: Golden Gate Bridge
Original image: Eiffel Tower
Gaussian/Laplacian stacks of the bridge/paris blended image
Example merging of the Taj Mahal and Trevi Fountain
Original image: Taj Mahal
Original image: Trevi Fountain
Irregular border
Example merging of fish from Finding Nemo and hiking picture
Original image: Fish from Finding Nemo
Original image: Hikers in the snow
Part 2.1: Toy Problem
The purpose of this portion of the project was to aid in implementing the gradient domain process for merging two images together. In this case, we simply copied a picture over by using its gradients to calculate what the proper pixel values should be.
Original image
Result generated
In this example, the output on the right was produced by using the method of gradient domain processing. To create this, we essentially had the goal of recreating the picture by solving for the pixel values in the target image that most closely matched the pixels in the original source image. We did so by using the gradients across different pictures. We create a matrix of linear equations representing the gradients of a pixel and its four neighbors, and use least squares to find the optimal pixel values satisfying these equations. We then copy the solved values over to the new image, and in this case reproduce the original image as shown.
Part 2.2: Poisson Blending
This portion of the project is an extension of the toy problem: in this case, we are allowing an object from a separate picture to be merged into a portion of the target image. The process is similar: we create a matrix of linear equations that allow us to solve for pixel values that minimize the differences in gradients between the solved values and the target or source pixels. However, this also includes an additional case of blending across the borders of the source and target image. This is handled by, for pixels on the border, treating only one pixel (the one inside the source image being pasted into the target image) is treated as variable while the neighboring pixel in the target image is treated as fixed. This allows the pixels inside the source region being copied over to be matched as closely as possible to the surrounding area in the target image. In order to select the portion of the image being copied over, masks are created (images of the same size as the source and target but composed of 0's and 1's, where 0 represents parts of the image that are ignored and the 1's represent the area where the source image will be copied over). These masks can be used to generate the final images. First, an im2var matrix is created to maintain a mapping between pixel coordinates and the location of the corresponding variable for the pixel's solved value. This is necessary because there will be multiple linear equations for each pixel and all must be solved simultaneously in order to produce a correct result, so a mapping between pixel coordinates and variables makes it easier to ensure that each linear equation references the proper variable. Then, once the equations are set up and solved, the solved pixel values are copied into the proper location in the target image using the aid of the mask of the target image. This mask tells where the source image should be pasted (the nonzero area is the location). Any pixel not in this area is simply copied over from the original target image. This method produces the following results.
Example using provided sample images from class: merging a penguin chick into a picture of hikers hiking in the snow.
Original target image: hikers in the snow
Original source image: penguin chick
Mask result on im2
Mask result on penguin chick
Grayscale image merging
Colorized image merging
Examples using my own images (courtesy of the internet).
Example merging of a picture of the Golden Gate Bridge and Nemo, Marlin, and Dory from Finding Nemo.
Original target image: Golden Gate Bridge
Original source image: Nemo and Dory
Grayscale image merging
Colorized image merging
An example merging of the House (with balloons) from Up with a balloon fish.
Original target image: House (with balloons) from the movie Up
Original source image: Balloon fish
Gray image with source pixels directly copied
Colored image with source pixels directly copied
Grayscale image merging
Colorized image merging
An example merging of hikers in the snow with Olaf from Frozen.
Original target image: House (with balloons) from the movie Up
Original source image: Olaf from Frozen
Grayscale image merging
Colorized image merging
Some issues that arose with this method of blending was if the source and target images had different background colors. As shown in the example of blending the golden gate bridge with the fish from Finding Nemo, the different background colors caused a change in the other colors of the target image when the pixel values of the picture were renormalized and displayed in color. The issue is less noticeable in grayscale images.
Comparison between Laplacian/Gaussian stacks blending and Poisson Blending:
Original angry/laugh react blend:
Example merging of angry and laughing reacts from Facebook
Result of using Poisson blending:
Example merging of angry and laughing reacts from Facebook
Example merging of angry and laughing reacts from Facebook
As shown in this case, for a sharp vertical edge, the Gaussian/Laplacian stacks method of blending was much more successful in producing a smooth transition between the images. The Poisson method blends the colors a bit better at the edges, but overall has a much less strong smoothing effect, causing a jarring shift in this example. I think in this case where it involves needing to blend out a sharp extended line or when the source and target images won't necessarily fit together if you were to just "copy/paste" one image into another, it's best to use the Gaussian/Laplacian stacks method to smooth the transition. Otherwise the Poisson method provides a more accurate matching, especially with colors.