Hybrid Images (CS 194-26 Fall 2018 - Project 3)

Overview

This project was split into two parts, both aimed at producing hybrid images. The goal of the first part of this project was to create hybrid images through manipulating frequency properties. Gaussian Filtering was implemented to construct Gaussian and Laplacian stacks, which allowed for creating images that change with viewing distance, as well as some nice blended images. The second part of the project focused on gradient domain fusion, using a least squares solver to both reconstruct images, as well as poisson blend 2 images together using masks.

Part 1.1: Warmup

The images below show the use of a simple Gaussian filter to pseudo-sharpen images by adding back to the original image the difference between the original image and gaussian blurred image. The alpha value controls the scaling of pixels added back to original image. The higher the alpha the "sharper" the image.

In the example of the penguin image above, as alpha is increased we see a sharpening of edges, feathers, and the feet in particular.

Part 1.2: Hybrid Images

High frequencies dominate perception up close, whereas at a distance only low frequencies can be seen. This part of the project uses this principle to blend the high frequency portion of one image with the low frequency portion of another, resulting in a hybrid image that looks more like the first image when viewed up close, but more like the second as you move further away.

The examples shown below have been created by applying a low pass filter to one image and a high pass image to the other. The two filtered images are then simply averaged to achieve the results shown below.

I made my images in colour to be more visually appealing (Bells & Whistles). Looking at the colour images presented, it is apparent that colour works much better for the low-frequency components. They show up more as the colour in images are generally stored in the lower frequencies, with sharp edges constituting the high frequencies lacking colourful information.

The hybrid image between my friend Nick (also in this class) and I failed pretty miserably as the low frequencies dominated the high frequencies too much - even up close my face dominates. Moreover the shapes of our faces did not quite match up, aligning based on face shape will lead to mouth locations being off, but facial feature alignment will lead to chin and hairlines being off.

Moreover, for Fourier analysis for 2 of my favourite hybrid images are displayed below in grayscale. Images pre and post low and high pass filtering are shown, as well as the summation of the 2 filtered FT images.

Part 1.3: Gaussian & Laplacian Stacks

In this part, I implemented Gaussian and Laplacian stacks to images. The Gaussian stacks are successive layers of Gaussian filtering applied to images, while the Laplacian stacks are comprised of the difference between a gaussian level and its prior level. The results for both are shown below on several images. Note that the Laplacian stacks below have been sharpened by pixel intensity rescaling for better visibility on lower resolution / small sized images on screens.

Part 1.4: Multiresolution Blending

In the last section of the first half of the project I used the laplacian stacks developed in the previous part to seamlessly blend 2 images together along a masked boundary. The first set of 3 images are blended along a simple vertical seam, while the next 3 utilize irregular masks to accomplish a wider variety of cool results.

Part 2.1: Gradient Domain Fusion - Toy Problem

Now moving onto the second half of this project, away from manipulating image frequencies, and instead redefining the way we think about images as gradients, rather than absolute intensities. The goal of this part is to reconstruct the Toy Story image of Woody and Buzz using gradients. To do this we take our original source image, and using vectorized numpy, construct an A matrix and b vector that can be fed into a least squares minimizer, to calculate output pixel values that will minimize the error between the final image and both the x and y gradients of the source image. Note that there are numerous possible solutions to the least square problem, thus a sort of "initial condition" is needed to fix the output pixels to be the same as the input images. To do this we "pin" the top left corner of the image by setting an additional constraint that the top left pixels in both images must be of the same intensity. The result is shown below.

Part 2.2: Poission Blending

Finally, in this last and most challenging section, we attempt to use gradients and least squares solving methodology to achieve a similar goal to that of part 1.4 - seamlessly blend images. We take a source image and attempt to blend it into a target image, by using least squares to calculate new pixel values for a masked region of the source image that will both maximally preserve original source image gradient, as well as minimizing its difference with the destination target image. This must be done without altering the target image. Again the bulk of this part of the project was in speeding up numpy code to construct a large A matrix and b vector to feed into a least squares solver, thus sparse matrices and vectorization were used. Thanks is given to Nikhil for sharing mask making code in python. The resulting images are shown below.

Although the last image inserting the balloon into the sky above the beach has a smooth blend into the background, it is a bit of a failure as the colour of the balloon has been completely lost. The beauty of the colourful contrast with the blue sky I was hoping to achieve has been lost - replaced by homogenous blue. The reason this failed was posisbly due to too large of a contrast between the background sky of the balloon (a ligher blue) with the background sky of the beach (a darker blue). To match the two backgrounds but maintain inner gradient, extra "blue" was added to the entire balloon image, thus saturating the image blue. This hypothesis is supported by the even clearer example of this shown below. I tried putting the penguin with a white background into the solar system picture which has a much darker black background - thus the entire penguin becomes black.

Finally, we can compare the differences between using Laplacian Stack blending (part 1.4) and the Poisson Blending done in this section by looking at the images we ran on both. For convenience find them side by side below. By comparing different examples blended using both methods we can see that each has its own merits and downsides - which is better depends entirely on application purpose. Multiresolution blending does a much better job at preserving colour in original source images (i.e. the ballon, or the galaxy), whereas Poisson blending does a better job at smoothly joining different backgrounds. The baloon at the beach image best exemplifies this difference, with perfect colour preservation but pixel artifacts visible in the multiresolution blending image, in comparsion to perfect blending but colour loss in the poisson blending result.