An image of Jupiter, unsharpened.
Jupiter, sharpened. Notice how the details pop!
The purpose of this project is to experiment with the different methods of blending images together, and to understand what's going on behind the scenes, both in the frequency domain and the time domain.
We start with sharpening an image using Gaussian and Laplacian filters. The idea is that using a Laplacian filter, which is approximated by an impulse subtracted by a Gaussian, one can ampilfy the high frequency details of an image. We can add a weighted version of the Laplacian filtered image to the original image, which will amplify edges. Here it is, in effect, on Jupiter.
In this part, the aim is to blend two images together so that from close distance, one image is seen, but at a distance, another is seen. We do this by taking the low frequencies of one image, and superimposing them on the high frequencies of the other image. High frequency details can be seen from close up, but at a distance, only the low frequencies of the other image ar visible. Here's the effect on Nutmeg (Laplacian Filter with sigma of 8), and Derek (Gaussian Filter with sigma of 5).
|
|
Here's an example with Bill Gates and Steve Jobs. Gates is filtered using a laplacian with sigma 5, while Jobs is filtered using a Gaussian with sigma 2.
|
|
|
|
A tribute to the greatest utensil. The fork's Laplacian has a sigma of 12, while the spoon's Gaussian has a sigma of 12.
|
|
I tried blending a picture of flowers and fireworks, but it didn't go well. Unfortunately, the fireworks overpower the flowers and are still seen no matter what, and the flowers in the background slightly occlude the fireworks, so it's hard to tell what's going on unless you really minimize the image.
|
|
I liked the Jobs+Gates result, so I decided to perform Fourier Analysis on each of the steps for creating that image.
|
|
|
|
In this part, we construct Gaussian and Laplacian stacks, which are ways of visualizing the information contained in different bands of frequencies in an image. In a Gaussian Stack, an image is passed through a series of low pass filters, showing more and more low frequency components, while in a Laplacian Stack, the image is passed through a number of Laplacian filters, showing subsequently higher frequency bands.
Let's take a look at our friend Jupiter again.
Gaussian Stack
|
|
|
|
|
Laplacian Stack
Notice how the details become finer!
|
|
|
|
|
Let's revisit the composite of Jobs and Gates through Gaussian and Laplacian Stacks!
Gaussian Stack
|
|
|
|
|
Laplacian Stack
|
|
|
|
|
In this part, we look at using masks to blend images together. The naive way to do this would be to take a feathered mask, multiply image 1 by it, invert the mask, multiply the inverted mask by image 2, and simply add them together. However, this can create noticeable seams and abrupt transitions if not done properly. So, the idea here is to take a Gaussian stack of the mask, and apply it to Laplacian stack of the images, and then add those together to obtain a much more seamless result.
I found that the sigma for the Gaussian stack of the mask had to be much higher than the sigmas of the images, and it was often best to use a low sigma for the laplacian stacks of the images to maintain clarity.
For example, here's the orapple:
Example 1: Galaxy Eye
|
|
|
Here's the laplacian stacks accompanying this, with the filtered masks applied at each step!
|
|
|
|
|
|
|
|
|
|
Example 2: Seascape
|
|
Instead of using frequency domain blending, we can also frame this as a spatial-domain optimization problem! The idea is that we humans only really care about the sharp edges: the gradients. So, when we're blending images, we should try to preserve the gradients of the section that we're adding in, but also the gradients around the edges of the region to match what was originally there in the background image. This can be framed as a least squares problems, and we can solve for the pixel intensities in this region using a sparse least squares solver. The downside of this approach is that I can't use it for large regions like multiresolution image, since the size of the matrix grows quadratically with the number of pixels. My computer literally runs out of RAM!
I first started with a small toy example, to understand how to construct the least squares matrices.
|
|
Pretty cool!
Poisson Blending Example 1: Run Forrest Run!
|
|
|
|
Example 2: Making Cruises More Exciting
|
|
|
|
Failure Example: Bringing Saturn Closer
The failure here is that the extreme gradient difference between the Berkeley sky and Saturn cause Saturn to be brightened considerably, causing an unnatural illumination of the planet, and washing out details, causing it to appear copy/pasted in, while a more natural blending result would have a hazy or darkened Saturn, fitting in to the twilight sky and being considerably darker.
|
|
|
I also implemented mixed gradients, which is great when you need to preserve the background gradient, such as for generating awesome motivational text posters. The idea is that instead of trying to optimize for the original source gradient, you instead optimize for whichever gradient has greater magnitude.
|
|
Obviously, the question arises, which blending method performs best? Here's a comparison with the galaxy+eye image from earlier, with both Poisson and Mixed Gradients.
|
|
|
|
Obviously, we see some pretty different results.
First off the bat, the mixed gradients seems to not do much of anything, but this makes sense!
The gradients of the eye, with its many details, are much stronger than the weaker-gradient
galaxy, besides some small stars which appear in the eye. This just means that there is only
a very specific scenario in which mixed gradients are useful.
The difference between the Laplacian and Poisson is that the Poisson preserves more of the eye,
creating a pretty cool result. It also doesn't require any blurring, so the image is sharper.
On the other hand, the multiresolution laplacian preserves more
of the black space region around the galaxy, and the eye shape, creating an effect within the eye.
Ultimately, in this case, it comes down to preference. It is worth noting that the multiresolution
blending took much, much less time than the Poisson blending!