Eilam Levitov - cs194-26-acx
This notebook runs on python 2.7
Here I will be demonstrating the sharpening technique developed in class.
# read in the image, convert to applicable representation, and display
# Use gaussian filter to extract high frequencies
# Original vs Sharp bunny
# Original vs Sharp road (Joshua Tree)
After applying the gaussian filters, some pixels become negative (red) or saturated (blue), thus a slight modification was needed to improve the final image. In order to fix that, I tried: (1) normalizing the image, (2) shifting and scaling, (3) both, and finally (4) clipping
In the end, (4) was found to be most effective, thus is the one demonstrated above.
Looking at bunny
the sharpness effect is most noticeable on the fur and the little carrot.
The road
image on the other hand, the effect is much more noticeable all around. We deduct from it that an image that orignally contained a lot of high freqeuncies will be more suitable for this sort of sharpening procedure.
We use Hybrid Images as described in the paper by Oliva et al. Hybrid Images is a tenchnique that produces static images with two interpretations, which change as a function of viewing distance. This technique enables us to layer the high frequencies from one image on top of low frequencies of another. Finally, the results shows that when seen up close, the viewer wil observe (mainly) the high frequency image, while from afar the low frequency image
# Read, convert, and display images
# Set cutoff frequencies, separate frequencies of each image, display
# Average both low and high frequency images, and display hybrid image
# Take the Fourier Transform and display detailed plot of frequencies in each image
# Mirrored faces
# Frequency Analysis
Using color to (try) enchance hybrid image effect
# Mirrored faces, with color!
# Low Frequncies in each channel
# High Frequncies in each channel
# Hybrid Frequncies in each channel
Combining several images using a high-pass, band-pass, and low-pass filters
# Read, convert, and display images
# First filtering layer
# Display result of 1st layer
# Second filtering layer
# Display result
# 3 image blend frequency plot
The Hybrid Image technique of layering high and low frequencies provides a way to create textures that become visible as a function of viewing distance. As seen above, it is necessary to find compatible
signals (or, equivalently, preprocess the signals thoroughly) in order to get good results. In our case, the eagle and F16 combined rather well, while the image of the couple seems quite visually odd (although funny). Finally, although the quality of the image could have been better, the simplicity of the technique makes it an attractive tool when considering such effects.
Fourier transform allows us to see the abundancy of frequencies in each signal. As seen above, the low frequency signal is much more abundant in low frqeuncies, and high frequency follows accordingly. The entire bandwich image shows a balanced
distribution of frequencies, which is what we would expect.
*A note on the presented frequency analysis: Gaussian is far from an ideal low-pass, thus as can be seen above, there are some residual low frequenices in the high frequency plots, as well as high frequencies in the low frequency plots.
In this part we will make use of Laplacian and Gaussian Stacks. Gaussian Stacks are generated by continuously filtering each subsequent level of an image, consequently stacking lower and lower frequencies. Using the Gaussian Stack we can generate a Laplacian Stack, which stores high frequencies, starting from the highest frequencies. To make the Laplacian stack, at each level i
we subtract the corresponding (i-1)
th from the i
th level of the Gaussian Stack, giving us the residual high frequencies in-between the Gaussian Stack levels.
# Decomposition and Reconstruction of Laplacian Pyramid (not sub/upsampling in our stack case)
stack
.¶# image found on pinterest, made by Teal Scott
# 4 layered Gaussian stack
# 4 layered Laplacian stack
# Read, convert, and dislay image
# 5 layered Gaussian and Laplacian Stack
# Read, convert, and display images
# 5 layeres Laplacian and Gaussian Stack
# Display Gaussian Stacks
# Display Laplacian Stacks
# Display result
In this section we showed the beauty and power of the Gaussian and Laplacian Stacks. As can be seen above, this technique allows the observer to visualize
the frequency analysis of the image, as well as generate a hybrid image using different levels of the laplacian stack!
We use the Multiresolution Spline technique described in the paper by Burt et al. for combining two or more images into a larger image mosaic. This method is applied in a layered manner, where we take advantage of the Gaussian and Laplacian stacks in order to establish a visibly appealing blended image.
i
th layer of the stacks, combine the two images in the Laplacian stacks by adding the two images with the corresponding mask from the Gaussian stack.# Read, convert, and display images
# Display Gaussian Stack
# Display Laplacian Stack
# Display Masks Gaussian Stacks
# Display result
# On the left - Golden Gate Bridge; Right - Brooklyn Bridge
# Display Gaussian Stacks
# Display Laplacian Stacks
# Display Mask
# Display result
# JLo and Trump
The method of Multiresolution blending technique provides an elegent way to blend images. The main strength of the technique is it's transition, which by applying it in a layered manner prove to be superior to prior techniques. But, quite like former techniques, this method also has its shortcoming, where white balancing can become an issue. For example, we can see in the Bridge example above, even manual balancing provides an imperfect transition. On the other hand, it showed a perfect transition in the case of headshots - see jt.
In this part of the project we takes advantage of the fact the human eye is tuned to the gradients and process the images in the gradient domain. As we know our visual perception is highly sensitive to changes, rather than real values, and so we a task like blending would theoretically be very suitable to such novel techniques.
In this part we effectively check if theory aligns with application, or more correctly, make sure our math is correct.
Our problem is as such: If we have 1 known pixel and the changes (gradients) of the image, are we able to perfectly reconstruct the image?
We have formulated the problem as a linear algebra problem, specifically as Ax=b
, using 3 questions:
where s denotes the source vector, and v is our result vector.
We will try this on an image from Toy Story!
# Read, convert, and display image
# find, display, and ravel the b vector
# Find A
spsolver
we can efficiently solve this least squares problem!¶# Solve Ax=b and shift + rescale image back to [0,1] interval
# Reshape and show result
# Display images next to each other for comparison
Reconstructed image looks good with very minimial error, thus our math is correct -> we can continue!
In the last part of the project we use the Poisson Blending technique described in the paper by Perez et al. for seamless editing of image regions. This technique follows closely to what we have done just prior in section 3.1.1, where me use the gradients while keeping some known
values to reconstruct. The difference in this part is we injects
the source's gradient patch into the target, leaving the border pixels as is, and solving it using least squares method. This will allow us to transition from one image to another without the need to white balance.
injection
siteInject
the patch into the corresponding location within the target We'll being with some Grayscale testings
# Read, convert, and display image
# Read, convert, and display image
# Manual Alignment
# Naive blending (for completeness)
# Find b
# Find A
# Solve Ax=b
# Reshape, show result and error from original
# Display Gradient Fusion
# Read, convert, and display (aligned) images
# Find b
b = b.ravel(order='F')
# Find A
# Gradient Fusion
San Francisco is now under the rule of Sauron!
For my surprise, in this example it seems as if the multiresolution blending takes the cake! Although the smooth transition of the graient fusion is superior, the white balance is actually too strong even after manually decreasing the average. Consequently, we have shown that in fact each method has it's advantages, and disadvantages, and each works better for different images.
Again demonstrating on the bridge image, if I hadn't manually adjusted the average value of pixels in the golden gate bridge image, the result of the gradient fusion would have been abysmal! Thus, once again, we see the power of this technique can actually overly effect the process and result in poor blended image.
In this project we played with frequencies and gradients. The most valued piece of information that I have learned in the process is the true power of the gradient domain (and efficiently solving sparse matricies). Besides that, I would like to mention that although the scientific community keeps coming up with more advanced, noval techniques (ehm ehm machine learning people), sometimes one must remember that a simple method might be more suitable for a specific task!