CS194-26 Project 3 - Fun with Frequencies and Gradients!

Parsa Fereydouni - cs194-26-agy

Part 1 - Frequency Domain

1.1 Warm up

Here I will be demonstrating the sharpening technique developed in class.

High level procedure explanation

  1. Read in an image
  2. get a smoothed image using a gaussian filter
  3. extract detail by taking the difference of the original and the smoothed image
  4. add detail back to the image
In [4]:
# load blurry image that needs to be touched up
import matplotlib.pyplot as plt
import numpy as np
from scipy.signal import convolve2d
import skimage.io as skio
import skimage as sk
from numpy import fft

imname = './images/blur.jpg'
blur = skio.imread(imname) 
blur = sk.img_as_float(blur)
plt.figure(figsize=(8,8))
skio.imshow(blur)
skio.show()
In [3]:
# smoothed and detail
imname = './images/smooth.jpg'
smooth = skio.imread(imname) 
smooth = sk.img_as_float(smooth)


imname = './images/detail.jpg'
detail = skio.imread(imname) 
detail = sk.img_as_float(detail)


plt.figure(figsize=(8,8))
plt.subplot(1,2,1)
skio.imshow(smooth)
plt.title("smoothed")
plt.subplot(1,2,2)
skio.imshow(detail)
plt.title("detail")
skio.show()

Contrast and Compare

In [5]:
imname = './images/sharpened.jpg'
sharp = skio.imread(imname) 
sharp = sk.img_as_float(sharp)


plt.figure(figsize=(8,8))
plt.subplot(1,2,1)
skio.imshow(blur)
plt.title("original")
plt.subplot(1,2,2)
skio.imshow(sharp)
plt.title("sharpened")
skio.show()

The image has improved, and we can see more detail now. If you look at the car in the top left corner, for example, not much of the headlights and the grill of the origianl are visible but in the sharpened you can clearly see them.

1.2 Hybrid Images

We use Hybrid Images as described in the paper by Oliva et al. Hybrid Images is a tenchnique that produces static images with two interpretations, which change as a function of viewing distance. This technique enables us to layer the high frequencies from one image on top of low frequencies of another. Finally, the results shows that when seen up close, the viewer wil observe (mainly) the high frequency image, while from afar the low frequency image

In [19]:
# first hybrid's constituent images
imname = './images/nutmeg.jpg'
close = skio.imread(imname) 
imname = './images/DerekPicture.jpg'
far = skio.imread(imname)

close = sk.color.rgb2gray(sk.img_as_float(close))
far = sk.color.rgb2gray(sk.img_as_float(far))
plt.figure(figsize=(8,8))
plt.subplot(1,2,1)
skio.imshow(close)
plt.title("close")
plt.subplot(1,2,2)
skio.imshow(far)
plt.title("far")
skio.show()
In [20]:
# hybrid 1
imname = './images/hybrid1.jpg'
hybrid = skio.imread(imname) 
hybrid = sk.img_as_float(hybrid)
plt.figure(figsize=(8,8))
skio.imshow(hybrid)
skio.show()

Another Example

In [21]:
# second hybrid's constituent images
imname = './images/lion.jpg'
close = skio.imread(imname) 
imname = './images/kitten.jpg'
far = skio.imread(imname)

close = sk.color.rgb2gray(sk.img_as_float(close))
far = sk.color.rgb2gray(sk.img_as_float(far))
plt.figure(figsize=(8,8))
plt.subplot(1,2,1)
skio.imshow(close)
plt.title("close")
plt.subplot(1,2,2)
skio.imshow(far)
plt.title("far")
skio.show()
In [22]:
# hybrid 2
imname = './images/hybrid2.jpg'
hybrid = skio.imread(imname) 
hybrid = sk.img_as_float(hybrid)
plt.figure(figsize=(8,8))
skio.imshow(hybrid)
skio.show()

Another Example

In [23]:
# third hybrid's constituent images
imname = './images/run.png'
close = skio.imread(imname) 
imname = './images/stand.png'
far = skio.imread(imname)

close = sk.color.rgb2gray(sk.img_as_float(close))
far = sk.color.rgb2gray(sk.img_as_float(far))
plt.figure(figsize=(8,8))
plt.subplot(1,2,1)
skio.imshow(close)
plt.title("close")
plt.subplot(1,2,2)
skio.imshow(far)
plt.title("far")
skio.show()
In [24]:
# hybrid 3
imname = './images/hybrid3.jpg'
hybrid = skio.imread(imname) 
hybrid = sk.img_as_float(hybrid)
plt.figure(figsize=(8,8))
skio.imshow(hybrid)
skio.show()

Frequency Analysis Of the Third Hybrid (The Running Stickman)

In [25]:
# frequency of the orignal images
imname = './images/hybrid3_1_fft.png'
close = skio.imread(imname) 
imname = './images/hybrid3_2_fft.png'
far = skio.imread(imname)
close = sk.color.rgb2gray(sk.img_as_float(close))
far = sk.color.rgb2gray(sk.img_as_float(far))
plt.figure(figsize=(8,8))
plt.subplot(1,2,1)
skio.imshow(close)
plt.title("close")
plt.subplot(1,2,2)
skio.imshow(far)
plt.title("far")
skio.show()
In [26]:
# frequency of the filtered images
imname = './images/hybrid3_high_fft.png'
close = skio.imread(imname) 
imname = './images/hybrid3_low_fft.png'
far = skio.imread(imname)
close = sk.color.rgb2gray(sk.img_as_float(close))
far = sk.color.rgb2gray(sk.img_as_float(far))
plt.figure(figsize=(8,8))
plt.subplot(1,2,1)
skio.imshow(close)
plt.title("close")
plt.subplot(1,2,2)
skio.imshow(far)
plt.title("far")
skio.show()
In [28]:
# frequency of the hybrid
imname = './images/hybrid3_fft.png'
hybrid = skio.imread(imname) 
hybrid = sk.img_as_float(hybrid)
plt.figure(figsize=(8,8))
skio.imshow(hybrid)
skio.show()

1.3 Gaussian and Laplacian Stacks

In this part we will make use of Laplacian and Gaussian Stacks. Gaussian Stacks are generated by continuously filtering each subsequent level of an image, consequently stacking lower and lower frequencies. Using the Gaussian Stack we can generate a Laplacian Stack, which stores high frequencies, starting from the highest frequencies. To make the Laplacian stack, at each level i we subtract the corresponding (i-1)th from the ith level of the Gaussian Stack, giving us the residual high frequencies in-between the Gaussian Stack levels.

In [32]:
# An interesitng multi frequency image by Teal Scot
imname = './images/multifreq.jpg'
pint = skio.imread(imname) 
pint = sk.img_as_float(pint)

plt.figure(figsize=(10,10))
plt.imshow(pint)
plt.show()

Gaussian and Laplacian Stacks

In [39]:
# A 5 level stack
imname = './images/multifreq_stacks.png'
pint = skio.imread(imname) 
pint = sk.img_as_float(pint)

plt.figure(figsize=(30,30))
plt.imshow(pint)
plt.show()

The first row of this image shows the gaussian stack of the image.

The second row shows the gaussian stack of the image.

By subtracting image (i + 1) from image (i) in the gaussian row (first row), you get image (i) in the laplasian stack (second row).

Stacks of hybrids from 1.2

In [41]:
# A 5 level stack of hybrid 1
imname = './images/hybrid1_stacks.jpg'
hybrid = skio.imread(imname) 
hybrid = sk.img_as_float(hybrid)

plt.figure(figsize=(30,30))
plt.imshow(hybrid)
plt.show()
In [42]:
# A 5 level stack of hybrid 2
imname = './images/hybrid2_stacks.jpg'
hybrid = skio.imread(imname) 
hybrid = sk.img_as_float(hybrid)

plt.figure(figsize=(30,30))
plt.imshow(hybrid)
plt.show()
In [43]:
# A 5 level stack of hybrid 3
imname = './images/hybrid3_stacks.jpg'
hybrid = skio.imread(imname) 
hybrid = sk.img_as_float(hybrid)

plt.figure(figsize=(30,30))
plt.imshow(hybrid)
plt.show()

1.4 Multiresolution Blending

We use the Multiresolution Spline technique described in the paper by Burt et al. for combining two or more images into a larger image mosaic. This method is applied in a layered manner, where we take advantage of the Gaussian and Laplacian stacks in order to establish a visibly appealing blended image.

High level procedure explanation

  1. Read in 2 images and a mask
  2. Make a Gaussian stack of the mask in ordre to have subtle transition from an img1 to img2 within the blended image
  3. Make a Laplacian stack of img1 and img2
  4. For every layer of the stacks, combine the two images in the Laplacian stacks by adding the two images with the corresponding mask from the Gaussian stack.

Bells and Whistels

I'm using color!!

The Oraple

In [44]:
# original images
imname = './images/orange.jpeg'
orange = skio.imread(imname) 
orange = sk.img_as_float(orange)
imname = './images/apple.jpeg'
apple = skio.imread(imname) 
apple = sk.img_as_float(apple)

plt.figure(figsize=(10,10))
plt.subplot(1,2,1)
plt.imshow(orange)
plt.subplot(1,2,2)
plt.imshow(apple)
plt.show()

Creation of the Oraple

In [46]:
imname = './images/oraple_layers.png'
layers = skio.imread(imname) 
layers = sk.img_as_float(layers)

plt.figure(figsize=(30,30))
plt.imshow(layers)
plt.show()

Start from the rightmost column. Take the last level of the laplacian stack for the apple and the orange, and mask them with a blurred mask taken from the gaussian stack of the mask (row 1 and 2 of this column). Add these two levels of detail to the last level of the gaussians of the masked apple and orange (not shown in this image) to get the last image in this column. Move to the left column and add the levels of detail corresponding to this column in the 2 laplacians and add them to the image we had before to get the last image in this column. Repeat for every column.

Result - Oraple!

In [50]:
# multiresolution blend vs. naive blend
imname = './images/oraple.jpg'
close = skio.imread(imname) 
imname = './images/pasted.jpg'
far = skio.imread(imname)

close = (sk.img_as_float(close))
far = (sk.img_as_float(far))
plt.figure(figsize=(10,10))
plt.subplot(1,2,1)
skio.imshow(close)
plt.title("multiresolution")
plt.subplot(1,2,2)
skio.imshow(far)
plt.title("naive")
skio.show()

Three-eyed Cyclops

In [54]:
# original images
imname = './images/eyes.png'
eye = skio.imread(imname) 
eye = sk.img_as_float(eye)


plt.figure(figsize=(8,8))
plt.imshow(eye)
plt.show()

Creation of the three-eye cyclops

In [52]:
imname = './images/eyes_layers.png'
layers = skio.imread(imname) 
layers = sk.img_as_float(layers)

plt.figure(figsize=(30,30))
plt.imshow(layers)
plt.show()

Result

In [53]:
# multiresolution blend vs. naive blend
imname = './images/eyes_blended.jpg'
close = skio.imread(imname) 
imname = './images/eyes_pasted.jpg'
far = skio.imread(imname)

close = (sk.img_as_float(close))
far = (sk.img_as_float(far))
plt.figure(figsize=(10,10))
plt.subplot(1,2,1)
skio.imshow(close)
plt.title("multiresolution")
plt.subplot(1,2,2)
skio.imshow(far)
plt.title("naive")
skio.show()

Please note that for this image I had to go through more steps than with the oraple. First, instead of having two different source images to blend together as in the case with the oraple, I'm taking an eye from the source image and putting on the forehead of the same source image. To be able to do this, I had to make a new copy of the source so that this new image's eye would align with the others forehead. For this purpose I used the starter code that was provided for the hybrid images in 1.2. Then I made a mask for the new source image using the started code that was provided for 2.2. The rest of the steps are the same as the oraple.

you can see the aligned source images and the mask below.

In [56]:
imname = './images/i1.png'
s1 = skio.imread(imname) 
imname = './images/i2.png'
s2 = skio.imread(imname)
imname = './images/i2_mask.png'
mask = skio.imread(imname)

s1 = (sk.img_as_float(s1))
s2 = (sk.img_as_float(s2))
plt.figure(figsize=(10,10))
plt.subplot(1,3,1)
skio.imshow(s1)
plt.title("eye")
plt.subplot(1,3,2)
skio.imshow(s2)
plt.title("forehead")
plt.subplot(1,3,3)
skio.imshow(mask)
plt.title("mask")
skio.show()

Gausian and Laplacian Stack of the Cyclops

In [57]:
imname = './images/eyes_stacks.png'
layers = skio.imread(imname) 
layers = sk.img_as_float(layers)

plt.figure(figsize=(30,30))
plt.imshow(layers)
plt.show()

First row: Gaussian stack

Second row: Laplacian stack

Part 2 - Gradient Domain Fusion

In this part of the project we takes advantage of the fact the human eye is tuned to the gradients and process the images in the gradient domain. As we know our visual perception is highly sensitive to changes, rather than real values, and so we a task like blending would theoretically be very suitable to such novel techniques.

Part 2.1 - Toy Problem

In this part we effectively check if theory aligns with application, or more correctly, make sure our math is correct.
Our problem is as such: If we have 1 known pixel and the changes (gradients) of the image, are we able to perfectly reconstruct the image?
We have formulated the problem as a linear algebra problem, specifically as Ax=b, using 3 questions:

  1. minimize ( v(x+1,y)-v(x,y) - (s(x+1,y)-s(x,y)) )^2 the x-gradients of v should closely match the x-gradients of s
  2. minimize ( v(x,y+1)-v(x,y) - (s(x,y+1)-s(x,y)) )^2 the y-gradients of v should closely match the y-gradients of s
  3. minimize (v(1,1)-s(1,1))^2 The top left corners of the two images should be the same color

where s denotes the source vector, and v is our result vector.

We will try this on an image from Toy Story!

In [63]:
# Read, convert, and display image
imname = './images/downscale.jpg'
source = skio.imread(imname) 
source = sk.img_as_float(source)
plt.imshow(source, cmap='gray')
plt.title("Source")
plt.show()

Result

In [74]:
imname = './images/reconstruct.jpg'
recon = skio.imread(imname) 
recon = sk.img_as_float(recon)
plt.figure(figsize=(20,20))
plt.subplot(1,2,1)
plt.title("Source")
plt.imshow(source, cmap='gray')
plt.subplot(1,2,2)
plt.title("Reconstructed")
plt.imshow(recon, cmap='gray')
plt.show()
print("magnitude of difference:",np.linalg.norm(np.abs(recon-source)))

magnitude of difference: 7.589340751418541

Conclusion

Reconstructed image looks good with very minimial error, thus our math is correct -> we can continue!

2.1 Multiresolution Blending

In the last part of the project we use the Poisson Blending technique described in the paper by Perez et al. for seamless editing of image regions. This technique follows closely to what we have done just prior in section 2.1, where me use the gradients while keeping some known values to reconstruct. The difference in this part is we inject the source's gradient patch into the target, leaving the border pixels as is, and solving it using least squares method. This will allow us to transition from one image to another without the need to white balance.

High level procedure explanation

  1. Read in 2 images
  2. Generate a mask to define the borders of the injection site
  3. Take the gradient of the source image
  4. Inject the patch into the corresponding location within the target
  5. Equate border pixels to correspodning target pixels.
  6. Solve least square problem to seamlessly blend the two images

Penguins in Tahoe!

In [76]:
# source images
imname = './images/penguin-chick_newsource.png'
penguin = skio.imread(imname) 
imname = './images/im3.jpg'
tahoe = skio.imread(imname)

penguin = sk.img_as_float(penguin)
tahoe = sk.img_as_float(tahoe)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(penguin)
plt.subplot(1,2,1)
plt.imshow(tahoe)
plt.show()

Masks

In [77]:
# masks images
imname = './images/penguin-chick_mask.png'
penguin = skio.imread(imname) 
imname = './images/im3_mask.png'
tahoe = skio.imread(imname)

penguin = sk.img_as_float(penguin)
tahoe = sk.img_as_float(tahoe)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(penguin)
plt.subplot(1,2,1)
plt.imshow(tahoe)
plt.show()

Result

In [99]:
# source images
imname = './images/out_penguin.jpg'
blended = skio.imread(imname) 
imname = './images/pasted_penguin.jpg'
pasted = skio.imread(imname)

penguin = sk.img_as_float(blended)
tahoe = sk.img_as_float(pasted)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(blended)
plt.title("belnded")
plt.subplot(1,2,1)
plt.imshow(pasted)
plt.title("naive")
plt.show()

Sunbathing in the South Pole!!

In [83]:
# source images
imname = './images/sunbathing.jpg'
penguin = skio.imread(imname) 
imname = './images/southpole.jpg'
tahoe = skio.imread(imname)

penguin = sk.img_as_float(penguin)
tahoe = sk.img_as_float(tahoe)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(penguin)
plt.subplot(1,2,1)
plt.imshow(tahoe)
plt.show()

imname = './images/sunbathing_newsource.png'
penguin = skio.imread(imname)
penguin = sk.img_as_float(penguin)
plt.imshow(penguin)
plt.title("rescaled source to fit target image")
plt.show()

Masks

In [84]:
# masks images
imname = './images/sunbathing_mask.png'
penguin = skio.imread(imname) 
imname = './images/southpole_mask.png'
tahoe = skio.imread(imname)

penguin = sk.img_as_float(penguin)
tahoe = sk.img_as_float(tahoe)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(penguin)
plt.subplot(1,2,1)
plt.imshow(tahoe)
plt.show()

Result

In [98]:
# source images
imname = './images/out_sunbathing.jpg'
blended = skio.imread(imname) 
imname = './images/pasted_southpole.jpg'
pasted = skio.imread(imname)

penguin = sk.img_as_float(blended)
tahoe = sk.img_as_float(pasted)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(blended)
plt.title("belnded")
plt.subplot(1,2,1)
plt.imshow(pasted)
plt.title("naive")
plt.show()

Anatomy isn't for everyone!!

In [87]:
# source images
imname = './images/disgust.jpg'
penguin = skio.imread(imname) 
imname = './images/rembrandt.jpg'
tahoe = skio.imread(imname)

penguin = sk.img_as_float(penguin)
tahoe = sk.img_as_float(tahoe)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(penguin)
plt.subplot(1,2,1)
plt.imshow(tahoe)
plt.show()

imname = './images/disgust_newsource.png'
penguin = skio.imread(imname)
penguin = sk.img_as_float(penguin)
plt.imshow(penguin)
plt.title("rescaled and rotated source to fit target image")
plt.show()

Masks

In [88]:
# masks images
imname = './images/disgust_mask.png'
penguin = skio.imread(imname) 
imname = './images/rembrandt_mask.png'
tahoe = skio.imread(imname)

penguin = sk.img_as_float(penguin)
tahoe = sk.img_as_float(tahoe)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(penguin)
plt.subplot(1,2,1)
plt.imshow(tahoe)
plt.show()

Result

In [97]:
# source images
imname = './images/out_good.jpg'
blended = skio.imread(imname) 
imname = './images/pasted_good.jpg'
pasted = skio.imread(imname)

penguin = sk.img_as_float(blended)
tahoe = sk.img_as_float(pasted)

plt.figure(figsize=(30,30))
plt.subplot(1,2,2)
plt.imshow(blended)
plt.title("belnded")
plt.subplot(1,2,1)
plt.imshow(pasted)
plt.title("naive")
plt.show()

Bad exmple of blending

In [92]:
# source images
imname = './images/source.jpg'
penguin = skio.imread(imname) 
imname = './images/rembrandt.jpg'
tahoe = skio.imread(imname)

penguin = sk.img_as_float(penguin)
tahoe = sk.img_as_float(tahoe)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(penguin)
plt.subplot(1,2,1)
plt.imshow(tahoe)
plt.show()

Masks

In [94]:
# masks images
imname = './images/mask.jpg'
penguin = skio.imread(imname) 

penguin = sk.img_as_float(penguin)
plt.imshow(penguin)
plt.show()

Notice that since the source image is aligned with the injection site, we can use only one mask

Result

In [96]:
# source images
imname = './images/out_bad.jpg'
blended = skio.imread(imname) 
imname = './images/pasted_bad.jpg'
pasted = skio.imread(imname)

penguin = sk.img_as_float(blended)
tahoe = sk.img_as_float(pasted)

plt.figure(figsize=(16,16))
plt.subplot(1,2,2)
plt.imshow(blended)
plt.title("belnded")
plt.subplot(1,2,1)
plt.imshow(pasted)
plt.title("naive")
plt.show()

Even though Possion blending tried its best and managed to blend parts of the image, this example didn't work so well. My guess is the reason for it has to do with 2 facts. The lighting in the two images is too different. the source of light in the two images are on opposite sides so the gradients were not agreeble enough. Also the source image gradients around the borders of the mask are False values. The source image has a white background with the image of the man inserted into that white background artificially. That means that the gradients around the borders of the mask in the source image don't capture the changes in the lighting of the image quite right.

For the last part I will compare the cyclops blending using the two different methods

In [102]:
imname = './images/posion_eye.jpg'
imname = './images/eyes_blended.jpg'
blended = skio.imread(imname) 
imname = './images/eyes_pasted.jpg'
pasted = skio.imread(imname)

poisson = sk.img_as_float(poisson)
blended = sk.img_as_float(blended)
pasted = sk.img_as_float(pasted)


plt.figure(figsize=(16,16))
plt.subplot(1,3,1)
plt.imshow(poisson)
plt.title("possion")
plt.subplot(1,3,2)
plt.imshow(blended)
plt.title("multiresolution")
plt.subplot(1,3,3)
plt.imshow(pasted)
plt.title("naive")
plt.show()

Comparison Conclusion

Multi resolution blending seems to be doing better thatn possion blending here, because here we don't actually want the eye to be following the same color pattern as the forehead, which is what possion is doing. We watn the edges of the eye (the seem) to be smoothed out with the forehead but we want to keep the colors of the eye intact so that it looks similar to the other two eyes.