Final Project

CS194-26 Fall 2018

Bernie Wang, cs194-26-afg





In this project, we create fake miniatures by simulating tilt-shift photography that is normally achieved using selective focus cameras. We are effectively narrowing the perceived depth of field in a scene and creates the illusion that it was taken with a lens was really close to the subject. To achieve this effect, we first create a mask around our desired subject. We then use the mask to blur the background such that regions that are farther from the subject are increasingly blurred.





Bells & Whistles

I implemented complex DOF regions using object masks. By using scikit-image's binary dilation function, we can grow the masked region so that we end up with multiple masks of different sizes. Then we can blur the defined background in regions, with regions that are farther from the subject being blurred more than regions that are closer to the subject.

Santorini, original

Santorini, mask



Santorini, miniaturized

Santorini, miniaturized & saturation-boosted





More Results

THe following images are more results of creating fake miniatures. The left images are the source images, and the right are the fake miniatures, where they are the blurred using dilated masks and then saturation-adjusted to increase the illusion of miniatures.

Berkeley, original

Berkeley, miniaturized & saturation-boosted



Safeco Field, original

Berkeley Field, miniaturized & saturation-boosted



Sproul Hall, original

Sproul Hall, miniaturized & saturation-boosted





Summary

This has been a fun project. I think this is very relevant with shallow DOF being the current trend and computional photography being integrated into every smartphone camera. It is amazing how DOF and percieved quality of a photo can be changed by a Python/Matlab script, altough the quality can never be as good as DOF from a lens due to imperfections of drawing masks. I think with edge detection and/or image segmentation algorithms, we can improve drawing masks for subjects we want to focus on.





In this project, we implement seam carving, a content-aware image resizing technique. A seam is a connected path from one side of the image to the other that chooses exactly one pixel from each column (or row). We remove seams until we end up with an image of our desired size. The resulting image should be a cropped image where we remove "unimportant" details.





Algorithm

First, we compute an energy map, which ranks pixels by "importance". For this project we use x and y gradient to compute an energy map.

Gradient energy function



Next we find a path of minimal total energy. We compute the minimal total energy using dynamic programming - we first traverse the image from the second-to-last row to the first row computing the minimal total energy up to each pixel while also keeping track of its neighbor of minimal total energy below:

Cumulative energy equation



Then we backtrack starting from the pixel with the smallest total energy in the first row all the way to the last row to obtain our seam to remove.





Results

Te following images are more results of seam carving. The left images are the source images, and the right are the resized images.



Vertical Seam Removal

Sky Tree, original

Sky Tree, with vertical seams removed



Hawaii, original

Hawaii, with vertical seams removed



Supertrees, original

Supertrees, with vertical seams removed





Horizontal Seam Removal

We can rotate the image to remove horizontal seams

Piling, original

Piling, with vertical seams removed



Hiroshima, original

Hiroshima, with horizontal seams removed





Failures

The choice of energy map and image to resize influence the quality of seam carving. Here, we used the gradient energy map on this image of a bullet train. Since the surface of the bullet train is smooth and curvy, some seams carve into it.

Shinkansen, original

Shinkansen, with vertical seams removed



Here it's easy to see where the seams were removed by the obscured braces.

Golden Gate Bridge, original

Golden Gate Bridge, with vertical seams removed



Bells & Whistles



Seam Insertion

I implemented seam insertion (described in paper by Shai Avidan and Ariel Shamir). The algorithm runs as follow: first find k seams found by seam carving. After collecting all the seams, we then add new seams next to the seams found in reverse order to the original origin, where the new artificial seams is computed by taking the average of seams to the left and right of the old one.

Sky Tree, original

Sky Tree, with vertical seams inserted



Supertrees, original

Supertrees, with vertical seams inserted





Object Removal

I also implemented object removal using seam carving and seam insertion. I first make a mask for the object I want to delete. I set the values in the mask such that the masking region for the object is negative and the rest of the mask to be zero. Then for seam carving I multiply the mask with the energy map, keeping the rest of the algorithm the same. Then I inserted vertical seams to resize the image back to the original size.

Hawaii, original

Hawaii, with tree removed

Sky Tree, original

Sky Tree, with Sky Tree removed and vertical seams inserted









Summary

Seam carving is a simple, yet functional approach to image resizing. Using the right energy function, it can work beautifully. The use of dynamic programming is elegant, although it can be computationally expensive if you want to remove many seams on large images.