CS194-26: Image Manipulation and Computational Photography


Final Projects


Project 1: Seam Carving

Overview

For this project, I implemented seam carving. The goal of this project was to shrink an image by removing the least important "seams," where a vertical seam is defined as a connected path of pixels that contains one pixel from each row, and where a horizontal seam contains one pixel from each column. Removing a vertical seam shrinks the image by a total of one column, and removing a horizontal seam shrinks the image by a total of one row.

Process

To accomplish this, I first identified a way to determine the "importance," or energy, of each pixel. To do this, I used the energy function defined in the paper, which is the sum of the absolute value of the x and y gradients of the pixel. The energy of a seam is defined as the sum of the energy of each pixel in the seam.
Until the desired amount of rows/columns were removed, I identified the seam (horizontal or vertical) with the smallest total energy and removed it. I then recomputed the energy of each pixel in this new image, and repeated the process.

Results

Here are some resized images that turned out pretty well!

Example 1: Dali

For this example, I used Dali's painting The Persistence of Memory. I think that part of the reason resizing this image worked so well is that it is a surrealist painting, and not a realistic scene, so it doesn't seem "wrong" when the objects in the image get distorted. We do not have expectations for what the objects in the scene look like, so it doesn't alarm us when the objects change. The algorithm also consistently removed the empty space in the painting before removing seams from the objects, so the important parts of the image are preserved fairly well.

Here is the painting with 50, 100, 200, and 300 vertical seams removed, respectively.


Here is the painting with 50, 100, 200, and 300 horizontal seams removed, respectively.


Example 2: SF

For this example, I used this picture I took of the Golden Gate bridge.

Here is the picture with 50, 100, 200, and 300 vertical seams removed, respectively.


Here is the picture with 50, 100, 200, and 300 horizontal seams removed, respectively.


Example 3: Beach Sunset

For this example, I used this picture I took of the sunset on a beach in San Diego.

Here is the picture with 100, 200, and 300 vertical seams removed, respectively.


Here is the picture with 100 and 200 horizontal seams removed, respectively.


Example 4: Boats

For this example, I used this picture I took of boats in San Diego.

Here is the picture with 50, 100, 200, and 300 vertical seams removed, respectively.


Here is the picture with 50, 100, 200, and 300 horizontal seams removed, respectively. Removing horizontal seams creates some artifacts in the rigging of the closest boat, but other than that it seems to behave fairly well.


Example 5: Tidepool

For this example, I used this picture I took of a tidepool in Oregon.

Here is the picture with 50, 100, 200, and 300 vertical seams removed, respectively.


Here is the picture with 50, 100, and 200 horizontal seams removed, respectively.


Example 6: Botanical Garden

For this example, I used this picture I took of a botanical garden in Seattle.

Here is the picture with 50, 100, and 200 vertical seams removed, respectively. The reflection begins to get warped when more seams are removed, but the rest of the objects in the picture seem to be fairly well preserved.


Here is the picture with 50, 100, and 200 horizontal seams removed, respectively. Removing horizontal seams preserves the image better than removing vertical seams in this case.


Example 7: Library of Congress

For this example, I used this picture I took in the Library of Congress in DC.

Here is the picture with 50, 100, and 200 vertical seams removed, respectively. Removing vertical seams distorted some of the straight lines in the photo, but preserved most of the geometry and detail in the photo.


Here is the picture with 50, 100, and 200 horizontal seams removed, respectively. Removing horizontal seams preserves the image much better in this case as well.


Results: Failures

Here are some images that didn't turn out so well when I tried to resize them.

Example 1: My Cat

For this example, I used a picture of my cat. At first, removing 50 vertical seams just made it look like my overly large cat had gone on a diet and shrunk to a normal cat size. However, as more vertical seams were removed, his body became more and more warped. Removing horizontal seams warped his body from the beginning, and almost completely removes him from the image as more seams are removed. The reason for this is because there is less change in the gradients of his body (due to his black fur) than there is in the tile behind him, so the seams that cut through his body are deemed optimal by my algorithm and were thus the first seams to be removed.

Here is the picture with 50, 100, and 200 vertical seams removed, respectively.


Here is the picture with 50, 100, and 200 horizontal seams removed.


Example 2: Celebrity Selfie, by Ellen DeGeneres

For this example, I used a celebrity selfie. I was expecting this example to fail as most of the space of the image is taken by faces, and humans are very sensitive to changes/distortions in faces. Surprisingly, removing 50 vertical seams seems to only warp one face, and preserves the rest of the faces. However, removing more seams quickly begins to warp the faces in the forefront, and eventually all of the faces in the photo are destroyed. Once again, removing horizontal seams creates noticeable and drastic changes much faster.

Here is the picture with 50, 100, 200, 300, and 400 vertical seams removed, respectively.


Here is the picture with 50, 100, and 200 horizontal seams removed.


Example 3: Butterfly

For this example, I used a butterfly photo that I took in the Cal Academy of Sciences. The first seams to be removed seem to be the seams that cut through the butterfly's wings, probably because there is a smaller gradient there than there is in the foliage behind the butterfly. Removing horizontal seams creates a more drastic distortion to the wings, but removing vertical seams still creates enough of a distortion to make them look wrinkled and warped.

Here is the picture with 50, 100, 200, and 300 vertical seams removed, respectively.


Here is the picture with 50, 100, and 200 horizontal seams removed.


Bells and Whistles: Diagonal Resizing

I implemented a greedy algorithm to resize both vertically and horizontally (instead of the optimal algorithm in the paper). While there were both rows and colums to remove, I computed the cost of the minimum vertical seam and the minimum horizontal seam, and removed the seam with the smaller energy. Then I repeated the process with the newly resized image. I continued this until there were only rows or columns left to remove, and then used the previous resizing algorithm to finish resizing the image.
Here is the picture of SF (previously used) resized to remove 100 rows and 50 columns.



Project 2: The Vertigo Shot

Overview

For this project, I duplicated the "vertigo shot", or dolly zoom, which makes it look like the background of the photo is compressing or expanding while the subject of the photo remains the same. I used a Nikon D40 camera with an 18-55 mm lens.
To create the effect, I first stood close to my subject and zoomed out as far as I could go. I then took a step (or a few) backwards and zoomed in until the subject was about the same size as before. I continued this until I could zoom in no further. I did my best to keep the camera level and the angle constant, but as I do not have a camera dolly, this is not as constant as it could be.

Examples

Example 1

For this example, I took photos of the concrete ball by the Campanile. I chose this space because there is a long, flat walkway on either side of the ball, and the angles of the trees, benches, and bricks add an interesting visual effect when perspective changes.






Here is an animated gif to make the effect easier to see:

Example 2

For this example, I took photos of my housemate, Rita. I took these photos in the trees by Mulford. I chose this spot because the line of trees created an interesting vanishing point, and the regularly spaced intervals between the trees emphasized the vertigo effect. In this sequence, I had her stand still as I backed away and refocused.






Here is an animated gif to make the effect easier to see:

Example 3

For this example, I took photos of Rita (again), but this time I had her walk away from me (moving the subject) while I stood still and refocused.





Here is an animated gif:

Difficulties

Some difficulties I had were keeping the subject a constant size. In most cases I zoomed in or out too far, but the effect was still achieved.
A difficulty I had with some sequences (not shown) was that as I increased my physical distance, the angle between my camera and the subject changed (particularly when my object was high in the air).