Final Projects

Seam Carving and Vertigo Shot     |     Eric Zhang (agq)
Seam Carving

Overview

Seam carving is a method of content-aware image resizing. Rather than cropping pixels out of the side, or simply downscaling the pixels uniformly, seam carving attempts to remove the pixels that have low energy. Energy is a measure of how important a pixel is to an image. In this implementation, energy is determined by the sum of a pixel's discrete gradients (change relative to its surrounding pixels). Then an 8-connected path of pixels (either vertical or horziontal path) with the least energy is found and removed. This is the seam that is removed. 8-connected simply means that for removal of a horizontal seam, only one pixel is removed from each row and each removed pixel must be in the same column or diagonally adjacent the pixels being removed in the rows above and below.

Energy

Each pixel is labeled with an energy. Warmer colors (closer to red on the spectrum) on the energy map indicate higher energy whereas cooler colors (closer to violet on the spectrum) indicate low energy.

Original Image
Energy Map

Finding the Best Seam

Next the lowest cost path is found through the image in the direction of the seam removal. We will use removal of a vertical seam as an example (reducing the width of the image by 1 column). The cost of each pixel in the first row is simply the energy of the pixel itself. Starting from the second row, the lowest cost of any connected path from the top of the image to that pixel is computed given the cost of reaching each pixel from the previous row. In implementation, for each pixel in row i this is simply the minimum of the accumulated path cost of the 3 adjacent pixels in row i - 1. The path is recovered by starting at the pixel with the lowest accumulated cost in the bottom-most row after the above process, and following the path of lowest adjacent accumulated cost up each row. Below are examples of the lowest cost vertical seam and horizontal seam, which is computed in a similar fashion, but from left to right.

Lowest Cost Vertical Seam
Lowest Cost Horizontal Seam

This process is repeated as many times as the number of seams needed to be removed to acheive the desired size reduction. After each seam removal, the gradient of the image is recomputed and the process is repeated. Ideally, this results in a size reduction that is able to maintain aspect ratio of some of the important parts of the image, which naive resizing does not.

Seam Carved Reduction
Regular Resizing

In this example, the height of the image is being reduced. At first, the difference between the seam carved version and the naive resizing may seem subtle or non-existent. However, in the naive version, the mountains are noticably compressed, whereas in the seam carved version, their height is maintained. The same goes for the rock at the bottom left. In the naive version, the ratio of the rock is noticably different than the original, whereas in the seam carved version, it is relatively untouched by the resizing. This is because the rock was an area of high energy, as indicated by the energy map.

It's important to note that seam carving does not always do better (subjectively) than the naive method in both dimensions. Below is a width reduction of the same image using both techniques.

Seam Carved Reduction
Regular Resizing

Again, the seam carved version preserves the aspect ratio of the rock. However, the seam carving visibly alters the appearance of the mountain. Shown in greater detail in examples later, seam carving does not perform very well with diagonal lines.

Real Time Resizing

The above process is very costly, especially if the same image needs to be resized into many different sizes in real time. It is recomputing gradients each time and finding accumulated path costs to each pixel and finding the best seam path through it each time. It would be nice to precompute some of these things and then allow real-time resizing.

To acheive this, we sacrifice a bit of accuracy. Starting with the enrgy map, at each iteration the lowest cost path (seam) through the energy map in the seam reduction direction is found and removed from the enrgy map itself. The pixels of the real image associated with that seam are labeled with the iteration number. In effect, this is labeling each pixel with the seam index, which seam it belongs to and thus at which iteration of seam reduction it should be removed. Now removing k columns from an image is as simple as keeping only pixels with seam indices greater than or equal to k. Below a heat-map of the indices are show. Cooler colors indicate a lower seam index number (removed earlier) and warmer colors indicate high seam index number (only removed in extreme size reductions).

Seam Indices for Vertical Seams
Seam Indices for Horizontal Seams

With the seam index map, image resizing can occur as quickly as the pixels can be copied. One minor caveat is that the process is not as optimal as the one above due to the fact that after removing a seam from the image, the gradients (aka the energy map) will have changed for the pixels surrounding the seam.

Optimal Seams Height Reduction
Precomputed Seams Height Reduction

Pretty hard to tell the difference, huh. Here's an AB version to help.

Other Examples

The Great Wave of Kanagawa
Energy Map
Seam Carved Reduction
Regular Resizing
The Campanile
Energy Map
Seam Carved Reduction
Regular Resizing
It's A Small World
Energy Map

Notice how the proportion of ocean decreases in the seam carved reduction.

Seam Carved Reduction
Regular Resizing

Failed Examples

As noted earlier, seam carving in one dimension does not work particularly well with diagonal lines. This phenomenon is more noticable in the following examples.

Datacenter
Energy Map
Seam Carved Reduction
Regular Resizing
Golden Gate Bridge
Energy Map
Seam Carved Reduction
Regular Resizing

Image Sources

Vertigo Shot (Dolly Zoom)

Overview

In this section of the project, I explore the dolly zoom effect, attributed to Alfred Hitchcock as the popularizer of the technique in his film Vertigo. This effect is achieved by decreasing distance from the camera to the subject while decreasing focal length. The reverse is also has a similar effect, increasing distance while increasing focal length. This keeps the subject the same size in the frame while either reducing or increasing the angle of view of the background.

Examples

Notice the amount of background that is visible change.