CS 194-26 Final Project: Vertigo Shot and Seam Carving

by Jessie Yang, cs194-26-agc

Contents


Introduction

For the final project of CS 194-26, we were allowed to choose two out of several pre-canned projects to complete. I decided to complete the Vertigo Shot and the Seam Carviing projects.


Project: Vertigo Shot

In this project, we tried to imitate the famous "dolly zoom" or "vertigo shot" effect. To produce this effect, one must move either the camera closer to the subject (or the subject closer to the camera) while simultaneously increasing the camera's field of view, or FOV. As a result, the subject appears to stay the same size within the image, even as the background scene has zoomed out/in.

To reproduce this effect ourselves, my friends and I wandered around Berkeley's campus, taking pictures with my Nikon D5100 DSLR and tripod. First, we set our camera's focal length to 200mm and stood far away from our subject, aligning the black-dot indicators on the camera's viewfinder to the subject and making a note of this alignment. Next, we zoomed out to 135mm (then 100mm, 70mm, 55mm) and walked closer to the subject until the indicators aligned with the subject in the same way as they did before. This resulted in two sequences of 4-5 images (I threw out 1-2 images if they seemed too misaligned with the rest of the sequence) that produced the effect we wanted. Of course, since we largely eyeballed the alignment between focal lengths, the subject may not stay in exactly the same position between photos. Still, the results can be seen below!

My Friend Sean

Here he is standing in front of the Campanile!




A Cool Bench

Here's a bench found next to the Memorial Glade, in front of Doe Library!




Something else I tried to do was reproducing the effect with actual video, rather than simply taking a series of still photographs. Unfortunately, I didn't have an actual dolly or any kind of moving platform... Thus, I merely winged the technique by walking away from my subject while zooming out and trying to keep my subject the same size in the frame. This produced really shaky videos that I ended up stabilizing a bit in post, but the vertigo effect can still be seen in the following gif!

My Friend Nikhil

Here he is also standing in front of the Campanile! Nyoom~


Project: Seam Carving

In this project, we implement the technique of seam carving, which allows us to resize images in a content-aware way. That is, plain resizing along a single dimension may squish the important objects in an image. Seam carving allows us to better preserve the most important content by taking it into account when scaling the image down.


Algorithm

To figure out what content we want to keep, we must use an energy function on each pixel of the image. Then, we find "seams" or paths through the image with minimal "energy" costs and remove them. Intuitively, removing the minimal cost seams will disrupt the content of the image the least, since we are leaving the most important, "high energy" details behind.

Below, we can see the result of applying the dual gradient energy function to one of our images. The dual gradient energy function sums the square magnitudes of both the x- and y-gradients of a pixel, essentially discovering the details of the image and giving them the highest "energy."

Image source.
Image with energy function applied. The areas with most detail (e.g. the tree) are unlikely to be seam carved.

Now that we have the energy map, we can use a dynamic programming algorithm to find the minimal cost path from top to bottom (vertical seam) or left to right (horizontal seam).

First vertical seam carved in this image.
Image source.
First horizontal seam carved in this image.
Image source.

Once we find the seam as a list of columns (one for every row in a vertical seam) or rows (one for every column in a horizontal seam), we can remove these pixels from the image and get back a seam carved image. We then repeat the seam finding and seam removal for several iterations to arrive at a resized image. The results can be found in the following section.


Results


Hadrian's Wall

Image source.

The same image vertically carved down by a third, i.e. to two-thirds of its original size. Much of the field to the left and right of the wall have been carved out while preserving almost all of the wall and the stony paths around it.


Night Barn

Image source.

The same image vertically carved down by a third. The seam carver once again mostly removes the field to left and right field of the barn, maintaining the barn's structure intact even as it condenses the landscape around it.


Petra, Jordan

Image source.

The same image vertically carved down by a third. The night sky is a lot more centered now, after a lot of the right-hand rock wall and a little of the left-hand rock wall has been carved away.


Taiga

Image source.

The same image vertically carved down by a third. This is a lot closer to a square aspect ratio while preserving much of the content (minus a few mountain peaks) without distortion.


Mountain Path

Image source.

The same image horizontally carved down by a third. We have carved away most of the fog, and the fence is now closer to the center of the image. More interestingly, the seam has changed the texture of the mountaintop. It's as if we have entirely new peaks, and reasonably believable ones at that. Although this is technically just a carving artifact, it's a visually interesting effect and it adds an unexpected artistic flare added to our image.


Rocks at Sunset

Image source.

The same image horizontally carved down by a third. We're much closer to a square ratio now while keeping most of the rock face the same. There are a few artifacts at the top of the new image, however, since the seam carving doesn't strongly preserve the gradient of the sunset.



Failure Cases

Seam carving can clearly produce some amazing results! However, it is not fool-proof and fails pretty clearly in some situations. Let us examine some of these failure cases.

A photo I took of me and my friends!

After trying to shrink down this photo by a third, I realize that, oh no, the seam carving has eaten away at our bodies and faces and even deleted two of my friends! There is a lot of structure lost in this image which might be for a few reasons. (1) The grass is pretty noisy and detailed in the original, which may deter the algorithm from simply carving away the sides. (2) In our perception, human faces and bodies have really distinct structures and features, so the imperfections of the image are especially obvious to us, even if they aren't as obvious to the algorithm.


A graduation photo of my friend Luke that I took!

After trying to shrink down this photo by a third... oh no, where did Luke's face go??? It seems that the algorithm once again doesn't especially care for preserving the structure of human faces, even though we humans would prefer it. Moreover, there is a lot of noisy/energetic detail in the bushes behind Luke's body, which is probably why the algorithm preferred to eat his face instead, since the sky in the background behind his head is a lot simpler.


Image source.

Although the content of the image might be reasonably resized, all of the gradients in the original make it very difficult to produce a satisfactory result. The energy function doesn't take these gradual shifts in color into account, meaning we get pinkish hues juxtaposed with strong blues without any blending.



Seam Carving: What I Learned

One thing I learned from this project is that while our computational tools can do incredible things - such as scale an image down to two-thirds of its original height/width or more while largely preserving its contents - there is still a human touch required to produce the photographs and results we want. Although I can arbitrarily decide to cut away a chunk of an image, this doesn't always produce successful results. Sometimes, you just get fewer artifacts and thus better results if you reduce the number of seams you carve, and there is a tradeoff between doing this and improving the composition of an image. At the end of the day, you'll have a photographer, an artist deciding what kind of tradeoffs they'd like to make - better quality or better proportions? Or a host of other considerations.

Nevertheless, sometimes the computational tool can produce unexpected results that are still surprisingly likeable, such as in the "Mountain Path" image discussed above. In other words, there exists a lovely synergy and synthesis between an artist and their tools...


Acknowledgements

Thanks to my friends Nikhil and Sean for being models in my vertigo shots!

Besides the photographs I took, I took images from the following sites: