Final Projects

Seam Carving

The first project I chose to do was seam carving. The main idea is that we want to be able to resize an image while still preserving the “interesting” parts of it. We can do that by finding ‘seams’ to remove instead of naively removing straight up columns and rows from an image to resize it.

A really good overview of this project can be found here: https://datastructur.es/sp17/materials/hw/hw5/hw5.html

Energy Function

The energy function is as described in the specification linked above. It’s a dual-gradient function, using the R/G/B values from each color channel for every pixel.

Finding a Seam

First, we need to compute the minimum energy path from the top to the bottom. This is the first pass; the base case is the first row/col depending on whether you are carving vertically or horizontally.

Next, in the second pass we can actually find the indices that correspond to a seam. We just need to choose the pixel with the minimum energy path, then from the pixels ‘connected’ to it, continue choosing pixels with the minimum energy path until we reach the other side of the image.

Carving Output

set2a
set2a - 67 horizontally carved
set3a
set3a - 22 horizontally carved
set4a
set4a - 30 horizontally carved
set4a - 101 horizontally carved

Failure Cases

Because the algorithm is not incredibly robust, and most of my input images have a lot of similar-looking space horizontally (i.e clouds, ocean etc), I thought it would perform poorly when carving many vertical seams.

Surprisingly it did pretty poorly even with a few vertical seams for one input:

set2a - 30 vertically carved
set3a - 102 vertically carved
set4a - 100 vertically carved

Bells and Whistles

I did pre-compute for bells and whistles; implementation-wise, that just meant I needed to first compute seams as arrays of indices to remove, in order, and then finally just ran the removal function with the pre-computed seams.


Lightfield Camera

The second project I chose was lightfield carving.

I feel like this project was difficult to conceptualize but surprisingly didn’t require too much complexity in code - the results are super cool though so definitely one of my favorites. Pictures being unfocused is basically my biggest fear as a photographer, so being able to adjust and essentially reconstruct your perfect photo is a really interesting concept!

I used two datasets to see if results differed:

Lightfield images are reconstructed by combining many images that are taken over an evenly spaced grid.

First, we simply naively average all the input images.

Eucalyptus Flowers

Lego

Notice that the Eucalyptus Flowers photo does seem to focus on the far side of objects as we expect, while the Lego Knights doesn’t really.

Depth Focusing

A key concept here is that the grid is 17x17 and evenly spaced. This makes it easy to align all images to one “point of view” by shifting from one to another by calculating a physical offset and scaling it (I used np.roll).

Naively, we can just use the values that are in the filenames of the Stanford dataset instead of relying on them being exactly equally spaced.

However, the choice of ‘target’ image to align to isn’t actually important. Aligning all the images will always produce the same image, if shifted a little overall. Thus, the variable we must change for depth focusing is actually how much to shift by, i.e the scaling factor we apply to the physical offset.

I found that the “best” values varied by image subject and image size. Past a certain threshold, the entire image will just appear blurred.

Example

I’ve included some output labeled by where it focuses the photo.

Eucalyptus Flowers

“Center” “Mid-back” “Far back”

Lego Knights

“Center” “Back” “Front”

Aperture

The results are even better when you combine the ‘focus’ effect by shifting along with taking a different number of images for the average to simulate different apertures, so I used factors from the previous part which made it focus on the “center” of the image.

For a “small” aperture photo, I used only a 5x5 grid of input images; “medium”, 10x10 grid; “large”, the original 17x17 grid.

Eucalyptus Flowers

Small Medium Large

Lego Knights

Small Medium Large