Computer Science 194-26: Image Manipulation and Computational Photography

Final Project

Ari Pickar

Seam Carving

This project implemented the algorithm for content aware image resizing. This algorithm finds the importance of each pixel with an energy function, and then computes the minimum cumulative energy from the top to bottom of the image, removing all of the pixels along the seam to resize the image. By repeating this until an image is of the desired size, the image can be resized in a way that preserves the focus of the image.

Energy function

To define the importance of each pixel, I used an energy function. Initially I tried to use scipy.ndimage.sobel operator as the way to define the energy, as that convolves the image with two kernels to find the approximate derivative. As a result, I decided that it would be best to use a different version of the operator, cv2.Scharr. By convolving each of the channels, finding the absolute value of the result, and summing them, I was able to get an accurate image of the energy of each pixel.

Computing the Cumulative Energy

In order to get the minumum seam, I took the energy map that had been created, and found the pixel with the minimum value for the first row. Then for each subsequent row, the cumulative minimum energy for that pixel is determined by the equation , where e(i,j) is the energy at pixel i,j and M(i,j) is the minimum energy of pixel i,j . This allows us to use dynamic programming to create cumulative energies for all pixels.

Removing Minimum Energy Seam

Since the cumulative energy is now calculated, backtracking can be used to find the minimum seam. We start by creating a mask of the same shape and size as the image with the pixel the minimum energy on the bottom row, and continuously add the pixel ajacent above it with the minimum energy in the row above, until we get to one pixel in every row. From there, we can mask the image to remove the low energy pixels, and resize the image to remove the gaps in the image. By repeating this until we have the desired number of seams removed, we can complete the resize.

Removing Horizontal Seams

Since the program is designed to remove vertical seams, we can remove horizontal seams by first rotating the image, removing vertical seams from the image, and then rotating the image back.
Original Image: Dog Original Image: Falls Original Image: Boalt
Carved to 50% of Height Carved to 75% of Height Carved to 75% of Width
Original Image: Lake Original Image: Wildfire Original Image: Rocks
Carved to 75% of Height Carved to 75% of Width Carved to 75% of Width

Failures

While this algorithm was usually very successful, it did not discriminate amongst different elements of the image. For the first image, the trolley was made smaller, while the buildings stayed the same size, leading to an ugly bottom of the image. For the second image, the football field was distorted.
Failure: California Street Failue: Neyland Stadium
Carved to 75% of Height Carved to 75% of Width

What I Learned

I think that the biggest takeaway from this project was about how much the choice of energy function matters. Initially I had tried to use a simple sobel edge detector, but this caused a poor implementation of the seam carving, so by switching the energy algorithm, I was able to get much better results.

Fake Miniatures

For this project, I simulated the effect of tilt-shift photography, which is a way of simulating a very large depth of field in a way that makes the image look like its of a miniature version of the actual scene.

Creating the fake minatures

The first part of creating the fake minature was to use ginput to select a focus line. From there, the image was split into two, the top and bottom. Then, the bottom part was convolved with a gaussian kernel to blur it. This split and then gaussian convolve algorithm was repeated until the last bottom had a total height of 50 pixels. Then, the image was flipped, to have the top repeatedly convolved. This meant that the image had the illusion of depth in it. Since convolving something is commutitive, repeatedly convolving the image with a gaussian kernel had the same effect as repeatedly convolving the gaussian kernel with another one, so it therefore had the stronger distortion.
Law school IHouse
Falls Nyc