Overview

In this project, we recreate a method of content-aware image-resizing in order to be able to crop images without losing important information.

Images

Energy

We begin by calculating the energy of each pixel in the image in order to find the least important path. Our energy function is the absolute value of the gradient of the pixel in the horizontal direction plus the absolute value of the gradient of the pixel in the vertical direction. Finding the gradient of a pixel in a direction involves convolving the image with a directional Sobel filter.

Image Energies

Carving Seams

We proceed to use dynamic programming to find the vertical (for now) seam of pixels whose energies are the lowest. This is computed by iterating through every row (starting at the second) and setting the energy of each pixel as the energy of that pixel plus the minimum energy of the three parent pixels, while storing the backtrack paths for efficiency. We then go through the minimum path and remove it. We repeat this until the desired width is reached.

Seams [Crop Percentage]

Results [Crop Percentage]

The first photo above highlights an interesting quirk of this algorithm in the mountain peak having been morphed into a clif. This algorithm actually changes the content of the image. This is shown very well in the second photo as well, in which the drone is farther from David and the rock surface is less menacing initially.

Vertical Seam-Carving

Vertical seam-carving (that is, carving horizontal seams to crop the image vertically) is actually the same problem as horizontal seam-carving, only transposing the image before operating on it.

Once again, the story of the photo has changed immensely. The tone has changed from leisure to action. It now looks like the boats are weaving between narrow openings in a pack of much denser islands.

The simple energy function described above has problems with certain types of images, which can be seen below.

Bobbleheads [Crop Percentage]

Seam-Inserting

We next insert seams by first computing the set of seams to insert at using the energy function described above. We then insert an average of the pixels at each of those seams. Precomputing the seams to insert at minimizes stretching effects on the image since we do not reselect inserted seams to insert at again, and it also makes the program much faster.

Lessons

It is clear that cropping a photo in the conventional sense from the edges in can change a photo's story. It can remove undesired outliers or change the center of attention. Seam-carving takes this to a new level. It can actually change the shape of elements in the photo and the space between them. We could go a step further and selectively target features in the photo for removal or distortion. This is why this sort of cropping should be disclosed when performed.