The art of resizing pictures while retaining as much information as possible
Tianhui (Lily) Yang and Kamyar Salahi CS194 SPRING 2020
Cover image courtesy @FelixMittermeier, shown is a carved version
When cropping images to fit a certain dimension we often have to chop off parts of pictures, losing important content in the process. The framing and the style of these cropped pictures are often much more different than how we originally intended. A way to circumvent this tension between constraints and artistic vision is through Seam Carving, which helps locate the least noticeable pixels to crop. Seams are a continuous line of pixels that reaches from one end of an image to its opposite end. In this report we explore how seam carving from “Seam Carving for Content-Aware Image Resizing” (Avidan and Shamir) works and its performance on some royalty free pixabay photos. Attribution for all will be in the results section.
The usual method of cropping images involves taking and slicing edge pixels via a vertical line as shown below:
Though the output still looks great, we lost a noticeable amount of the picture (shown faded). And unfortunately, straight lines don’t fit well with the organic forms of real life objects and scenes. The bushes to the left are gone and it’s clear where our seam starts and ends.
Above is a seam chosen via seam carving. As you can see, it’s pretty irregular. By bending and curving in a multitude of ways, the pixels we remove become less noticeble. In the next section, we examine how these seams are chosen.
Previously, we said that seam carving helps locate the least noticeable pixels. How noticeable a pixel is is defined through an energy function. There are many energy functions out there including, histogram of gradients and Harris corners. We decided to use gradient magnitudes and the Sobel operator for our energy function.
The Sobel operator can be seen below. It takes the difference between pixels among the x and y directions and smooths the difference with an average. This difference can be summed up with the partial derivative. Large differences signify edges and important image content that should be kept. By taking the norm of the x and y convolutions with our image, norm(x, y), we end up with a gradient magnitude mapping for our input image.
Sx = [[1, 0, -1], Sy = [[1, 2, 1], # Sobel edge filters for the x and y directions
[2, 0, 2], [0, 0, 0],
[1, 0 -1]] [-1, -2, -1]]
Our goal in the next section is to identify seams with minimum energy cost. This ensures that we are protecting our high impact content.
To find the least energy seam in the picture, we take the following steps:
Find the cumulative energies for all pixels along a seam starting from the second row using the formula below: M[i,j] = E[i, j] + min(M[i - 1, j - 1], M[i - 1, j], M[i - 1, j + 1]), where M is the matrix of cumulative energies and E is the energy for any pixel located at row i and column j, and the first row of M is intialized to the first row of E. To ensure that our seam has minimal energy, we always pick the smallest energy neighbor (left, middle, or right in the row above the pixel) to add cumulatively.
After constructing our cumulative energy matrix M, we find the minimum value of our last row. This represents the ending point of the seam with the least cumulative energy. From here, we backtrack to retrieve all of the positions of the pixels in the previous rows that led us to the minimum. We keep track of this seam, and delete it from our original image.
We continue steps 1 and 2 until we have deleted a target amount of seams from our image.
To accommodate for horizontal seams, we transposed our image (effectively rotating by 90 degreees) and passed it through the same algorithm.
After some struggling comes the fun part. Below you can watch the program work. Red lines jumping in the pictures are the seams that have been deleted.
Seam Carving 1 from Lily Yang on Vimeo
We experimented with both horizontal and vertical carving. Here are some examples of vertical seam carving, where the width of the image is reduced. One super cool thing about this section is the apparent movement in space for objects. As you can see in the champange glasses, the space around each glass significantly decreased. Since we used Sobel, it makes sense that the blurred, lower frequency background was the first to go in seam carving. For the fourth scene of an autumn lake, the scene is very changed, with bushes being bent and clouds being sequeezed, but somehow it still looks like it belongs.
And now for horizontal seam carving. Here the results are just as interesting. We can see the compression in the clouds of the palm tree. I found the winter churh really interesting because even though we loose such a large portion of the top, the tree on the side still looks like a perfectly normal tree. Even for the cap of the church itself, though pressed down, appears to blend well into the scene. The glider picture displayed almost 50% compression in height. The parachute itself seems a little strange, but otherwise, most unnatural things are unnoticeable.
And now for a bit of both. Below, both vertical and horizontal carving were used. The butterflies themselves haven't changed sizes, nor has the main buds of the two flowers they are standing on. However, the distances of the surrounding buds are a lot closer. Since plants are more wobbly in shape, the way that the flowers squiggle in the center region appears to blend in pretty decently.
And now for some fails. When too many seams are taken out, it becomes obvious where the issues are. Our poor monk has lost half his body and the beach palm tree is a little skinny. Faces don't work well under pressure, either.
Since calculating the minimum energy seam requires much time, we decided to implement a way of caching previous computations. This way calculating a 400 seam deletion would not only generate the image for the 400 seam removal, but all seam removals before that. To do this, we save the image at every step during precompute, and have on-line retrieve the image corresponding to the given seam reduction. Our implementation involves creating a simple folder and file writing and reading system. This lets us retrieve our seam removals in constant time with respect to image size.
We’ve talked about deleting and cropping this entire time, so why not think of the flip side? Although we ran into similar issues as did the paper regarding choosing the same seam, we were able to fix it via changing the indices of the seams. Seam insertion is much like its carving counter part, except after identifying the minimum energy seam, we duplicate the seam instead of deleting it. When spread out through the image, the duplicated seams will blend, ;) seamlessly, with the rest of the image. Below, you can see what happens when seams are not updated, causing the algorithm to choose the same seam every time.
To fix this, we take a two step approach. First, we find all the seams to carve, then we translate the seams and duplicate. This is done via keeping track of index numbers of each pixel in a separate array, and carving this array along with the images to get each new seam.
The best way to solve a problem comes with breaking it down. Into rows. then neighbors. then itself. We also learned that you should never mix seam carving with a face. That was quite horrifying. But jokes aside, vectorizing saves lives and hours. And images with shallow depth of file takes the cake for being easiest to carve.