Carving Seams


Let's Carve!

Context and Method

For this project seam carving is the act of scaling image dimensions by removal of 'seams'. By applying a funciton across the image to determine the importance of each pixel. By assigning each pixel a value we can then find a path from one side of the image to another (either vertical or horizontal) which has the least importance. In the case of a vertical seam, the path only takes a pixel from each row, whereas a horizontal seam uses one pixel from each column. By picking the correct importance function for the job we can remove pixels while keeping the objects of interest in the photo.

Original

Gradient Magnitude

Seam Finding

In our case we used the gradient magnitude of each pixel to determine its importance. This is a fairly good measure as it will give things like backgrounds low importance as they have fewer changes in general, and as the human eye essentially uses gradients to perceive vision, it will also remove what should be the least noticable parts of the image. The seam is computed using dynamic programming methods. I will use vertical seams in my example, but the same is achieved for horizontal seams by rotating the image. We use the fact that the lowest energy path for a given pixel is the lowest energy path of the adjacent pixels above it plus its own energy. So from this we can build our path by going through the image bottom up. From the second to last row each pixel's value is determined as its own value in addition to the lowest value of the adjacent pixels below it. This iterates through the image until reaching the top. From this, we can start at the lowest value on the bottom row, and then pick the minimum of the adjacent pixels above it to recover a seam, delete that seam, then repeat.

Results

Original


Result

Width reduced by 33% with vertical seam carving

Result 2

Height reduced 33% with horizontal seam carving


Original[2]


Result

Width reduced by 30% with vertical seam carving

Result 2

Height reduced 40% with horizontal seam carving


Original[3]

Result

Width reduced by 15% with vertical seam carving


Original

Result

Width reduced by 20% with vertical seam carving


Original

Result

Width reduced by 25% with vertical seam carving


Original

Result

Height reduced by 25% with horizontal seam carving

Original

Result

Width decreased by 17% with vertical seam carving

Comments

The algorithm worked very well on landscapes where the irregular edges disguised any discrepencies caused by removing seams. Later in the page I will show how it has trouble with straight edges in pictures as removing scenes will create jarrring breaks in diagonal lines causing the images to appear distorted. It also worked well on 2D images where the edges were softer. In cases of clouds and other background objects, the algorithm did well to resize them and keep their general shape when shrinking. One issue was that it would sometimes carve people and they would suddenly lose a size or two off their waist. In the case of the stairs image above it looks like the man is fairly skinny with a new haircut, and without the original image for comparison it might not seem off at first glance.

Failures

Original

Result

Width reduced by 15% with vertical seam carving


Original

Result

Width reduced by 20% with vertical seam carving


Original

Result

Width reduced by 20% with vertical seam carving


Original[4]

Result

Height reduced by 25% with horizontal seam carving

Comments

The method failed on images with many straight edges as some times seam carving will not preserve the continuity of these edges. This was the case for a lot of images with architecture as seen by the parking lot image above the cables and parking structure come out distorted. This is because the algorithm doesn't focus on the entire structure of the image, only the paths that have the least importance. Even if we were aware of this structure, if a cable stretches entirely across the iamge it would be difficult to remove seams without distorting it. Another case of failure were sillhouetes of people. Because they are generally a single color it may consider them unimportant and often made people very skinny, even more than sticks and bones! This could be solved by selecting areas to be very important first, maybe by a mask, to keep images.

Bells and Whistles

Image Enlargement

An idea offered by the research paper was that we could enlarge images by inserting seams in the order we would remove them. You can't just add the same low energy seam without it looking stretch, so by progressively adding low energy seams we can enlarge an image. Images enlarged larger than 50% had the process done twice rather than all at once, otherwise the program would begin to insert seams and enlarge the more important objects. This worked very well for pixel art where the pixel representations are 1:1 as the scene is very discrete.


Original[5]


Result

Height increased by 33% with horizontal seam insertion

Result 2

Width enlarged by 20% with vertical seam insertion


Original

Result

Height enlarged by 18% with horizontal seam insertion

Original[2]

Result

Width increased by 70% with vertical seam insertion

Mask to Remove or Keep Objects

Another idea offered was to use masks to keep or delete important parts of an image. I implemented this by drawing a mask in photoshop and then giving whites a very high or low value and finally adding it to the computed image after the importance function is run across it.

Original

Mask


Keep Target

Remove Target

Original

Mask


Keep Target

Remove Target

Original

Mask


Remove Target

Conclusions

Since last year I was super excited for this project. Mostly because I had quite a few pictures that I'd like as wallpapers but the aspect ratio was off or maybe it was a few pixels too wide. It's an awesome application of dynamic programming and it's impressive how much the simple algorithm can do. The greatest impact is how easy it is to modify images that to the human eye can look completely normal, when some parts of the image are totally fabricated. Most digital images have more than meets the eye in them. Choosing the right importance function can change drastically how the algorithm acts and means that as long as one can find the right importance function, seam carving can be used to find and insert or remove almost any scene.

Sources