Seam Carving

Zhen Qin

Introduction

Seam carving is an algorithm for content-aware image resizing. A seam is an optimal 8-connected path of pixels on a single image from top to bottom for vertical seam, or from left to right for horizontal seam. A seam's optimality is defined by an image energy function. By repeatedly carving out the seams with minimum energy in one direction we can change the rise of an image in one direction. By applying these operators in both directions we can retarget the image to a new size.

Implementation

In the first step, I calculate the energy map, m*n matrix E[i,j] for an input color image with dimension m*n*3. I find the energy map for each R, G, B color layer separately and average them. To calculate the energy map for each layer, I took its x, y gradient through convolving the image with [[-1, 1]] and [[-1], [1]] filters. Then calculate the energy E[i, j] = sqrt(dx**2 + dy**2), for each pixel.

For vertical seam carving,with the energy map, E[i,j], I applied dynamic programming to find the minimum vertical seam to remove. M[i,j] represents minimum cumulative energy of all possible vertical seams that starts row 0 and ends at pixel (i,j). I use the recursive relation: M[i, j] = E[i, j] + min{M[i-1, j-1], M[i-1, j], M[i+1, j]} to find M[i,j]. Finally, I find the minimum M[i,j] on last row and backtrack t find the minimum seam. Remove the seam and repeat until it reaches the desired size. Here is an example of energy map, the minimum vertical seam and the result 20% vertical shrink image:

For horizontal seam carving,I transpose the image, apply the vertical seam carving implemented above, and transpose the image back. Here is a horizontal seam carving result that shrink the same image horizontally by 80%.

Successful results:

Vertical Seam Carving

Horizontal Seam Carving

Failure example:

Since the energy is defined by gradient, the seam with minimum accumulate gradient is always removed. In the following example, all the peoples are in the shadow, which is completely black, with small gradients. However, the pixels in background are changing with lights varying, so the background pixels have higher gradient. Therefore, seams from people’s shadow is picked first and the people’s shape are easily distorted.

Reference

HTML/CSS for this page was taken from https://v4-alpha.getbootstrap.com/components/card/