This project follows the paper Seam Carving for Content-Aware Resizing by Avidan and Shamir to resize images.
To remove seams, we calculate the energy of each pixel $(u,v)$ of image $I$ as $$E(I(u,v)) =\left| \dfrac{\partial}{\partial x}I(u, v) \right| + \left|\dfrac{\partial}{\partial y} I(u, v) \right| $$ We find the lowest energy seam in the image, and remove it. This is sort of the gradient magnitude of the image, so removing the lowest energy seam removes the seam that differs the least from its neighbors, so the removal is a lot less obvious and does not significantly distort image content.
To implement this, we use a dynamic programming algorithm using the cost function $$M[i, j] = \min( M[i-1, j-1], M[i, j-1], M[i+1, j-1]$$ Of course, we can switch $i$ and $j$ depending on whether we are removing horizontally or vertically.
We select the lowest energy seam, and remove it from the photo. Then we recalculate the energies of all the pixels and repeat.
We repeat removal until we have removed enough seams.
| Howl's Moving Castle movie poster. (source) | Removal of 400 seams. | |
|
|
|
| April 12, 2020 (photo of my backyard) | Removal of 508 seams. | Removal of 206 seams. |
|
|
|
| Rajah Brooke's Birdwing(source) | Removal of 180 seams | |
|
|
|
| Doge (source) | Removal of 359 seams | |
|
|
|
| The Great Wave off Kanagawa (source) | Removal of 328 seams | |
|
|
|
|
| Broadway Tower (source) | Removal of 428 seams. | |
|
|
|
To perform insertion, as described in the paper, we first find $k$ seams for removal, and then we interpolate between the left and right of each seam and insert those pixels.
We find $k$ seams first instead of finding one seam than inserting to avoid always picking the same seam.
| Easter Gray Treefrog by David Caughey Photograpy (source) | Insert 250 Seams. |
|
|
| NIve Getaway album cover. (source) | Insert 220 seams. |
 |
|
| Birdbox. (source) | Insertion of 200 seams by inserting 100, then inserting another 100. This works a lot better because the original image probably didn't have 200 low enough energy seams, so doing 200 in one go would cause distortion. |
|
|
Images with too much texture/structure proportionate to plainer background and images with things like faces that need to have the right ratio to not look distorted tend to fail once we start removing seams.
Removing too many seams will also, regardless, cause an image to be distorted, since we run out of the lower energy seams.
| Professor Efros (source) | Removal of 283 seams. If we remove too many seams, we eventually have to start cutting into higher energy portions of the image, like the face. Additionally, faces are very depending on their ratio to look like faces, so seam removal from a face causes a very distorted face. |
|
|
My Animal Crossing New Horizons house. | Removal of 200 seams. The very structured portion, the wallpaper becoms severely distorted, again because the integrity of the image depends on the hexagons remaining polygonal. Seam removal ignores these strutures. |
 |
 |
| Avengers Infinity War movie poster. (source) | Removal of 257 seams. Letters and human proprtions distorted. |
|
|
| Insertion failure: Bob Huncback Ross (source) | Insertion of 221 seams. Simply not enough unstructured background. The same problems that cause problems for seam removal also cause problem for seam insertion. Not enough low energy seams causes us to have to add seams at high energy locations, which can really distort the image. |
|
|
| Cameran Palace from Lucario and the Mystery of Mew (source) | Insertion of 298 seams failure: Again, much too structured of an image, not enough rows of just blank / simple patterns. |
 |
|
I really enjoyed a lot about this project, especially learning how a simple algorithm can do a lot. I enjoyed reading the paper and seeing how there is a lot more that can be optimzied, such as multi-scale resizing or protecting certain areas, or using different energy functions.
