Seam Carving

Overview

Seam carving is a novel approach to image resizing. Most image resizing algorithms involve loss of meaningful image data, either through the shrinking or outright removal of objects. Seam carving instead attempts to preserve the size of important objects in the image. By deleting individual lines from the image with low difference from surrounding pixels, unique parts of the image will be kept while the surrounding areas shrink.

Energy Maps

To decide which parts of the image are kept and which are removed, an energy map of an image is created. Similar to edge detection, the L2 gradient of the image is found and saved. From this energy map, the seam is calculated. To find the deletable seam, we first find the minimum possible path in the row before, and add the smallest reachable path to each pixel. Once every pixels optimal path is calculated, we take the lowest energy path and delete each pixel from it.

Below is an overlay of calculated energy paths over a picture of a forest. The brightest parts of the image are the trunks of the trees, which are uniformly colored, and low energy.

Seam Carving

Ironically, the simplest part of seam carving is the carving itself! Because the minimum path has already been calculated, we only have to delete it from the image to get our result. Additionally, our seam carving algorithm works easily for both vertical and horizontal carvings. The code naturally supports horizontal carving, and by flipping an image ninety degrees, we can use the same algorithm for horizontal carves. To carve multiple pixels, we just run the seam carving algorithm again.
Below is an animation showing subsequent seam carvings and their energy diagrams from the forest image.

Some Data

The goal of seam carving is to maintain the high energy parts of an image while removing the low energy sections. To quantify the success of my seam carving algorithm, I created graphs of average energy in the image over time. If the algorithm is working as expected, the graphs should show a gradual upward trend. Luckily, my graphs confirmed that my algorithm was removing low energy seams, with a strong upward slope from image to image.

Subsequent carving from vertical to horizontal allow us to drastically change image size, while maintaining general composition

A downside of my current energy algorithm is the low energy designation on out of focus portions of an image. In the images below, the right side of the couch is blurrier and has less frequency, causing the algorithm to delete these parts significantly more. In this image the algorithm still succeeds in preserving the cat, but causing some strange artifacts at the right edge.

Some Failure Cases

Some images do not fair well with Seam carving. Specifically, highly detailed images lose much of their fidelity when seam carving cannot find a full low energy seam to delete.

Some Reflection

This project taught me a lot about processing images and the difficulty of creating good energy functions. There is a massive range of possible image compositions, and there is no real one size fits all approach to capturing the content that matters. I was especially proud of some of the optimizations I found in this project. The usage of np.rot90() to reuse my vertical seam carving code for horizontal images felt like a smart shortcut that still accurately solved the problem. My other optimization was using np.roll() and np.amin() to quickly calculate multiple permutations of paths quickly, and find the minimum value for each one.

Image Quilting

Overview

Image quilting is a useful technique that allows for the expansion of textures and similar surfaces in images. Through taking advantage of small similarities between micro-sections of the image, large, seemingly unique textures can be created from a small sample

Random Quilting

The simplest image quilting technique is random quilting. By taking random samples of an image and tiling them together, a larger image can be created. Unfortunately, this technique is quite rough, and leads to harsh lines and contrasts in an image. From a further distance though, these effects are much more subtle.

Overlapping Patches

By selecting images that have similarities between each other, we can improve the quality of image quilts, while maintaining variety across the total resulting image. Additionally, overlapping image sections can create smoother boundaries from tile to tile. This technique leads to excellent improvements, but still falls short in causing distinct edges in sections that don't have a good matching tile.

The technique for finding overlapping patches is, by computing the squared difference of each possible tile on an overlapping portion. The map below shows the difference map created for a given sample.

Seam Finding

Like in the seam carving project, calculating low energy image seams is a good way to find parts of the image that can be manipulated and deleted. In image quilting, seam finding allows for better blending between tiles, as natural seams between images can be deleted so object edges within an image (e.g. individual rocks) are preserved between tiling, regardless of overlap.

The above images were all quilted with seams, leading to no hard edges in the results

Texture Synthesis

By weighting our image quilting with a target image, we can force patterns that resemble the target. This can even be extended to making a photo resemble an artwork.