CS 194-26 Final Project

Shide Li

Part A: Seam Carving

Overview

The first pre-canned project that I choose to work on is Seam Carving. In this project I implement a basic algorithm presented in the paper "Seam Carving for Content-Aware Image Resizing" by Shai Avidan and Ariel Shamir, which aims to resize images while preserving the important content and layout, instead of simply rescaling pixels or cropping.

The basic idea is to compute the "energy" for each pixel and repeatedly remove the seam with minimum energy until the size of the image is desired. A vertical seam would be a path of pixels from the top row of the image to the bottom row, and a horizontal seam is defined similary. For example, the following images show the minimum vertical and horizontal seam of the images found by my algorithm. To plot out the seams, I used a thicker red line, but there should be exactly one pixel in each row (or column) along the path.

Examples

Here are the actual results that I got.

Failed Cases

The following two are sort of failed cases. The first one, an image of the great wall, fails a bit because I think the vertical seams that pass though the background have high energy due to the multiple layers of clouds and mountains, so more than necessary seams through the architechture is removed. On the other hand, the second example fails because the campinile's pixels aren't too dissimilar from the cloudy sky, so energy along the campinile sometimes construct the munimum seam, leaving the campanile thinner than we would expect.

Part B: Gradient Domain Fusion

Overview

In this project, we use Poisson Blending to blend objects from images to other images. The basic idea is that we expect the relative relationship (horizontal and vertical derivatives) between pixels of within the mask to be the same, while adjusting the edges of the mask to be consistent with the region in the other image. Essentially this boils down to solving a least squares optimization problem:

Part 1: Toy Example

To start off, we start with a toy example. We denote the size as r * c. First I set up the linear system, Ax = b, where x is pixels of the mask reshaped into a vector with shape (r * c, 1), b (shape (2 * r * c, 1)) is the horizontal gradient maxtrix (dx) and the vertical gradient matrix (dy) reshaped into vectors and stacked up, and A is a matrix of shape (2 * r * c, r * c) that corresponds to computing each derivative. I also included rows in A, which corresponded to setting the first row and first column of the reconstructed image to be the same as the respective pixels in the target image.

Part 1: Poisson Blending

Next, we continue with possion blending. For each source image, we define a mask, and blend it into the target image.

First Example

Second Example

In this example, The penguin is reconstructed fine, the region around the penguin is abviously blurry. This is because of the rough background that the penguin covers. There is too much noise in the sunset sky and clouds, along with the buildings on the ground

Third Example

In this example, The whale is reconstructed fine, but for the background, the mountains in the source image make the blend rough, and since they are attached to the edges, it make the whole region bluer/greener.

Conclusion (What have I learned?)

The seam removing project was intuitive and easy to implement. Personally, I really enjoyed the dynamic programming part of the project, because I like alogrotims and appreciate the simplicity and power of dynamic programming.

The gradient domain fusion project was took me a long time to understand the concept and idea behind the algorithm, and it was very hard to implement without source code for masking and aligning for python. However, once I understood the least squares problem and implemented the algorithm (which took another long while), it was really rewarding to see the results. If I had more time, I would definitely work on improving the results. For example, use polygons for masking instead of rectangles.