Project 1: Seam Carving

I implemented seam carving as described in "Seam Carving for Content-Aware Image Resizing" by S. Avidan and A. Shamir. In order to shrink a given image, we can continuously remove seams (connected paths with low energy pixels) until our image has reached the desired size.

1. Energy Function

Before we choose seams to remove from the image, we first have to determine the importance of each pixel using an energy function. According to the paper, there's no universal energy function that works well with all images. In my implementation, I initially chose to use the e1 energy function.

2. Minimum Energy Seam

For each pixel, I used dynamic programming to find cumulative minimum energy for all possible seams to go through that pixel. The reoccurrence relationship is as below:

Next, I found the pixel in the last row of the image which has the smallest cumulative energy and set it as the endpoint of the minimum energy seam, and then backtrack from the endpoint to recover the rest of the seam. I then delete the seam from the image, and repeat this process of finding and removing the minimum energy seam until the image has shrank to the desired size.

3. Seam Carving Results

A. Stadium

B. Waterfall

C. Big Game

D. Bay

4. Comparison of Energy Functions

The e1 energy function doesn't work with all images, as you can see in the fail result images above. Therefore, I decided to compare and contrast e1 with the L2 norm of the gradient and the Harris corner energy function.

500x700 Game

600x400 Bay

Based on the results above, it seems that the Harris Corner Energy Function does better with images that contain more straight edges and or corners, while e1 and L2 norm produce similar results with all images.

Project 2: Haze Removal

In this project, I implemented Haze Removal as described in "Single Image Haze Removal Using Dark Channel Prior" by K. He, J. Sun, and X. Tang.

A haze image can be formated as the following equation:

Where I is the haze image, J is the scene radiance, A is the global atmospheric light, and t is the transmission or the unscattered light. What we want to do is to recover the scene radiance J, and to do so, we need to solve the other parameters A and t.

1. Dark Channel Prior

In order to recover the atmospheric light A, we first need to find the dark channel prior J_dark of our image, where the dark channel for a pixel x is the minimum intensity of all pixels in a 15x15 patch around x across color channels.

2. Estimating Atmospheric Light

Since the dark channel, J_dark, should be low in haze-free regions of the image, and high in haze regions and sky regions of the image, we can use J_dark to estimate the atmospheric light of the image: We first find the top 0.1% brightest pixels in the dark channel, then from these pixels, identify the ones with the highest intensity in the original hazy image I and use them as our atmospheric light.

3. Estimating Transmission

We then estimate the transmission by subtracting the dark channel prior of the normalized image from 1. We can keep a small amount of haze for distant objects by controlling the parameter w.

4. Refining Transmission

We can refine our transmission mask using soft matting techniques. The paper uses the Levin Matting Laplacian matrix to refine the estimated transmission. However, the matrix was too complicated to implement and inefficient to use. In "Guided Image Filtering", K. He suggests to use a guided filter to refine transmission instead. The guided filter approxmiates the Levin Matting Laplacian matrix and is easier to implement.

The refined transmission mask is given by

Where w is a window around the ith pixel and a and b are

5. Recovering Radiance

Finally, we can recover the scene radiance J:

Where t0 is a lower bound on the transmission.

However, scene radiance by itself looks very dim because it's not as bright as the atmospheric light, so we can increase the exposure of the radiance to make the resulting image look better.

Compare and contrast

6. Additional Results

A. Nanjing

B. Chongqing

C. Town in Haze

Reflection

From seam carving, I learned about different energy functions and ways to speed up my code in numpy. From haze removal, I learned about guided filter and followed tutorials on how to implement it.