CS 194-26 Final Project 1: Seam Carving

Overview

Seam carving is a content-aware image resizing technique proposed by Shai Avidan and Ariel Shamir. Using seam-carving, we can shrink images vertically and/or horizontally to a desired dimension, while attempting to preserve interesting or important features within them.

Our Algorithm

To resize an image, our algorithm works by greedily removing the lowest-energy vertical seam until the desired width is achieved, then greedily removing the lowest-energy horizontal seams until we achieve the desired height.

To find the lowest-energy vertical seam, we calculate an energy map of the image by scoring the grayscale image with the gradient squared magnitude. We then use dynamic programming to identify the seam with the lowest total energy along its path, and remove it from both the image and energy map. To remove k vertical seams (reducing the image width by k), we perform this process k times.

We only explicitly implemented the carving of vertical seams in images. To carve horzontal seams, we simply rotate the image by 90 degrees and run it through our vertical seam removal method. Before saving the image, we rotate the image 90 the other way to revert it back to its original orientation.

Resized Image Examples

The following three sections contain examples of resizing images using horizontal carving, vertical carving, and a combination of both. For horizontal carving, image width is reduced by removing vertical seams from the image. In vertical carving, image height is reduced by removing horizontal seams from the image. To reduce both width and height, both vertical and horizontal seams are removed.

Horizontal Carving Examples

Original Horizontally Carved Image

Vertical Carving Exmaples

Original Vertically Carved Image

Horizontal and Vertical Carving Examples

Original Carved

Bells and Whistles

We sped up our code significantly by making use of numba, allowing us to use just-in-time compiling (“jit”) instead of running interpreted python code. Utilizing numba gave us a 20x speed increase from our original numpy code.

Failure Cases

Because our energy function is sensitive to lots of edges/color variations in images, our code does not work well on images where the subject is in a noisy background as the algorithm chooses to crop the subject more than the background. In the examples below, the boba cup, dogs, and pumpkin are all situated in front of and around noisy objects, causing the algorithm to try and preserve the surroundings more than the intended subject of interest.

Original Resized More Resizing Even More Resizing

What We Learned

In our original attempt to speed up seam removal, we attempted to remove all requested seams at once. We quickly found that this only works for a small number of seams, as it may not be possible to find all k seams to be removed with a greedy approach as some seams may end up overlapping with others.

Overall, we found that the most important thing to note to get the best images is picking good images to seam carve. In our failure case examples, although we really wanted to see carved pictures invoving the white dog or cat-pumpkin, the subjects were simply not significant comapred to their backgrounds.

Final Project 2: Light Field Manipulation

Depth Refocusing

The first part of this project was depth refocusing. As the images were taken on a regular grid, we could choose the center most image and create offsets using its location and the location of all the other images. Once we have these offsets, to refocus is simply to weight these offsets uniformly by a factor alpha when taking the average of all the photos. This alpha refocuses the image at a different depth. As we can see in the results, this gives a pretty decent result and is fairly simple to implement.

Amethyst

Eucalyptus

Chess

Aperture Adjustment

To modify the code to be used for aperture adjustment, all we need to do is instead of weighting the offsets and averaging, we simply average with a subset of images that are within a certain radius of the center image. Below shows the results for different radii.

Radius Bean Chess
r=0
r=3
r=5
r=7

Beans

Chess

What We Learned

In this project we learned really how simple and easy it is to manipulate lightfield data to generate really cool effects. The code for this project was relatively simple, but we still got really good looking results!

Performance

We managed to get this to run pretty fast using the Python library Numba to JIT compile our Python code. This sped it up by around > 10x. We also made heavy use of the Python multiprocessing library to generate different images in the series on different cores (although this uses a lot of RAM due to the lack of shared memory support in python versions <3.8).

Fun Failures

When trying to optimize the RAM usage of our code, we played around with different numpy data types for our files. Here we learned that ints just really did not work for the aperture adjustment, but we got some trippy looking results!

Radius 0

Radius 1

Radius 3

Radius 5

Final Project 3: HDR Imaging

Method

In this project we construct HDR images using a series of LDR images at different exposures. To do this we first recover the pixel response curve of the camera sensor. This is formulated as a least squares program using a number of samples from each of the exposures, combined with number of points derived from the second derivate of the pixel values, and the log of the exposure times themselves. This gives us an estimation of the pixel response curve, which we can then use to reconstruct the radiance map of the entire scene. Then once we have this HDR radiance map, we perform local tone mapping to map as much of the dynamic range to our display device as possible.

Results

Response Curves

Short Exposure Medium Exposure Long Exposure Response Curve

HDR Images – Locally Tone Mapped

These images each had more than three LDR images that contributed to the final HDR image, but for display purposes I chose a medium exposure image out of the middle exposures to display here.

Bonsai

Short Exposure Medium Exposure Long Exposure

Chapel

Short Exposure Medium Exposure Long Exposure

Garage

Short Exposure Medium Exposure Long Exposure

House

Short Exposure Medium Exposure Long Exposure

Mug

Short Exposure Medium Exposure Long Exposure

Arch

Short Exposure Medium Exposure Long Exposure

Window

Short Exposure Medium Exposure Long Exposure

What We Learned

In this project we learned how to combine various LDR images to create HDR images. The most interesting thing was how big of an impact the local tone mapping parameters had on how good our output looked. By choosing various techniques and tuning parameters, we were able to get pretty good looking results for all of our input images. We were pretty excited with how well some of them came out (especially, chapel, house, and bonsai).