Introduction

For the final project for this class I decided to explore two concepts that really fascinated me during the semester: Seam Carving and Image Miniatures. Apart from two images required for the miniature component, the images I used were all taken by me over the course of the semester. Read on for the project report and details!

Seam Carving

To implement content aware resizing I followed the approach described in this paper by Shai Avidan and Ariel Shamir. I implemented an image operator called Seam Carving that allows for content aware resizing of images. The overall idea is to repeatedly carve out a single seam (which is a continuous line of horizontal or vertical pixels) where the seam that's taken out contains the least energy out of all possible seams. To achieve this behavior, I first defined an energy function for my image based on horizontal and vertical image gradients:

Once I had my energy function, I used a dynamic programming approach to find the next seam. I iterated through every row (starting at the second row) computing the minimum possible cumulative energy M for a given seam till that points. The function took this form:
Now, to find the best seam, I just found the minimum value in the last row and backtracked. I then wrote code to remove that seam from the image (first very slowly using vstack in O(minutes) and then optimized using a mask in O(seconds)). Below are some of my best results:

Huntington Beach

The original image is a 600x600 pixel square image:
And here are the results of my algorithm with 100, 250 and 400 pixels removed from the original image:

Golden Gate at Dusk

The original image which was taken using a DSLR with long exposure after the sun set was resized to 800x530 pixels:
A visualization of the first 100 removed seam looks like this:
And after removing 100 seams, I was left with this image!
I also tried removing 250 seams, resulting in this image:

Yosemite

The original image was taken with an iPhone camera at Yosemite and was resized to 600x540 pixels
Visualizing the seam removal process and resulting images:
The first 100 seams
Photo after 100 seams removed


The first 200 seams
Photo after 200 seams removed

Keyboard

At some point while working on this project, I came to the realization that the best results must come from images that have lots of geometric patterns. So I snapped a photo of my computer keyboard on my iPhone and resized it to 600x600 pixels. I was really pleased with the results! It even kept the little dirt specs :)
These are the results for 100, 200 and 400 seams removed:

LACMA Lights

Continuing in the spirit of patterns, I dug up this photo of the LACMA Lights from when I visited LA. I resized it to be 600x600 pixels:
This is the visualization of the first 100 lowest energy seams:
These are the results for 100, 200 and 400 seams removed:

I'm really happy with how this turned out. Even with 66% of the seams removed, the entire scene structure remains intact.

Best Friends

After playing around with patterns, I wanted to try people. This photo was taken on a DSLR and cropped to 415x415 pixels
Since the size of the image is smaller, I wanted to also see what happens when a huge portion of the seams are removed (up to 75%). These are the results for 50, 100 and 300 seams removed:
I was pretty surprised that for the most part the structure of the image remains intact! Non destructive resizing really works!


When things don't quite work as intended

Despite the dazzling success of the algorithm, there were also some pretty interesting failure cases. For example, while shooting long exposure photos of the Golden Gate Bridge pictured above, I also captured some darker images of the bridge (with really long exposure and a tripod when it was already almost fully dark):
When I ran the algorithm, however, it ended up cropping out the entire second tower as can be seen here:

There are also some funky edge imperfections in the corner after the crop.
In general, the algorithm stops working well after some point (that can be seen pretty well in the photo titled Best Friends above where 75% of the seams have been removed. The legs of both the subjects in the image are completely distorted since the algorithm does not know it needs to preserve the general composition of human legs and simply looks for low energy gradients).

I think overall my learning and observation for when this algorithm worked best was in the case of patterned images where these is distinct differences between the objects in the scene (like in the LACMA Lights example). The other case where the algorithm performs really well is when there is a singular distinct object and the rest of the scene is monotonic (like the sky or the water). In both these cases, the algorithm does really well by being content aware and keeping the core structure of the desirable objects intact.

Miniatures

The second topic I wanted to explore for the final project was creating image miniatures. Read on for the description of my algorithm as well as the resulting images I came up with!

Miniatures

The objective here was to create photos that look like fake miniatures objects using real-sized objects. This idea uses the concept of selective camera focus which is also known as Tilt Shift to focus on only a particular region while blurring the rest of the image. This technique narrows the perceived depth of field in the scene and creates the illusion that the lens was actually really close to the subject, thus making it appear like a miniature. Traditionally this can only be achieved using software like Photoshop or GIMP, but this algorithm achieves similar results with only a couple data points of input from the user.

Algorithm Overview

The best results were achieved by using a Mask, so I will describe that approach here:
To begin, I transformed my images from rgb to hsv format and increased the saturation by about 10%. This step helps increase the illusion of the miniature.
Next, I collected user input about the specification of the mask edges around the image. Generally 4-5 points from the user yielded sufficient results.
With the mask set in place, I applied a Gaussian filter on every pixel outside the mask, while pixels within the mask remain unchanged.
I then went through an iterative process where I increase the size of the mask by 10% in each direction and repeated the step above with an increased sigma by convolving Gaussians. I started with sigma set to 0.5 and repeated the process 10-14 times for different sized images while incrementing sigma by 0.25 at each step.This process allows me to achieve the desired effect where the blur intensity is gradually increased around the outer most edges of the mask.

Orange Yellow Red and Blue

To start the project, I looked for some photos from the Internet that are either aerial or that capture a large scene with distinct objects. I first used this photograph of colorful buildings in Groningen, Netherlands captured by Peter Nijenhuis:

Original Photo
Miniature Computed Using a Single Focus Line


Boats by the Shore

After getting the initial algorithm to work, I realized that using a mask would work better that just a single line of focus (as suggested in the Bells and Whistles section). I added that to my algorithm and used it for the remaining images. Here, I showcase a comparison of the results using a plain line of focus vs. an irregular mask. This image was captured by drpavloff :

Original Photo
Miniature Computed Using a Single Focus Line
Mask
Miniature Computed Using the Mask to the Left


Cars and Rainbows

Now it was time to move onto images that I've taken. This was captured somewhere in the middle of California on an iPhone7 camera:


Original Photo, Mask, and Resulting Miniature


Yosemite Lake

This photo was captured on an iPhone7 by Gaylor Peak in Yosemite:


Original Photo, Mask, and Resulting Miniature


Golden Gate

This pair of photos of the Golden Gate was captured from Marin Highlands as the sun was setting using a DSLR and a tripod. I used an irregular mask to accentuate the results of the miniature effect:

Original Photo at Dusk
Original Photo after Dark
Resulting Miniature
Resulting Miniature


Mask

Bells and Whistles

In addition to using masks to create complex DOF regions which I described above, I created a stop motion animation of images of dice. The sequences of images contains 16 photos taken using a DSLR with a tripod on my kitchen table:

Samples of the Original Images
I run the algorithm on each image individually selecting a different mask and then combined the results into a single GIF:

Final Result

Final Thoughts

This has been my favorite CS class at Cal. I really enjoyed working on all of the project and getting to implement algorithms that are so visual. I really hope to be able to come back to this stuff in the future and I want to continue exploring topics in computational photography and computer vision in my career.