CS 194-26 Final Project

CS 194-26 Final Project

Evan McNeil and Shreyas Krishnaswamy


Overview

For our final project, we completed the Light Field Camera, Seam Carving, and Tour Into the Picture Projects.

I. Light Field Camera

Overview

In this project, we take a picture of scene from slightly different places and compile the images into a light field representation. 289 pictures of a scene are taken in a 17 x 17 grid formation. We used images of a chessboard taken from the The (New) Stanford Light Field Archive. These images contain (x, y) real-world coordinates of the images in the filenames as well as which (u, v) grid position ranging from (0, 0) to (16, 16). We take the image located at grid position (8, 8) as the center image.

Depth Refocusing

First we calculate a shift for each image by subtracting its (x, y) position from the center image's (x, y) position and multiplying by a scalar c: shift = c * [(x, y) - (x_c, y_c)] We then translate the image with its shift and then average all the images. The average will seem like a depth refocused version of the scene. A high c value will bring the foreground of the image into focus and a low c value, the background.

c = -0.2
c = -0.1
c = 0

c = 0.1
c = 0.2
c = 0.3

c = 0.4
c = 0.5
c = 0.6


Aperture Adjustment

Using the same (x, y) and (u, v) coordinates for each image as the last section, we now choose a radius r. We calculate each image's distance in the (x, y) coordinates from the center image. If the image's distance is within the radius r, we keep the image; otherwise we don't use it. We average all images within radius r, with a large r simulating a wider aperture.

r = 0
r = 20
r = 40

r = 60
r = 80


Takeaways

We enjoyed learning about lightfields and realized that the concepts can be simulated fairly simply.

II. Seam Carving

Overview

In this project, we "carve" (i.e. crop) images by removing "seams" of low energy rather than naively cutting columns or rows. We use an energy function to assign each pixel in an image an energy value. We then calculate a "seam" or a path from the top to the bottom or the left to the right of the image of lowest total enery. To calculate minimum seams we use a dynamic programming approach. For example, in the case where we find vertical seams, beginning at the top of the image, we cache the value E(i, j) = min(E(i - 1, j - 1), E(i - 1, j), E(i - 1, j + 1)) + e(i, j) where E(i, j) is the total minimum energy up to pixel (i, j), and e(i, j) is the individual energy of pixel (i, j). Our energy function was the magnitude of the x and y derivatives of our image, which we calculated by convolving our image with [-1 1] and [-1 1]^T kernels, and calculating the magnitude between the two convolutions. We then remove k seams by calculating the lowest energy seam, removing it, and then repeating k - 1 more times.

Examples

Beach Original
Vertically Carve 100 Seams
Horizontally Carve 100 Seams

Sahara Original
Vertically Carve 100 Seams
Horizontally Carve 400 Seams

Sky Original
Vertically Carve 300 Seams
Horizontally Carve 300 Seams

Grassland Original
Vertically Carve 300 Seams
Horizontally Carve 300 Seams

Docks Original
Vertically Carve 50 Seams
Horizontally Carve 100 Seams

Tundra Original
Vertically Carve 200 Seams
Horizontally Carve 150 Seams

Failures

Unfortunately, if we carve too many seams the process warps our image. A good example of this process is carving over a third of a 300 x 300 pixel image of Ben Stiller.

Ben Stiller Original
Vertically Carve 100 Seams
Horizontally Carve 200 Seams

Takeaways

We enjoyed learning about how to represent the "noticeability" of image features using the gradient, and we thought the idea of content-aware cropping was really interesting. Overall we liked getting a chance to try to edit an image based on its features rather than using the naive methods that we were used to.

III. Tour Into the Picture

Overview

Following the method outlined in Tour into the Picture by Horry et al. we created an automatic method of generating a 3D scene from a single point perspective image and some user defined points. We did the project in Unity, using only C# scripting. Thus, the code can be packaged as a Unity extension and used in any Unity project.

Results

The results came out very good. Pictures that closely followed our assumptions (single point perspective, camera plane parallel to back wall plane, no objects in the room) looked incredibly realistic.

Peacock Room Original
Peacock Room Front Homography

View 1
View 2




"Saint Jerome in His Study" (1860) Original
Saint Jerome Front Homography

View 1
View 2




Blue Room Original
Blue Room Front Homography

View 1
View 2




Tuol Sleng Genocide Museum Original
Museum Front Homography

View 1
View 2

Bells & Whistles

We used Unity to create a movie flythrough/virtual tour through the Peacock Room and the museum.

Flythrough

Walkthrough

Takeaways

The most challenging thing about this project was doing it in C# and learning a lot of Unity programming in order to apply what we learned in class to the project. The reason we chose Unity is because it made the later parts of the project easier, such as doing a flythrough, but this made the difficulty more front-loaded. In the end we think our decision was good because Unity was fun to learn and might be useful in the future.