Project 5: Lightfield Camera

Part 0: Overview

This project analyzes collections of images of the same object taken by cameras at positions in a 17x17 grid. We will explore different ways of combining these images to create different editing effects such as depth refocusing and changing aperture. In this project we will use lightfield image sets from the Stanford Lightfield Archive.

Main Algorithm: Shift-and-Add

For both parts of this project I use a generalized shift-and-add algorithm which takes as an input a set of lightfield images, the relative camera position and a shift magnitude. For each image we are able to calculate a vector that represents the distance from this image position to the center image. Assume current image grid position is (x, y) and center image grid position is (cx, cy). We then multiply this distance vector (cx - x, y - cy) by alpha to calculate how much we should shift this image by in both x and y directions. After applying such a transformation to each image we output the average between all shifted images.

Part 1: Depth Refocusing

Overview: in this part of the project we play around with the idea of focusing on different parts of the image. If we simply average together all original images we will observe a result which is focused only on the far away objects. This is because those are naturally closer together for any image position that the camera takes the photo from. We apply the shift-and-add algorithm with an adjustable alpha parameter where positive and large alpha values correspond to focusing on a plane closer to the viewer and negative values of alpha correspond to moving the focus away from the viewer.

Here are some examples of images with adjusted shift parameter.

alpha (shift magnitude) = -1	alpha (shift magnitude) = 0	alpha (shift magnitude) = 1
alpha (shift magnitude) = 2	alpha (shift magnitude) = 3	alpha (shift magnitude) = 4

Here is are some gifs of a sequence of images. The first two are shifted by alpha in a range from -1 to 4 with 0.2 increments. The last one is shifted by alpha in a range from -5 to 1.6 with 0.2 increments.

Part 2: Aperture Adjustment

Overview: in depth refocusing by default all images are shifted and averaged. This corresponds to the maximum aperture and the minimum attainable depth-of-filed with this collection of images. Therefore only small portion of the image is in focus. To adjust the part of the image that is in focus we want to limit the aperture. This is done by limiting the number of images we choose to shift and average together. We pick images that are within a certain distance of the center (8, 8) image on the grid. We introduce new variables radius_x and radius_y that allows us to pick the vertically and horizontally limited subset of images. If we set both of these equal to each other, we achieve a radially adjusted aperture. If we want to achieve directional blurring, we can set these variables unequal to each other.

Here are some examples of images with adjusted aperture. For the flower collection I chose to set the depth focus that corresponds to alpha = 1.2. For the chess collection I chose to set the depth focus that corresponds to alpha = 1.6 to focus on the center of the image and observe aperture adjustment better.

C(focus depth) = 2, radius_x = 2, radius_y = 2	C(focus depth) = 2, radius_x = 4, radius_y = 4
C(focus depth) = 2, radius_x = 2, radius_y = 7	C(focus depth) = 2, radius_x = 7, radius_y = 2
C(focus depth) = 2, radius_x = 2, radius_y = 2	C(focus depth) = 2, radius_x = 4, radius_y = 4
C(focus depth) = 2, radius_x = 2, radius_y = 7	C(focus depth) = 2, radius_x = 7, radius_y = 2

Here is are some gifs of the crystal sequence of images shifted by a set alpha = 1. The first gif is a radial decrease in aperture. We start with a very high aperture and an extremely focused image and we narrow the radius all the way down to only include the center image. The second gif starts off horizontally blurred and increases its aperture in the x direction. The third gif starts off vertically blurred and increases its aperture in the y direction.

Part 3: Summary

By capturing a high-dimensional image dataset, photographic techniques such as depth refocusing and aperture adjustment can be added in post-processing rather than captured in an image directly. This project helped me imagine more of what the plenoptic function would allow us to achieve.