Lightfield Camera

CS194-26 Fall 2018

Andrew Campbell, cs194-26-adf

Introduction

A light field describes the amount of light flowing in every direction through every point in space. The plenoptic function is a five-dimensional vector function that generates the space of all possible light rays in 3D space. An ordinary camera samples only a small part of the plenoptic function.

This project is based on the idea presented in this paper by Ng et al. By using a specialized light field camera that can capture multiple images over a plane orthogonal to the optical axis, we can capture more of the plenoptic function. This allows us to post-process the image in many complex ways. Here we demonstrate depth refocusing and aperature adjustment using data from the Stanford Light Field Archive.

Depth Refocusing

Approach

Normally, depth focusing occurs prior to taking a photograph. With light field data, we can actually refocus after the fact. In the Stanford datasets, we are given images taken in a 17x17 grid where each of the photos is shifted along the plane to capture a different perspective. By averaging the images aligned to a particular depth, we can simulate the effect of shifting focus to that particular depth.

Specifically, our algorithm is based on the shift-and-add algorithm discussed by Ng, which is as follows:

1. Calculate the coordinates of the average photo location (x, y) (such coordinates can be parsed from the filenames in the dataset).
2. For each image, compute the offset (u, v) from the central coordinates.
3. Shift each image according to C * (u, v), where C is some scaling factor.
4. Average the resulting shifted images.

By varying the value of C, we simulate a varying focal distance. A large C brings the focus to the foreground and a small C brings the focus to the background.

Results

Some results for different scaling factors on the chess dataset are shown below.

C = 0.50	C = 0.25	C = -0.10

The following is an animation as C ranges from -0.10 to 0.65:

Additional results on the tarot card dataset:

C = 0.70	C = 0.25	C = -0.50

The following is an animation as C ranges from -0.60 to 0.75:

Aperture Adjustment

Approach

The idea behind aperature adjustment is very similar to depth refocusing. A single image contains almost no depth of field effect, but by adding nearby samples of the plenoptic function, we can increase the depth of field effect and simulate an increase in aperture size.

The algorithm is a slight tweak of the depth refocusing algorithm. Instead of averaging over all images, we only include an image in the average if both (u, v) offsets are within some minimum range R. A small R will make use of a single image and thus represent a small aperature size, while a large R will average over many images and thus simulate a large aperature size.

Results

Some results for different values of R on the lego dataset are shown below:

R = 1	R = 31	R = 66

The following is an animation as R ranges from 1 to 66:

Summary / Reflection

As someone who’s not great at fiddling with camera settings or photography in general, the idea of adjusting focus/aperature after the fact is pretty exciting. While a camera like Stanford’s is not practical, the Lytro cameras developed by Ng sound very cool. Lastly, the idea of light fields is an interesting way of thinking about the visual world and provides a useful descriptive model.