Overview
Using a lightfield camera, we can capture multiple images across a plane, where the plane is orthogonal to the optical axis of the camera. We collect this grid of images as a lightfield. The data from this lightfield can then be used to compute and produce some interesting effects on our images, as seen below.
Depth Refocusing
Objects that are far away do not change much as we move across the grid (the object looks the same even when we move a little); the pixels are in pretty much the same position. However, objects that are closer change much more as we move across the grid; the pixels of the object have a larger change in position across images.
Therefore, depending on how much we shift the pixels in the images, we "align" objects at different distances. This has the effect of changing the focus of the image to different depths. With a larger shift, we can bring into focus closer objects; with a smaller shift, we bring into focus farther objects.
The images below have had the lightfield images shifted various amounts before averaging. Click to enlarge:
Far Focus
Middle Focus
Near Focus
Super-near Focus
Animation of Changing Focus
Aperature Adjustment
Averaging across more images from the lightfield creates an image that mimics a larger aperature, while using less images mimics a smaller aperature. When using more images to compute an average, the images come from a wider set, so there tends to be more variability amongst the images. In contrast, given that we are focusing on the same position, averaging across less images means less variance between the images. Thus, the images are more aligned, and our generated average is less blurry, like a photo taken with a smaller aperature.
The following images are computed averages from lightfield data, differing in the number of images used in the average. They are shown here in decreasing number of samples (click to enlarge).
Animation of above sequence
Summary
From this project, I learned that some really cool things can be done using lightfields, even with just extremely simple computations! In this way, I learned that using data is less about fancy math, and more about knowing how to properly use the math, and take full advantage of the information stored in the data.