Lightfield Camera: Depth Refocusing and Aperture Adjustment with Light Field Data

Phillip Kuznetsov

cs194-26-aea

Lightfield images are a variation of traditional images that attempt to capture more features from the underlying Plenoptic Function. A lightfield basically captures information not only about the landing point of an array (aka the value at a pixel) but also the angle of the ray. This allows one to use the lightfield to dynamically create photos that can be refocused, move the viewpoint, and change the size of the aperture all after the images have been captured. Lightfield images can be created by taking mutliple images at slight spatial shifts. Our particular images were taken by The Stanford Multi-Camera Array pictured below. The camera array took 289 images simulataneously in a 17x17 array.
The Stanford Multi-Camera Array
Because of the spatial orientation, each camera captures a slightly different perspective of the object being captured. This difference in orientation provides enough information to then perform the cool post-processing stuff that come with lightfield captures. Below we can see the different perspectives when we look through the pixels that originate just from a single point in the lightfield with respect to each of the cameras.
Sampling the same points seen by different cameras
Sampling the images produced by the different cameras

Depth Refocusing

This technique is fairly simple. Basically, the objective is to change the focal point of the resulting image. All one needs to do is shift the images taken at the edges of the array closer to the center of the array to increase focus on closer objects.
Refocusing on an image of an amethyst taken by a lightfield camera.

Aperture Adjustment

Aperture adjustment is also relatively simple to do with a lightfield image. Instead of taking the average of all the images captured by the lightfield, instead we select a subset within a certain range. This simulates the closing of an aperture. By decreasing the area of images chosen from the camera array, we can simulate the change of a full aperture slowly shrinking to become a pinhole by gradually using less and less images for the averaging operation until we are left with only one.
Changing the simulated aperture from fully open to pinhole size.

Perspective Adjustment

The simplest rendering we can make with a lightfield is a perspective adjustment. By just keeping the aperture size at pinhole level (ie only use one image from the array), we can simulate movement around the field of view to create a spatial effect in an image. With proper software alignment, one could use this technique to follow the perspective of a viewer and simulate spatial depth.
Changing the perspective alignment of the image.