CS 194-26: Image Manipulation and Computational Photography

Project 5: Lightfield Camera

Ryan El Khoury, CS196-26-aah

Overview

2D photographs are a measure of the light entering in at a fixed direction and accumulating at each point on the camera's photosensor. The direction these light rays enter is a function of the camera's position in the world space and the way the camera's main lens redirects these rays towards the sensor. The nature of 2D photographs means that the position, focus and depth of the image are permanently assigned once the photo is captured.

Lightfield cameras reject the notion of the static image by capturing the 4D lightfield information present once the photo is captured. The lightfield captures additional information about the directional lighting distribution at each location on the sensor. This can be done with an array of cameras, all of which are focused on the same subject but in slightly different positions, or with a microlens array (which extracts more information from the light rays entering in a camera by using them to inform additional "microimages"). This project works with data from the Stanford multi-camera array which employs the former technique.

Even though the subjects of each image taken by the cameras in the camera array are the same, they were constructed from different light distributions due to the difference in the camera's positions. Therefore, each image represents a sampling of the light field present when the scene was captured. The lighting info used to construct these array of images can be resorted to create new images that vary in focal length, aperture or even location taken.

Part 1: Depth Refocusing

Depth in an image is a result of a camera's focal length and aperture. In this exercise, we will be computationally adjusting the focal length to generate a series of images that focuses at different depths in the scene.

The Stanford camera array is a 17 x 17 array of cameras. This means there is a total of 289 images for each light field photograph. The center image was captured by the camera at position (8, 8). We will use this camera and its focal length as a baseline. For each image taken by the other cameras, we will calculate the offset of the center of its image from the center of our baseline image. We will then scale this offset by a constant value and shift the image by this shifted offset. These images will be summed together and divided by the total count. The result of performing this operation on the entire image array is an alignment of the focal point in the image, where the location of that focal point is determined by the constant value.

Chess board - image taken by camera at index (8, 8).
Chess board - constant value set to -0.1. The focus is near the back of the scene.
Chess board - constant value set to 0.2. The focus is near the middle of the scene.
Chess board - constant value set to 0.5. The focus is near the front of the scene.
Chess board - constant values in range [-0.2, 0.7] with step size 0.1.
Eucalyptus - constant values in range [-0.2, 0.7] with step size 0.1.

Part 2: Aperture Adjustment

Aperture in a camera is a measure of how much light enters through the lens. Aperture is controlled with a circular diaphragm that can be expanded or contracted to block out light. A large aperture means that a lot of light is allowed to pass through to the sensor. The higher amount of light information that passes in results in a brighter image and a shallower depth of field (the size of the focal plane is reduced and objects outside it will appear blurry). A small aperture will make the image darker but expand the depth of field.

Aperture can be programmatically adjusted in a light field image by finding all images contained within a certain radius from the center image and finding the mean image from that. Radius is measured in distance from the center image. A radius of size 9 will include the entire collection of 17 x 17 (289) images. A radius of size 3 will include 5 x 5 (25) images, clustered around the center image at location (8, 8). A radius of 1 will only include the center image. The procedure is similar to how the aperture is adjusted in a camera. By excluding the outer ring of images, we are rejecting the light information that was used to form those images. The less images we take in, the larger the depth of field will be in the resulting image.

Chess board - radius of 1, or the center image. The entire scene is in focus.
Chess board - radius of 3. Since the focus of the image is near the back of the board, the front becomes blurry as our aperture size increases.
Chess board - radius of 5. The depth of field is becoming shallower.
Chess board - radius of 7. The depth of field is now shallow enough that the very back of the scene is completely out of focus.
Chess board - radius in range [1, 9] with step size 1.
Eucalyptus - radius in range [1, 9] with step size 1.

Summary

The applications of light field photography led me to believe the process was more complex than they actually were. Capturing the directions of the light rays that enter an image is simply accomplished by taking an array of photos from slightly different positions. Redistributing this light information was a matter of shifting other images to match a singular focal point and finding the mean from the images selected. Working with a set of images really helped to visualize what information is being taken from which photo and how that is incorporated to form a final image.

Light field cameras offer exciting possibilties, not just in photography but video applications as well. I foresee light field technology being used extensively for VR. The ability to adjust focus in an image coupled with eye tracking could enable the easy capture of immersive VR video. The many perspectives that form a light field image lends an opportunity to personalize/democratize the static nature of photography.