Project 5: Lightfield Camera

CS 194-26: Image Manipulation and Computational Photography, Fall 2018
Jonathan Fung, CS194-26-abs

In this project, we aimed to reproduce light field cameras' depth refocusing and aperture adjustment effects detailed in Professor Ren Ng's paper. Here, we use Stanford's Light Field Dataset, which are pictures obtained by taking pictures of an object with a 17x17 array of cameras. Each one takes pictures at a slightly different position, which allows us to recombine them with a series of shifts and averages to produce photos with refocused effects.


Part 1. Depth Refocusing

With this 17x17 array of light field data, we can refocus on any plane in an image after taking the picture. Each of the 289 photos can be shifted by a certain amount, and then averaged out in order to "focus" the picture on one plane at once. The algorithm is: 1) calculate average photo location (mean_x, mean_y), and 2) shift each photo by c * ((x - mean_x), (y - mean_y)), and average all the shifted photos together. By ranging the scaling factor c from a range such as [-0.5, 0.5], we can simulate a range of depth focuses. If c = 0, the image would have depth focus at the very back of the photo, as the further the objects get from the light field array, the less movement it has between images. A high C value would focus on the front, because it would align elements at the front of the array.

Focusing on back row, c = -0.1
Focusing on middle row, c = 0.28
Focusing on front row, c = 0.56
Gif cycling through different depths

Focusing on back wall, c = -0.5
Focusing on middle lego man, c = 0.0
Focusing on front row of studs, c = 0.5
Gif cycling through different depths, c = [-0.5, 0.3]


Part 2. Aperture Adjustment

For the second part of the project, we can simulate taking a picture with different aperture. The less images we blend together, the sharper the image is, which simulates a smaller aperture. The more images we blend together, it appears to be a larger aperture. Here, we average together photos in a square radius around a center point, the photo at the 8,8 location in the light field array. For example, for N = 1, we average 9 images, and for N = 2, we average 25 images. This aperture technique works because a smaller amount of images means there is less light rays to place in the photograph, like a small aperture. A large amount of images simulates more light rays.

Chess cycling through N = [0,8]
Lego cycling through N = [0,8]

Part 3. Summary

I learned a lot about lightfield cameras in this image! There's a lot of power to be found in a post-processing technique that allows you to adjust focus, depth and aperture of an image. I know that some smartphones and apps use this technology to create new photos after the photo was taken! A lightfield as big as Stanford's (289 cameras) may not be too practical, but lightfields can be applied to images taken from space and different angles, such as the overhead New York photo!