Lightfield Camera

CS 194-26 Project 5

cs194-26-aek

Introduction

For this project, we explore the lightfield technology described in this paper by Ng. et al. The paper demonstrates how we can achieve some complex effects such as refocusing and aperture adjustment by capturing multiple images over a plane orthogonal to the optical axis. We can process each of the sub-aperture images captured to focus at different depths in the image or vary the amount of defocus blur by changing the aperture size.

Depth Refocusing

For the first part of this project, we will implement the depth refocusing effect, which allows us to see the same scene focused at different depths. In order to achieve this effect, we implement the shift-and-add algorithm which can be summarized as follows:

  1. Compute the (u,v) coordinates for each sub-aperture image
  2. Shift each sub-aperture image by Δ(x, y) = C * (u, v)
  3. Average all the shifted images together to create the resulting refocused image.
The dataset used for this project, taken from the Stanford Light Field Archive, consists of a 17x17 grid of sub-aperture images. The u and v coordinates are embedded in each of the file names, so we don't have to manually compute them. With this information, we can shift each of the sub-aperture images to align with the center image, which I chose to be the image in the center of the grid at (8,8). Proceeding with the algorithm above will result in a refocused image. We can vary the depth of the refocusing by changing the C parameter. For the scene below, I varied it from -0.3 to 0.7.

Refocused image at c = 0.6
Refocused image at c = 0.2
Refocused image at c = 0
Refocused image at c = -0.2
Refocusing from c = -0.3 to c = 0.7

Aperture Adjustment

For the second part of this project, we reproduce the aperture adjustment effect by averaging a specific portion of the images. For a smaller aperture, we average fewer sub-aperture images, and for a larger aperture, we average more sub-aperture images.

To achieve the results below, I fixed C at 0.1 and then averaged images around the center image. For an aperture of 3, for example, we would take all the images in the 7x7 box centered on the center image. We can compute photos at different apertures and see how the depth of field varies. For smaller aperture images, the depth of field will be wider, and for larger aperture images, the depth of field will ba smaller. We can see in the images below that as the aperture increases, the depth of field decreases and the background and foreground gets blurrier.

Aperture of 1
Aperture of 4
Aperture of 5
Aperture of 7
Aperture adjustment

Summary

Overall, I learned a lot from this project and was able to see first-hand how light-field photography can help us achieve some cool effects. It also demonstrates how, in a single photograph, we can potentially capture enough information to simulate different effects that would normally take multiple photographs. Light-field technology is a powerful new photography technique, and using a few simple algorithms, we were able to produce some neat results.