Project 5

Lightfield Camera: Depth Refocusing and Aperture Adjustment with Light Field Data


Overview

The goal of this project is to reproduce the effects of image refocusing at various image depths, and readjusting the image aperture using the rectified and grid aligned light field images taken from the Stanford LightField database. The light field is a 5-D data structure storing images at every sub-aperture at different location in an 2D array of sub-apertures.

Part 1: Depth Refocusing

The objects which are far away from the camera do not vary their position significantly when the camera moves around while keeping the optical axis direction unchanged. The nearby objects, on the other hand, vary their position significantly across images. Averaging all the images in the grid without any shifting will produce an image which is sharp around the far-away objects but blurry around the nearby ones. Similarly, shifting the images 'appropriately' and then averaging allows one to focus on object at different depths. In this part of the project, I implemented this idea to generate multiple images which focus at different depths. To get the best effects, I used all the grid images for averaging. Using the sub-aperture (8, 8) as the center since we have 17 by 17 sub-apertures, for a general sub-aperture located at (x, y), we can denote its shift as: shift_amount = alpha [(x, y) - (8, 8)], for the constant alpha. After applying this transformation for all the images, the refocused image is found by computing the average image. In the case that alpha is greatly positive, there is a depth refocusing further from physical focus, while in the case that alpha is greatly negative, there is a depth refocusing closer towards the physical focus. Below are the static images produced by various stepsize values of alpha as indicated:

Depth Refocusing Image 1: Chessboard, alpha: [0, 3]
Snow
alpha = 0.0
Snow
alpha = 1.0
Snow
alpha = 2.0
Snow
alpha = 3.0


Chessboard Refocus Animated:

Snow



Depth Refocusing Image 2: Amethyst, alpha: [-1, 1]

Snow
alpha = -1.0
Snow
alpha = 0.0
Snow
alpha = 1.0


Amethyst Refocus Animated:

Snow



Part 2: Aperture Adjustment

In this part of the project, I generated images which correspond to different apertures while focusing on the same point. More specifically, to change the image aperture, only some apertures around the image center is used instead the aperture of all the images. The parameter radius that we specify determines this boundary of which sub-sample of apertures we use. An image aperture at (x, y) is used if it satisfies: x such that x: [8 - radius, 8 + radius] and y such that y: [8 - radius, 8 + radius]

Aperture Adjustment Image 1: Chessboard, radius: [0, 8]

Snow
radius = 0.0
Snow
radius = 1.0
Snow
radius = 2.0
Snow
radius = 3.0
Snow
radius = 4.0
Snow
radius = 5.0
Snow
radius = 6.0
Snow
radius = 7.0
Snow
radius = 8.0


Chessboard Aperture Adjustment Animated:

Snow



Aperture Adjustment Image 2: Amethyst, radius: [1.5, 2.5] & [3.5, 5]

Snow
radius = 1.5
Snow
radius = 2.0
Snow
radius = 2.5
Snow
radius = 3.5
Snow
radius = 4.0
Snow
radius = 4.5
Snow
radius = 5.0


Amethyst Aperture Adjustment Animated:

Snow



Summary

I now have a better understanding of the effects that aperture adjustment and image refocusing produces on the image, and how cinematic photography & professional photography use these features to produce blurring background effects (aperture adjustment), and different image perspectives (refocusing different parts of the image). Furthermore, this is a powerful technique that can be used to edit photos after the fact.