Introduction

As the paper¹ by Ng et al. demonstrated, capturing multiple images over a plane orthogonal to the optical axis enables achieving complex effects using very simple operations like shifting and averaging. The goal of this project is to reproduce Depth Refocusing and Aperture Adjustment using real light field data.

A thorough set of light field data can be downloaded from The (New) Stanford Light Field Archive. These light field data are captured by The Stanford Multi-Camera Array, which is a grid of 17 by 17 front-facing cameras.


Example of a camera array

In the following image, we can see what each camera in the array is capturing in u-v graph. Whilst in s-t graph, we can see each patch of pixels in different cameras.


Example of resulting images

1 Depth Refocusing

1.1 Theory

Depth Refocusing would work is based on one very simple idea - across different camera, objects from a distance will seemingly be unmoved while the objects in the foreground will appeared to be shifted due to parallax effect. As a result, if we stack all images taken by the camera array and average them, we will get an image with blurry foreground and sharp background due to the relative shift from parallax. The reason the following averaged images' focal plane is not at the background is probably related to how are the images were re-aligned.

Three example images showing simple averaging over all the images from the array.

Now to adjust the depth of focus, we can shift all the images relative to the centre image in the array. This image shift is seemingly undoing the parallax effect and at the same time, creating the parallax at a different point. The amount of shift should be proportional to the distance of an image from the centre image. The further away an image is from the grid centre, the more we need to shift it towards the centre.

imgShifted = shift(img, (α * (r - i), α * (j - c), 0)); where
    imgShifted is the shifted image.
    r, c represent the view point; default to be grid centre (8,8).
    i, j represent the grid location of the image.
    α is the proportional constant. This determines the new focal plane.

1.2 Results

As the following images, we can see the focus shifts from foreground to the background.


Gif animated result of Chessboard


α = 2	α = 2	α = 1

α = -1.5	α = 0	α = -1

α = -4.5	α = -1	α = -4

2 Aperture Adjustment

2.1 Theory

To adjust the aperture, we just need to modify the size of window of the images we are including. Say centred at (8,8), to make the aperture wider, we can extend the window by averaging images n images away from the centre in all direction.

To illustrate, as shown in the following diagram. Suppose each polka dot is a camera. When n=0, we will include the images taken by the cameras within the red square.