Background and Approach

In this project we used lightfield data to programatically change the focus and simulate aperture size change after the photos have been taken. See each section for details of the approach.

The Data: We use a rectified set of images of a chessboard. This set consisted of 289 images taken on a 17 x 17 grid that lies perpendicular to the optical axis along which light travels to the camera. The data can be found here.

Depth Refocusing:

We can shift and average the images together to give a result that appears to be focused at different depths. In the dataset, each image is already labeled with an offset that aligns all images such that far away objects are in focus. This is equivalent to aligning the blue dots in the pictures below. We can shift even further by multiplying each of these offsets by a fixed number, found empirically. This allows us to focus on closer objects the more we shift (equivalent to lining up the red dots in the image below).

View of two cameras in grid. Note: to align blue dots (far away objects) very little shifting is needed.

Can shift further to align for near images.

Multiply offset for all images to align for a near object and average to get the final image.

Results Refocusing on Chessboard:

Aperture Adjustment:

We can also simulate aperture size by changing how many images we average together. Images on a small grid around the center will mimic a smaller aperture as we only take into account rays from a smaller set. Images on a large grid will mimic a larger aperature. We use the center camera in the grid and continously average more cameras to simulate increasing the aperture size.

Results Adjusting Aperture on Chessboard:

Bells and Whistles

Interactive Refocusing

We created an interative refocusing program which displays an image and allows the user to select the point to focus on. We use the camera in the middle of the grid as a baseline and take in the selected point. We then search over all possible shifts of the furthest camera on the edge of the grid such that the 100 x 100 patch around the selected point best matches up with the image taken by the center camera. We determine "best" simply by using the sum of the squared difference between the patches.

Demo of Interactive Refocusing Tool: