Ajay Ramesh, October 27, 2018

Depth Refocusing and Aperture Adjustment with Light Field Data

Overview

In this project we used lightfield data from Stanford Light Field Archive to simulate depth of field effects. The Stanford Light Field Camera is aa 17x17 size array of cameras, or "sub apertures" which capture images of the same scene at slightly different positions and angles. By using simple matrix operations like shifting, adding, and averaging, we can produce some cool artistic effects.

1. Depth Refocusing

The key insight for depth refocusing is that objects in the background move less than objects in the foreground when the camera taking the picture moves. Keeping this in mind, we can express the translation (in pixels) between (absolute) sub-aperture centers by the vector (u, v) - (u', v'). The vector (u', v') belongs to some reference image which I have chosen to be the one at the center of the camera array. These positions are encoded in the file names of the lightfield data set. We can represent "extrapolation" and "interpolation" in 2D space by the quantity alpha((u, v) - (u', v')). If |alpha| > 1 we are extrapolating, and if |alpha| < 1 we are interpolating. So, if we translate every image by some alpha((u, v) - (u', v')) we are aligning it to some fixed (u', v') and then perturbing the alignment by alpha. In the context of this project being "in focus" means that all the images are aligned at the point in focus, and they are misaligned at points not in focus. When averaged, points that moved a lot (ones in the foreground) look blurrier than points that moved a little. alpha < 0 implies a shallow depth of field while alpha > 0 implies a deeper depth of field. This GIF illustrates a "focus sweep" of |alpha| < 0.5 over 10 frames.

2. Aperture Adjustment

Using the same trick above, we can simulate the closing and opening of an aperture by sampling images within some radius (aperture size) from the center image. Since aperture size is in units of images (or sub apertures) to sample, we used the grid positions of the images to compute their distance from the center grid position (8, 8). In this GIF, we focus on a fixed position (the crane) while varying the aperture in [4, 10]. In this example we focus on a fixed point alpha = 0.2 and vary the aperture size. As the aperture size increases, features in the periphery of the image appear blurrier, since more light is being let in but the camera fixes the focus near the center of the image.

3. Summary

I learned the basics of Light Field photography from reading sections of Ren Ng's paper. I also learned that the blurring effect observed in images taken by high quality lenses can be approximated by averaging (or other low pass filters).