Capturing multiple images over a plane orthogonal to the optical axis enables achieving complex effects using very simple operations like shifting and averaging. In this project, we use the real lightfield data provided by Stanford Light Field Archive and tries to achieve Depth-of-Field re-focusing and minic the effect of varying apertures.
The light-field data is captured by a array of 17 by 17 cameras mounted on the same plane. As such, each camera will capture the object from a slightly different perspective. If we just naively average the 17 * 17 images, we can see that the result is sharper in the background whereas it is very blurred in the foreground. This is because further objects have less displacement with respect to each camera than nearer objects. Therefore, to refocus on the nearer plane, we have to translate each image appropriately then average. The primary challenge is then to figure out the amount of shift (dx,dy) for each image.
To figure out the (dx,dy) for each value, we capitalize on the fact that the cameras in the grid is equidistant from each other. If we pick the center image as the reference, images from each camera has a displacement value of (u,v) where u and v are the column number and row number of that camera in the grid subtracting away the center image coordinate. Then we can had (dx,dy) = C * (u,v) where C is picked experimentally. More positive C makes the image focus furhter, more negative C makes the image focus in the foreground. From my experiment, the best range for C is [-3.5,1]
In the previous part, we average images from all cameras. By selecting less images, we can mimic the effect of varying apertures. Specifically, if we only select the center image, i.e. no averaging at all, it essentially corresponds to a pinhole camera with the ultimate sharpness. However, if you select more images around the center image and do the averaging, then we will have effectively a larger aperture.
In this project, it is really amazing to see in first hand that simple algorithm can have profound effect on light field data. I was paying attention to Lytro camera since its release and now I finaly have a grasp of its mechanism. It is also enlightening to see the effect of aperture and re-focusing with real world light field data.