Project 5: Lightfield Camera

CS194-26: Image Manipulation and Computational Photography

Xin Chen afh

Overview

In 2D photography, there are 3 focus-related problems. First, we need to focus before taking the shot. Often in real world situations, we have taken a photo and when we look at that photo in post processing, we may find that it's out of focus or it's focusing on the wrong subject. Second, there are trade-offs between depth of field and motion blur. When we want to take pictures of different aperture size and keep the brightness of the photo constant, the bigger the aperture, the smaller the shutter speed, the shallower the depth of field, and less likely motion blur. The third issue is the complex lens design due to optical abberations. In this project, we will tackle on the first two of these issues by implemeting depth refocusing and aperture adjustment.

Depth Refocusing

We are given a set of 17x17(286) images as our lightfield data. In this part, we will generate images that are focused at differnt depth. We will follow the steps below:

1. Find the center image. Since our images are 17x17, the center image would be (8, 8).
2. Determine $\alpha$. The smaller the $\alpha$, the closer the point of focus. The range of $\alpha$ selected is arbitary. But I found -2 to 2 with a step of 0.25 is relatively enough.
3. Compute the displacement. $[d_x, d_y] = [\alpha(x - x_{center})， \alpha(y - y_{center})]$
4. Shift the images by the displacement vector. I will use scipy.ndimage.shift to shift each of the image by ($d_x$, $d_y$, 0). I've chosen order = 0 to speed up the process and mode = reflect to avoid the black margins.
5. Find the average of all the shifted images. Lastly we will average all the shifted images to get a new image with focus at the selected part and burry at other parts of the image.

Lego

Below are images at differnt alpha values. (Left: $\alpha$ = -2; Middle: $\alpha$ = 2; Right: $\alpha$ = 3.75 )

Here our alpha changed from -2 to 4 with a step of 0.25.

Glass Ball

Aperture Adjustment

Similar to the idea of averaging images for depth refocusing, we can change the aperture of the images. In order to get images of different aperture sizes, we can change the number of images that we are averging over. For example, to get an image with a smaller aperture, we will average over less images, resulting in deeper depth of field. To get an image with a bigger aperture, we will average over more images, resulting in a shallow depth of field. The images we are using to average over is in a square grid of side length $radius \times 2 + 1$ with the center at (8, 8). The smaller the radius, the smaller the aperture.

Lego

Below are images at differnt radius. (Left: radius = 0; Middle: radius= 4; Right: radius= 7 )

Here the radius change from 0 to 7.

(Just a side note) Our result of computationally adjusting aperture does a very good job at decresing lens breathing. There seems to be no lens breathing at all and even the most advanced camera lenses cannot avoid this.

Glass Ball

Summary

It's so fascinating that from the use of lightfield camera, we can change the depth of focus and adjust aperture in post processing. This allows us to save a well-constructed image that perhaps was out of focus from being deleted. We can also change the aperture if we change our mind later and wanted to include more information in the image or exclude background from the image. Lightfield camera can also reduce the cost of spending on more expensive camera lenses with bigger aperture.