Programming Project 5 (proj5)
CS194-26: Image Manipulation and Computational Photography

Lightfield Camera:
Depth Refocusing and Aperture Adjustment with Light Field Data
Due Date: October 31, Tuesday, 11:59PM, 2017

Overview

As this paper by Ng et al. (Ren Ng is the founder of the Lytro camera and a Professor at Berkeley!) demonstrated, capturing multiple images over a plane orthogonal to the optical axis enables achieving complex effects (see this gallery - hover around different part of images) using very simple operations like shifting and averaging. The goal of this project is to reproduce some of these effects using real lightfield data.

Details

The Stanford Light Field Archive has some sample datasets comprising of multiple images taken over a regularly spaced grid. You can use any of the available datasets for this project (use the rectified images). You are also encouraged to go over Section 4 of the paper mentioned above to truly appreciate how a very simple idea of placing multiple sensors combined with elementary operations can lead to such beautiful results.

The project has the following two parts, both of which will use the data above -

1) Depth Refocusing (30 pts):

The objects which are far away from the camera do not vary their position significantly when the camera moves around while keeping the optical axis direction unchanged. The nearby objects, on the other hand, vary their position significantly across images. Averaging all the images in the grid without any shifting will produce an image which is sharp around the far-away objects but blurry around the nearby ones. Similarly, shifting the images 'appropriately' and then averaging allows one to focus on object at different depths.

In this part of the project, you will implement this idea to generate multiple images which focus at different depths. To get the best effects, you should use all the grid images for averaging. The effects should be similar to what you observe when you change the depth of focus in the dataset's online viewing tool (example) under the 'Full Aperture' setting.

2) Aperture Adjustment (20 pts):

See this image of San Francisco which was produced by averaging multiple images collected using a satellite. Averaging a large number of images sampled over the grid perpendicular to the optical axis mimics a camera with a much larger aperture (can you think why ?). Using fewer images results in an image that mimics a smaller aperture. In this part of the project, you are required to generate images which correspond to different apertures while focusing on the same point.

3) Summary (0 pts)

Tell us what you learned about lightfields from this project !

Deliverables

• Please submit all the code you used to create your results, along with a main.m script that can be used to run your code and a README describing the contents of each code file, the bCourses course site.
• Your webpage submission should include the following :
  • Text giving a brief overview of the project, and text describing your approach.
  • gif images showing results for both parts of the project
  • Describe any bells and whistles you implemented.
  • Remember not to use any absolute links to images etc on your computer, as these will not work online. Only use relative links within your folder.

This assignment will be graded out of 50 points, as follows:

• 30 points for simulating a camera focusing at different depths using all the grid images.
• 20 points for simulating a camera of different apertures focusing on a common point.

Bells & Whistles (Extra Credit)

• (up to 5 points) : Interactive Refocusing : See this gallery for examples of interactive refocusing. For any input point, find the optimal pixel shifts that aligns all images wrt that point (using techniques similar to Project 1 but for a patch around the point instead of the whole image). Hint : Instead of searching in all images for the corresponding patch, the regular grid structure reduces the search to just one parameter across all images - if you know the shifts for one image, you can find the shifts for all images! This is the case because this parameter implicitly represents the depth of the pixel - if the pixel was far far away, the shift would be nearly zero for all images. If the pixel is closer, the shifts across the images are bigger but are tied to each other due to the grid structure.
  
• (up to 5 points) : Using Real Data : Collect your own data by clicking multiple images with a normal camera and implement refocusing / aperture adjustment. If it does not work, why do think that is the case ?
• (Final Project Idea) : Here is a cool iPhone app which samples multiple images as you move the camera around and then uses this set of images to refocus on any given point. You could implement and improve this - its basically a lightfield camera where you don't know the grid structure !

Acknowledgements

We thank Frank Dallaert for helping in adapting this assignment from his Computational Photography class