In this project my partner and I worked with light fields. Light field cameras, also known as plenoptic cameras, are unique as it supports being able to change the focus plane of an image even after it has been captured. The light fields can be captured using different techniques such as a rotating arc of cameras, an array of cameras/camera modules, or inserting a microlens array in a camera lens. A light field is a vector function that expresses the total light flowing in every direction through every point in space.
The image dataset we used comprised of 289 rectified chess board images (on a 17x17 grid) which we were able to source from The Stanford Light Field Archive. In this project our goal was then to appropriately shift and average the images to implement depth refocusing and aperture adjustment.
An issue that emerges when capturing multiple images but keeping the optical axis direction unchanged is that objects far away from the camera do not fluctuate their position much while nearby objects do. Consequently, if we average the images we have without any adjustment the farther objects will be more in focus than the closer ones. However, through implementing methods for appropriate shifting and averaging you can view the image at various depths. So, we did exactly that.
To implement depth refocusing, we iterate through each image and find the offset between it and the center position of the image grid, which we choose to be (8,8) as the grid was 17x17. After finding the difference, we multiplied it by a scalar C and shifted the image accordingly and averaged. Experimenting with various values for C allowed us to focus on different depths of the image. This is displayed in the GIF below going from -2 to 2 with a step of 0.25, and stitched selected frames to create :
Another issue we can encounter when averaging a larger number of images over the grid perpendicular to the optical axis creates a camera with a much larger aperture. We can resolve this issue by not using as many images to reduce the aperture. Thus, we can reduce the radius of images around the center of our grid at (8,8). When experimenting with radius size, I noticed that increasing radius size increases our aperture (creating objects that are blurrier), while a smaller radius reduces the aperture size. Implementing aperture adjustment, I began by creating a reference image array, used a specified radius and averaged the images within this radius from the center.I was able to do this [u-radius:u+radius] and [v-radius:v+radius]. Below is an example of the aperture ranging from 0 to 7, thus we are able to limit the amount of images that are shifted and averaged and create the gif below:
Through this project we were able to discover and experiment with photography principles/concepts and gain a much better understanding of how light field cameras operate. It was crazy to see how through simple manipulations of shifting and averaging we were able to focus on objects of various depths and aperture. This was one of our more favorite projects that we got to work on this semester, and was truly awesome to see the complex effects we can achieve with light fields.
I used ginput to select the 20 points on my box. I used a medium sized box and wrapped it with paper, then drew dots on it. I took around an 8 second video. I labeled the first frame with the points in order, and also drew a diagram to understand the 3D world coordinates.
I used an off the shelf tracker to propogate the points from the first image to the subsequent frames. I used the tutorial for an opencv tracker (MEDIANFLOW). I created a multi tracker in opencv with one MedianFlow tracker for each point on the box (20 trackers). Then I used bounding boxes (16x16) to isolate each point and track it throughout the entire video, for each frame. Here is the resulting video:
I used cv2.calibrateCamera() to calibrate the camera, after inputting the 2D and 3D points. It returned to me the camera matrix with 11 degrees of freedom. Then I set up linear equations and used the least squares method (similar to proj4, our mosaic project) to solve this matrix.