CS 194-26 Project 5
Completed by Alex Krentsel
Description
In this project, we are taking data from light field cameras are manipulating it to mimic taking images with different aperatures and different focus points. By shifting some images and keeping others in their place, then averaging them all, we can combine our set of images to simulate depth refocusing. Then by selectively keeping some images and not keeping others, then averaging, we can simulate changing the aperature width. More details provided below in the respective sections.
Depth Refocusing
We can refocus depth by using the procedure describe by Ren Ng in his slides. We shift each image by C (which is the focus factor) times the u, v positions from which the image was taken. Objects that are closer to the camera vary their position more as we move the camera than the objects that are farther away from the camera. By shifting images to align a particular point, we can simulate focusing on that point. By not shifting them at all, we end up focusing on the farther objects in the image, as they are the ones that vary the least between shots.
This is the result:
The chess image used a C value range of -0.3 to 0.7, with a step size of 0.05, and the jellybean image used a C value range of -0.7 to 0.25, with a step size of 0.05.
Aperture Adjustment
We can also adjust the aperature size by picking a point to focus on (in the demo, it's towards the back), and then averaging a particular subset of the images to simulate a particular sized aperature. We get a larger aperature by averaging more images. We pick which images to average by taking a circle in the "grid" of images that we have of a particular radius and using those images.
This is the result, using a radius ranging from 1 to 10 while keeping the focus constant:
Notice that the image gets blurrier farther away from where we focus as we increase the aperature size. This is because we are, in effect, letting in more "rays" into our virtual aperature, and consequently our image gets less clear.
Bells and Whistles
I decided to try gathering some of my own data to do the focusing with. I gathered a 3x3 grid of photos. I then cropped them to not have the random boxes that are in my apartment. Here are the first couple of them:
I then ran the same algorithms as I ran to get the above results on this set of images. For the different levels of focusing, I got this:
For the aperture changing, I got this:
The results actually aren't that bad, considering I took the images by hand with my phone. I think the reason that they don't look fantastic is that my hands moved a bit when I was taking the photos, and so the relationships aren't exactly right. I had to approximate u and v values, so those approximations are likely wrong. Also, I definitely tilted the phone a bit between photos, which I'm not supposed to do.
Summary
I learned a lot about how light fields work. It's pretty cool how much information we can encode with just some sets of images (so many different levels of focus). I also learned that it's very, very difficult to try to repurpose your phone to be a light field camera.
Jelly bean images with aperature size 1 (left), 4 (middle), and 8 (right)
