Homework 5: Lightfield Camera

I used three different lightfield images for my testing. I used the chessboard, the flower vase, and the amethyst. All three of these image sets were pulled from the Stanford Light Field Archive. The data is annotated based on the file name. There are four values, delimited by underscores. The first is the camera row, the second the camera column, and the third and fourth being the u and v coordinates, respectively. I parse these values when loading the image to use for calculations.

Depth refocusing is a relatively simple problem, as we simply need to shift each image by some fraction of their offset from the center of the image. These offsets are determined by the u and v coordinates of each image. The first calculation I make is to compute the average u and v values, which will be the values I shift each image towards. Then, for every image in the image set, I take the difference between it's respective u and v coordinates and the center's u and v coordinates. I then scale these differences by some constant alpha, which determines the depth that the image is refocused on. Every image needed different alpha values to bring different parts of the image into focus, but I found that almost all images fell into the range of -0.4 to 0.4 alpha values to get good results. After this I simply average the shifted images to obtain the final refocused images. As an implementation note, I implemented my function to load only a single image into memory at a given time, as the datasets are quite large (both a large amount of images and a large resolution), perform its calculations, and then remove the image from memory before loading the next image. This allowed my function to run in usually under a minute for refocusing an image. Below are the results on all three lightfield images.

To create the effect of aperture adjustment, I modified my depth refocusing to also take in an aperture size parameter. Then, when iterating through the images to add them to the final output image, I only add them if they are within the aperture size (i.e. I compute the difference between the camera coordinates and the center camera coordinate, and if it falls within the aperture size given I add it, otherwise I do not). I had two definitions for picking images within the aperture. The first is a square aperture, where I calculate the offset in both the x and y directions, and if both of them are less than the aperture radius then I add them to the output image. The second is a diamond aperture, where I take the sum of the absolute value of both offsets, and if the sum is less than the aperture radius (which is a larger value in this case to account for using the sum of the offsets as opposed to individually) I add it to the outut image. I computed just the square aperture for the chess and amethyst images, but I computed both the square and diamond apertures for the flowers to compare the two. The bottom row is the diamond aperture, but frankly I don't see much difference in the two methods. Additionally, I chose to depth refocus all of the images with an arbitrary alpha value that I think makes the picture look most interesting. The aperture sizes increase from left to right.

This assignment was actually really interesting to revisit, coming from cs184 last Spring. When I took that class, the lightfield camera section was actually my least favorite section, which is a little surprising since that was Ren's own research. However, when we revisited it in this class it felt a lot more intuitive to me and less centered around the math. In that way, I feel that I learned a bit more about the aesthetic benefits of lightfields. Although I still don't feel there are that many practical benefits to capturing this data, it feels like an area where very interesting artistic pieces could be made. I could envision really cool time lapses using depth refocusing or aperture changes throughout the video which would be much harder to capture in real time! I didn't have much difficulty writing the code to deal with lightfields for this project, so there wasn't too much to learn as far as the actual lightfield processing was concerned. Really fun project though!