I explored light fields in this project, driven by data supplied from the Stanford Light Field archive. Each dataset consisted of a set of images, taken of the same scene but from a different viewpoint (or at least, something that yielded something equivalent to this result). Thus, the set of images represented a two-plane parameterization (4 degrees of freedom) of the light field of the scene. By selectively choosing/altering these differing viewpoint images, I was able to achieve effects such as manual refocusing or emulation of aperture changes. Having the extra information given by light field, as compared to a standard image, allows for an interesting variety of visual effects!
In this section, I used the shift-and-add algorithm as discussed by Ren Ng to manually emulate refocusing of the image. Because nearer objects move more as the camera viewpoint moves, we can simulate varied focus by shifting images based on where they were taken to emulate virtual focal planes.
I found the u, v offsets (the offsets of where the camera viewpoint is from the camera viewpoint center shot) of each image and shifted the image towards this direction, multiplied by a constant factor C. This shifting per-image emulates a virtual focal plane, such that for varying values of C the focal distance changes, and the overall image after averaging the shifted images is focused upon a different distance.
View from the center:
Averaging with a C value of 0.05 (interpolating fractional shifts):
Averaging with a C value of 0.05 (rounding fractional shifts):
Averaging with a C value of 0.25 (rounding fractional shifts):
Averaging with a C value of 0.50 (rounding fractional shifts):
Averaging with a C value of -0.05 (rounding fractional shifts):
Note that I tried two different methods to handle fractional shifts. I first tried interpolating the proper values, but as can be seen from the images the same C value of 0.05, simply rounding out the shifts obtains a result that is essentially as good. Together with the fact that interpolating was slow, I decided to not use it for the rest of the results. Note how as we expected, the C value can be tuned alter the focus distance! More positive values of C shifted the focus nearer, and negative values are also demonstrated to be permitted. There is some blurryness, particularly at the edges, due to the nature of us shifting the images. With more images, this blurryness/aliasing would be ameliorated, but for the image set it is not bad at all, and does not affect our core results.
In this section, I chose only a specific subset of the images with varying viewpoints to average. Specifically, I chose these pictures such that their camera viewpoints were within a specific zone from the center camera viewpoint. This subset selection allows us to emulate changing the aperture size, since using less images with viewpoints nearer to the center simulates blocking large-angle rays.
I tuned a parameter APERTURE_FRAC which delineated the zone of camera viewpoints that were permitted to be used in the averaging - this fraction represents the distance from the center to the furthest viewpoint from the center, in both the vertical and horizontal directions. I chose images that had viewpoints within this zone, and then averaged them without further processing.
Averaging with an APERTURE_FRAC value of 0.3:
Averaging with an APERTURE_FRAC value of 0.6:
Averaging with an APERTURE_FRAC value of 0.9:
As expected, we illuminate a clear trend where using more images ("increasing the aperture" fraction) leads to a smaller depth of field (as a larger aperture does), and vice versa. Although the core functionality is clearly demonstrated, I should make a small note about undesirable artifacts. Unlike in real life where decreasing the aperture size means increasing exposure time, decreasing the aperture fraction in our setup results in some discrete-natured artifacts (because we are using less images). This effect is most present for images with very clear, standout features; the boundaries of the legos in the wall of our image dataset demonstrate this issue most clearly. Using a more dense set of images in terms of the viewpoints would ameliorate this problem, but I have no control over the dataset, and this issue is not that troubling.
Wrestling to figure out how to achieve the said visual effects (computational refocusing and virtual aperture adjustment) made me learn a lot about light fields. At the highest level, I really started to see them as simply providing extra degrees of information that we do not normally think about. I thought about how lightfields are parameterized; for example the two-plane parameterization versus the geometric two-dimensional plane of two-dimensional ray orientations. I learned about how selecting the right rays is all that really matters when we want to emulate a certain sort of camera or effect (like selcting the small-angle rays for aperture adjustment or the spreading rays out for depth refocusing). The idea of light fields is something I had really not thought much about! I'm interested to see what better methods there are of obtaining images from different viewpoints, to parameterize these light fields.
None implemented.