Project 5 - Allan Zhou

Depth Refocusing

I implement depth refocusing by computing a shift for each image in the light field grid following the formula: $$ \begin{bmatrix}\Delta x \\ \Delta y\end{bmatrix} = C \begin{bmatrix}u \\ v\end{bmatrix}$$ where $(u,v)$ are the coordinates of the image in the light field grid, relative to the center of the grid. $C$ is a constant that I use to control the focus--a positive $C$ shifts images towards the center (focus closer) while a negative $C$ shifts images away from the center (focus further).

After each image has been shifted the mean is taken over the grid to produce the output. By varying $C$ in a range I produced videos of the depth refocusing effect:

Aperture Adjustment

To adjust the size of the aperture, we control the number of images we use from our grid. Specifically, I set a radius $R$ and only include images where $\sqrt{u^2 + v^2} \leq R$. A small $R$ corresponds to fewer cameras being included, and thus a smaller aperture. A bigger $R$ increases in the aperture size. By adjusting $R$ in a range, I produced videos of the aperture size changing effect:

For both cases, I used the depth refocusing code from Part I to set a focus depth that is closer to the camera.

Bells and Whistles: Interactive Refocus

I want to interactively focus around a selected point $(a,b)$. I choose an an image from the grid $I_{u,v}$ and solve the following alignment problem: $$ \text{arg}\min_{\Delta x, \Delta y} \sum\limits_{i=a-k}^{a+k}\sum\limits_{j=b-k}^{b+k} (I_{u,v}[i+\Delta x,j+\Delta y] - I_0[i,j])^2 $$ Where $I_0$ is the center image, and $k$ determines the size of the patch around the selected point that we care about aligning. I used the grid search approach from Project 1 to solve the alignment problem, with the addition of edge detection (Sobel filter) first to make the alignment work better.

Since I chose $(u, v)$, once we find $(\Delta x, \Delta y)$ we can estimate $C$ and then use that with the refocusing code I described above to focus at the described point.

I also used ginput to make a simple input for selecting the focus point. Below are videos of the interactive refocusing at work. With very simple GPU acceleration, refocusing takes about 1-2 seconds.

Summary

Certain light field operations, such as depth refocusing, can be accomplished with very simple algorithms such as shifting images and then averaging.

However, light fields are also a massive amount of data and performing operations on them can be very slow if you are not careful in the implementation.