Final Project 1: AR

Kevin Zhang and Anna Brewer

Step 1: Label image

First we need a set of correspondences between points in our image and in the real world. We get these correspondences by drawing a grid on the box and orienting the coordinate systems using an arbitrarily chosen intersection as our origin and the sides of the cube as our axes.

Step 2: Track correspondences through frames

We used a tracker from opencv (median flow tracker) to track the points we chose in the first frame in each frame of the video. More specifically, around each point we sampled a box (whose size is an adjustable parameter) and tracked the location of that box in each frame and took the center of the box to be a good estimate of where the corresponding point was in that frame.

Step 3: Solve for camera projection matrices

Once we have our correspondences, we use them (we need at least 6) to solve for the camera projection matrix for each frame. This matrix is a 3 X 4 matrix which projects 3D homogenous coordinates to 2D homogeneous coordinates. We let the last entry of our 3D homogeneous coordinates be 1, and solve for the last coordinate for our 2D homogeneous coordinates.

Step 4: Project cube for each frame

Now that we have a camera projection matrix for each frame, we simply map the world coordinates of our cube to image coordinates. The result is below:

Final Project 2: Lightfield

Depth Refocusing

If we average all the images in the lightfield dataset, we see that the points which are far away from the camera positions are not blurry while the ones which are close are blurry. This is because features which are far away shift less than features that are close to the camera positions. Then to switch the focus onto the features which are close instead, we can shift all the images before averaging to align the pictures near to the camera and unalign the features far from the camera. This amounts to shifting the pictures away from the center to align the features closer to the camera. The results are below:

Aperture readjustment

In order to make the aperture wider or narrower, we average more or fewer images around the center. The reason this works is because the wider the aperture, the more rays we are sampling from an imaginary circle in the center of the image. The results are below: