Tony Lian's Final Project

Project 1: Lightfield Camera

Part 1: Depth Refocusing (30 pts)

Refocusing effect is achieved by changing the offset to average the image with. Taking the offset as C * (u, v), where u, v is the offset of the carema for image capture, with different C, allows different refocusing effects.

Refocusing with a different C:
(Chess)
(Bulldozer)

Part 2: Aperture Adjustment (20 pts)

Through adjusting what range of image to average, we can achieve an effect of adjusting aperature. Since the image is on a 17 x 17 grid, we have 8 different choices for averaging images. The focus is the lego ground and the men close to us.

Adjusting aperature:
(Lego)

Part 3: Summary (0 pts)

This is a very cool project which gives me more understanding about the lightfield.

Bells and Whistles: Interactive Refocusing (up to 5 pts)

The browser needs to support HTML5 for interactive refocusing (for example, use Chrome/Safari). Click on the image to refocus to where you click.

The Chess images:

If it doesn't load, you could view the video with source frames and drag the control to "focus" interactively.

Project 2: Poor Man's Augmented Reality

Part 1: Keypoints with known 3D world coords

Labeled points ((0,0,0) is the lower left point):

(Labeled points)

(Axis from the projection matrix)

Part 2: Propogating Keypoints to other Images in the Video

I use a corner detector and pass the keypoint to the consecutive frame. If there is no match in the next frame, keep the current keypoint position. I use a moving average to stabilize the points.

(Note: red is from past frame, blue is current frame. They don't differ a lot.)

Part 3: Calibrating the Camera

I recalibrate the points for every frame.

(Intermediate frame.)

(Another intermediate frame.)

Deliverable

The box AR with the corresponding input images:

Bells and Whistles

Placing an arbitrary mesh onto the scene. This is non-trivial. I loaded a model of effiel tower and put its triangular mesh into the scene. This requires much more adjustments than the box. I apply a linear adjustment to the position of the object at the end of video and smoothing of the keypoints.

The model:

You can see the height of the tower varies. This is because the tip is too far away from where I put the points. Even a slight shift of points leads to a large variation. This is more pronounced when I accelerate the video to create this GIF due to upload size limitations (you can see that the input video also "jumps" because I don't have a tripod to control the movement of camera).