In this project, we are able to seemingly change the focus of an image as well as the aperture by averaging a dataset of images captured from a grid of camera locations and shifting the images accordingly prior to the mean computation. This project uses a rectified chess dataset.
In order to refocus the depth of our average image, we first need to choose an image to base our camera displacement calculations from. For simplicity, we choose the center camera. We then scale these displacements by a value alpha and adjust each image with the corresponding scaled translation. By varying alpha, we're able to change the focus point by taking advantage of the fact that an object that moves a lot across all images will end up blurry and looking out of focus, while an object that doesn't move a lot will end up looking sharp.
In the results below, I vary alpha between -0.2 and 0.6 (starting at -0.2, progressing to 0.6, and bouncing back to -0.2).
We adjust the "aperture" of our image by adjusting the range of images that we choose to include in our average, with a larger aperture corresponding to a larger selection radius. I rely on the same center camera as mentioned in part 1, and choose the range of images to be a [2r + 1, 2r + 1] square centered at the center camera. Below, I vary r between 0 and 8 to capture the range between using 1 camera and using all cameras.
In order to ensure the quality of my data, I drew a grid of points on a sheet of paper and took pictures with my phone of my microphone. This worked great for me, as I had little to worry about in terms of camera rotation.
The coolest thing that I learned from this assignment is that we can use multiple, properly spaced cameras to produce an image that looks like it was taken from one camera!
In this project, I made a small paper box with points marked on it, recorded a video of the box, and used augmented reality to make objects appear as though they were actually physically there.
Here's the video of the box that I made.
For these parts, I chose some of the points on the box by hand from the first frame of the video. I chose only to mark keypoints that appear throughout the entire video, though my code is prepared to handle keypoints that will occasionally disappear. I then used CSRT trackers for each of the selected points, with a bounding box size of 21 pixels for each point. This large bounding box size ensured that points never got lost. To propogate the keypoints, we iterate over the frames of the video and update our tracker based on these new frames, one at a time. The results are as shown below.
In each frame, to calibrate the camera, we used least squares to find the matrix that allows us to project world coordinates into image coordinates. The world coordinates always stayed the same, but the image coordinates varied based on the locations of the centers of the bounding boxes. Here's a video with the X, Y, and negative Z axes shown.
In order to project a cube onto the box, I adapted the draw() function used here . This function relies on both the current frame, and the current camera projection matrix. I had to change the function slightly to move the cube on top of the box, since having the cube inside of the box wouldn't have made sense.
The coolest thing that I learned in this project was how to actually solve for the camera matrix using least squares.