Augmented Reality and Seam Carving

CS 194: Computational Photography, Spring 2020

Final Projects

Abby Cohn

"Poor Man's" Augmented Reality

Overview

In this project, I used video tracking, 2D image coordinates, 3D world coordinates, and camera matrices to project a 3D cube into a video. This resulted in a “virtual reality”-like scene in just a few steps.

Tracking

To begin, I filmed a video that I eventually used for the projection and tracking. I found a small white box, on which I mapped out 1-inch squares with clear points on each corner. Next, I decided to manually select key-points and use an off the shelf tracker. Using plt.ginput, I selected my points and labeled the corresponding 3D points accordingly, using the bottom corner of my box as the origin. I played around with the MedianFlow tracker and utilized the OpenCV MultiTracker to create and store separate trackers for each point. I used 4x4 patches around each point for the bounding boxes. Most of the points were tracked successfully throughout the whole video, while a couple were lost. My tracking results are shown below.

Camera Calibration

Next, I calibrated the camera. I created my camera matrix for each set of 2D and 3D points by using a “focal length” constant and homogenous coordinates. I repeated this step for all frames of the video and saved each computed matrix.

Cube Projection

Finally, to project the cube into the scene, I defined the axes for the 8 corners of the cube by 3D coordinates: [[0,0,2], [0,1,2], [1,1,2], [1,0,2], [0,0,1],[0,1,1],[1,1,1],[1,0,1]]. Using the projected imgpts, I used the cv2 draw cube function from the cv2 pose estimation and camera calibration tutorial.

Putting it all together, I used the camera matrix calculated for each frame inside my tracking loop and projected the cube into every frame. This resulted in a video with the cube tracked throughout!

Seam Carving

Overview

Seam carving is content-aware, meaning we can resize images without losing pixels and information that are determined to be “more important”. The general steps for a successful seam carving implementation are determining the energy of each pixel in an image, using dynamic programming to find the minimum seam, and finally removing the seam.

Energy Function

For my energy function, I used the gradient magnitude with finite difference operators, dx and dy. In order to get stronger edges, I ended up leaving the threshold pretty low. Below are the results of the energy function on a few images.

Finding The Minimum Seam

Next, I implemented the bulk of the seam carving algorithm by finding the minimum vertical seam. I began by starting at the second row and added up the total energy for all possible seams. Next, I “backtracked” by starting at the minimum value of the last row, which indicates the end of the minimum seam, and tracing the entire path back up. Some minimum seams are visualized below.

Seam Removal

After finding the seam, I focused on the seam removal. I targeted the pixel to be removed in each row and constructed a reshaped image. I iteratively removed seams until the new dimension was reached. I also implemented a function to remove seams horizontally by rotating the image and running the vertical seam removal code. Some carving successes and failures are shown below. In some images with lots of sky, the horizontal carving resulted in large bits of sky getting removed. I think an improved energy function could help fix this issue a bit.

Successes

Failures

Bells & Whistles: Seam Insertion

I also implemented seam insertion functionality. I used my minimum seam detection code to create a list of enough minimum seams for the desired dimension. Next, I took the average of neighboring pixels in each row for every minimum seam pixel of interest and inserted these new averaged pixels. My seam insertion results are shown below.

The most important thing I learned was that seam carving requires a strong energy function, and my code may have performed better with a different energy function. Seam carving also works pretty well on landscapes, but gets a bit interesting when people are included in the photos!