Part A

Shoot and Digitize Images

Here are some of the photos I shot

car_left car_right

hallway_left hallway_right

panera_left panera_right

Homography + Warp + Rectification

First, we compute a homography that will help us warp one image into another. To do this, we define correspondences across both images and compute the homography matrix H that will map each point to its corresponding one in the second image--this matrix is then applied to every single point in the image to give us a new warped image. We can generate some warps that can change our perspective of what we are seeing by selecting certain points that we know *should* form a rectangle from a certain point of view and warping them into a rectangle. Below, we show a front-on view of the trash cans near soda and a birds-eye view of a countertop table in Soda Hall.

center image hallway rectified

center image table rectified

Mosaics

After some translation of both images (enough to get the warped image's top leftmost corner point to (0, 0)), we overlay one image on top of the other. This created a mosaic for my images. This was done through linear blending -- I recovered the overlap region through creating a mask and then, across the overlap region, computed the pixel value as alpha * im1 + (1 - alpha) * im2, with alpha starting at 1 and linearly decreasing until we got to the right edge of the overlap regions.

car_left car_rightcar_mosaic

hallway_left hallway_righthallway

panera_left panera_right panera_right

Part B

In this section, we automate the point selection process and create panoramas automatically given two images. I will demonstrate the steps on the Soda Hallway image.

Harris Corner Points

First, we detect all Harris corners on both images; these are potential "points of interests" that could possibly be of use to us in our algorithm. The get_harris_corners function was given to us, and when I set min_distance to 20 (specifying at least twenty pixels between points), I got the following points of interest.

harris points

ANMS

Note, however, that there are far too many points and that not all of them are exactly useful. We want the "strongest" corners; that is, the points with the highest h value as returned by the get_harris_corners function. For this, I would calculate the radius of influence of each point; that is, take the h value of each point (call this f(x_i)) and calculate its "influence radius" -- that is, the distance to the closest point x_j such that f(x_i) <= threshold * f(x_j). Then, I would take the 250 points with the largest radius of influence. My threshold value was 0.9.

ANMS points

Feature matching

We've trimmed the number of points down by a bit, but there are still quite a few too many, and there are still several corners in both images that do not actually correspond to anything. To figure out features that correspond to each other, we took a 40x40 patch centered around each of the points in both the left and right images and downsampled the patch to 8x8. Following this, we flattened the patch and computed the euclidean distance/L2-norm between each patch in the left image and each patch in the right image. Then, we took the ratio of the distance between the 1NN and the 2NN of each patch; if the ratio was less than some threshold (set as 0.40), then we would keep the patch; the idea is that each patch should really have one corresponding feature in the other image, and if a feature does have a corresponding point in the other image, then the 1NN should be a lot closer than the 2NN.

feature points

RANSAC

This isn't good enough, though. We still need to trim some more points, as there are some notable outliers that do not correspond to anything; for instance, look at the leftmost corner point in the left image. To get rid of these, we will run the RANSAC algorithm, which takes the remaining points, samples 4 randomly, computes an exact homography on those 4 points, and then computes "inliers" -- for all points p, find the number of points such that |Hp - p'| < epsilon, where epsilon = 0.5. Compute the number of inliers for this iteration. Then, repeat the loop as many times as you specify (for me, it was 100 iterations) and take the maximal set of inliers. Then, recompute the homography matrix on those inliers. Then, we perform the image warp. Below, we see the RANSAC points.

ransac points

Image Warps

Here are the image warps. On the left, you see the mosaic given manual correspondences from part A. On the right, we see the automated mosaics from part B. There is not too much difference with image quality, but the difference in the images lies in the automatic points picked by RANSAC, leading to slightly different warps. They both look good, but I favor the warp from part B more; I feel that the correspondence points picked are better.

hallway warp hallway warp

panera warp panera warp

car warp car warp

And on an ending note, here is my favorite mosaic, even though I did not submit this in part A

library_left library_right library warp

Takeaways

I learned a great deal from this project about warping and blending images, as well as exactly how we can stitch together images to create one giant shot by picking the right correspondences. I also learned how difficult it can be to actually create a panorama when picking points manually -- if you don't have enough corners, things get very hard very quickly. In part B, the coolest thing I learned was how we can use corner strengths and algorithms like RANSAC to automate a very painful process of finding point correspondences and warping images.