Part 1: Homography

Recitfying images

For this project, rectification means warping a 4 sided polygon into a rectangle, or mimicking a straight-on point of view. I chose this picture of Jacobs hall because it has strong perspective lines and clear facades. Looking southwest from the corner of Ridge Road and Le Roy, we have a diagonal point of view.

I wanted to see what Jacobs would look like head on, so I took the four corners from its left facade and computed a homography matrix. I used this non-affine transformation to warps the image and rectify its perspective. Now it looks like we are staring straight down ridge road, facing west. Note that Etcheverry looks smaller and further away.

Another Example

Here's a picture from inside an art gallery.

I rectified two of the paintings on the wall.

Stitching Mosaics

For the image mosaic, I took two pictures of my parents' living room.

I manually defined 6 correspondance points, including the four corners of the TV. The homography matrix was computed using least squares this time.

I warped the right image to the left image. This is what my first attempt looked like. Note that the overlapping portion is brighter because I simply added the two images pixel intensities.

To fix the brightness issue, I created a mask for the center of the image, and averaged its pixel intensities. There are still some edge artifacts, which can be fixed using a gradient mask.

Cropped panorama

More mosaic examples can be found below, in the comparisons section.

Part 2: Autostitching

Now we will try stitching mosaics without manually defining correspondence points. To do this, I roughly followed the steps outlined in this paper.

Harris Points

The first step is the Harris interest point detector. Harris corners are shown as white dots below.

ANMS

We need to cut down the number of points, and the method I used is Adaptive Non-Maximal Suppression, or ANMS. In short, the algorithm suppresses points that are lower in strength and close to other, stronger points. Points are ordered by the minimum suppression radius r_i, given by:

I sorted the points by decreasing r_i, and chose the first 100. The result of ANMS can be seen below.

Feature Extraction

After performing ANMS on the source images, I extracted 8x8 features for each of the points on both images. These were taken from a 40x40 pixel window, and downsampling to get a blurry 8x8 patch. Here is an example 40x40 window, which corresponds to the upper-left corner of the blue and white picture frame in the above image.

Feature Matching

The next step is matching features across the two images. The method is brute force nearest-neighbor search. I used Lowe's trick, which discards the match if the ratio between the first nearest-neighbor and the second nearest neighbor is greater than some threshold (0.2). The matching feature points are shown below.

Random Sample Consensus

Once the features were matched, their are still some outlier pairs to get rid of. We will use four-point random sample consensus (RANSAC) to get the ideal set of inliers for homography. The RANSAC algorithm involves repeatedly choosing 4 random points, computing a homography, and checking the accuracy of the homography against the entire set of correspondence points. After many iterations, we choose the 4 points that produced the largest set of "inliers", and compute a least-square homography on those inliers. I used an epsilon of 2 pixels when checking accuracy of the 4 point homographies against the matching paired coordinates. Here is the final result of RANSAC:

Using these as my correspondance points, I re-calculated the homography and stiched together the mosaic image.

Cropped panorama

Comparisons: Manual vs. Auto Stitching

Now it's time to compare results between manual correspondence and auto stitching.

Palm Trees

Manual

Auto

Pretty similar results here, but Manual produced sharper tree trunks.

Girl and ghost

Manual

Auto

Auto does better, if we look closely at the metal chair's grills in the back.

What's for dinner?

Manual

Auto

Neither are great here, but Auto better aligned the bowls on the table. The wine bottle, chair, and meat (middle right area of image) are very out of focus. This might be because this pictures were taken at close proximity, and my camera center shifted instead of only rotating.

Final thoughts

This was a fun project! I learned that it is important to consider the w coordinate when calculating homographies. This caused a bug in my program which took me a long time to fix. It was cool to see RANSAC, such a simple but effective algorithm, in action.