Project 6 Part 2: Feature Matching for Autostitching

The goal of this part of project 6 is to automatically detect matching points in two images so that they can be warped in the same manner as in part A.

Detecting corner features

Using the Harris corner detector code provided in the starter code, we can retrieve a lot of points landing on corners. Here is an example:

Original Harris Points

However, we can see that there are just too many selected points. We can use Adaptive Non-Maximal Suppression to choose a more sparse selection of points that are spread out. Here are the resulting points.

Image A Image B

Extracting Feature Descriptors

For each point, we can extract a 40x40 size feature, and downsample the features to 8x8 each. We also have to normalize the image (mean = 0, stdv = 1). Here is one such example.

Matching Feature Descriptors

To match the features in image A and image B, we do the following. For each point in A, check the closest and second-closest matching features in B. If the ratio of the SSD of the two is less than some threshold, then there is a good chance that the two features match. After doing this process, we remove many of the non-matching points.

Image A Image B

Using RANSAC

This removed many of the non-matching points, but to really remove all of the points that do not match, we need to run RANSAC. This basically involves repeatedly selecting random candidate matches, computing the homograpy matrix, and seeing which one creates the most number of inliers.

Image A Image B

As you can see, RANSAC succesfully chose all of the points that actually matched in image A and image B. Now we can automatically merge the images from part A. Here are the resulting images. As you can see, the alignments are pretty similar. However, the automatically selected points tend to be more accurate, so the blending is noticeably cleaner.

Manually Selected Points Automatic Selected Points

Conclusion and Thoughts

Overall, I've learned a lot about mosaics and really had fun with this! Now I know how Google Photos learns to automatically merge photos.