Project 6b: Feature Matching for Autostitching

About

In this project, we explored projective transformations and homographies through mosaic construction. Given multiple images, we find the best transform from all the images to some reference image and stitch them together to form a mosaic.

In this project, we explore automatic feature matching to automatically stitch pictures together using the method from part A. Instead of hand picking the correspondence points, we do the following:

  1. Use Harris Edge Dectector to find edge features.
  2. Perform Adaptive Non-Maximal Suppression (ANMS) to filter points in a well-distributed manner.
  3. Extract features for each point (i.e. a box around each point)
  4. Match features using the ratio between best 1-NN and 2-NN.
  5. Use Random Sample Consensus (RANSAC) to choose the best of the remaining points
  6. Compute the homography and morph the images as in the previous part

Shooting the Pictures

To compare performance, I use the same first set of pictures as the previous project. The remaining portions will only display the matching between the first and the second image.

Harris Edge Dectector

The following are edge features from the Harris Edge Dectector on each of the three images. As we see, there are way too many points.

Adaptive Non-Maximal Suppression (ANSM)

ANSM cycles through each point and rates each points importance by their distance to the nearest point with a Harris feature value greater than it, with a large distace meaning greater importance. Intuitively, ANSM chooses the most important points in its vicinity. This prevents clustering of points in ares with high Harris feature values. The following are the results of performing ANSM on the two sets of points.

Feature Extraction and Matching

We now want to filter out the points even more and find the best correspondences between points in one set with points in the other. To extract features for matching, I simply extract a 40x40 box around each point, and down-sample it to an 8x8 with Gaussian blurring to remove noise.

To find the best correspondences, simple SSD will not work well because points in such a high dimensional space will have distances that are close to all the other points, and when it comes to feature matching, we cannot have a single mismatch. A better method of matching features is to take the ratio between the distane of its first nearest neighbor and its second nearest neighbor. This works because of the assumption that the first neighbor neighbor's feature should match much better than the second (i.e. there is only one good match for each point in on set with the other). The following is the result of running this extraction and matching algorithm.

Random Sample Consensus (RANSAC)

Lastly, there may still be noise in the matches, so we perform RANSAC. First we take a random sample of four points and compute the exact homography. Then we cycle through each point in one set and see if the projective transformation of this point lies within some error of the expected correspondence. If so, we count it as an inlier and add it to set of inliers. We repeat this process over many iterations and take the largest set of inliers. The following show the remaining points after performing RANSAC.

Blending

With the point correspondences, we can perform the same blending as the previous project. I compute the homography between the two sets of points and warp one image to the other using inverse-warping. The following is the result of performing the algorithm on all three images and warping to the middle image.

More Mosaics

Here are some more pictures and their resulting mosaics.

Summary

This project has shown me how even something that may seem intrinsically human (i.e. determing point correspondences between two images) can be done programatically and with much better results. I also learned a lot about image processing and feature detection and matching.