Feature Matching for Autostitching

CS194-26 Image Manipulation and Computational Photography Project 6 Part B

A CS 194-26 project by Kristin Ho, cs194-26-aai

The goal of this project is to create a system for automatically stitching images into a mosaic. We extract the algorithm from the research paper Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al. with slight simplifications. This is a follow up to Part A of a larger project.

Running the Code

numpy and skimage are needed to run the code. Matplotlib and a whole bunch of other stuff too.

Part 1: Detect corner features in an image

I use a Harris corner/Harris interest point detector.

I limit the number of Harris corners/feature points detected by performing Adaptive Non-Maximal Suppression (ANMS). Essentially, the concept is to retain only the strongest interest points, but also ensure that they are spread throughout the image.

I will not elaborate on the theory too much, but in practice, one ensures that one selects only feature points that have a large radius around them before reaching another, similarly strong feature point i.e. that the radii for selected features fulfill:



Original image.
Image with detected Harris corners.
Image with Harris corners after performing ANMS.

Here are the ANMS filtered points for the left and right hill images.

Left hill image.
Right hill image.

Here are the ANMS filtered points for the left and right piano images.

Left piano image.
Right piano image.

Part 2: Feature Descriptor Extractor For Each Interest Point

A bias/gain normalized sampling of a local patch (8x8) from a higher pyramid level (40x40) is used to create a vector feature descriptor for each interest point. These descriptors are used for feature matching.

Part 3: Feature matching/matching descriptors between two images

The goal is to find pairs of feature that look similar and thus are likely to be good matches between the two images we want to stitch together. The similarity between two interest points is computed by taking the SSD between their two descriptor vectors.

Lowe's paper found that the difference between first nearest neighbors and second nearest neighbors was a more accurate marker for matching than the absolute distances of the first nearest neighbors.

Thus order to determine what a good "match" is, after running a simple nearest neighbors algorithm, the ratio between the first best match and second best match for each interest point in one image is compared to a threshold ratio based on the outlier distance.

The matching feature points between two images are marked below with "X"s.

Left flags image and interest points
Right flags image and matching interest points

Part 4: Using 4-point RANSAC to compute a robust homography estimate.

Random Sample Consensus: select one match, count inliers.

Over many iterations: one selects 4 random feature pairs. The homography H is computed exactly as in Part A of this project. Then, one computes inliers on all pairs of interest points, where SSD(pi', Hpi) is less than some threshold error. Then, after many iterations, one takes the iteration that generated the largest set of inliers, and re-computes the least squares H estimate on all of those inliers.

Below I show the "inliers" that resulted from running RANSAC 10,000 times for a few pairs of photos. They are labeled with numbers so one can see the correspondences better.

For the flags images:

Left flags image feature point inliers from RANSAC
Right flags image feature point inliers from RANSAC

For the hill images:

Left hill image feature point inliers from RANSAC
Right hill image feature point inliers from RANSAC

For the piano images:

Left piano image feature point inliers from RANSAC
Right piano image feature point inliers from RANSAC

Part 5: Stitching into mosaics

Using the H computed from the automatically detected corresponding feature pairs, I warp the images and stitch them together into mosaics.

Below I will show the side by side results of auto-stitching versus manual stitching (part A).

Flag mosaics:

Autostitched flag mosaic
Manually stitched flag mosaic

Hill mosaics:

Autostitched hill mosaic
Manually stitched hill mosaic

For the piano images:

Autostitched piano mosaic

Manually stitched piano image n/a since I did not manually stitch one in part A.