Running the Code
numpy and skimage are needed to run the code. Matplotlib and a whole bunch of other stuff too.
Part 1: Detect corner features in an image
I use a Harris corner/Harris interest point detector.
I limit the number of Harris corners/feature points detected by performing Adaptive Non-Maximal Suppression (ANMS). Essentially, the concept is to retain only the strongest interest points, but also ensure that they are spread throughout the image.
I will not elaborate on the theory too much, but in practice, one ensures that one selects only feature points that have a large radius around them before reaching another, similarly strong feature point i.e. that the radii for selected features fulfill:
Here are the ANMS filtered points for the left and right hill images.
Here are the ANMS filtered points for the left and right piano images.
Part 2: Feature Descriptor Extractor For Each Interest Point
A bias/gain normalized sampling of a local patch (8x8) from a higher pyramid level (40x40) is used to create a vector feature descriptor for each interest point. These descriptors are used for feature matching.
Part 3: Feature matching/matching descriptors between two images
The goal is to find pairs of feature that look similar and thus are likely to be good matches between the two images we want to stitch together. The similarity between two interest points is computed by taking the SSD between their two descriptor vectors.
Lowe's paper found that the difference between first nearest neighbors and second nearest neighbors was a more accurate marker for matching than the absolute distances of the first nearest neighbors.
Thus order to determine what a good "match" is, after running a simple nearest neighbors algorithm, the ratio between the first best match and second best match for each interest point in one image is compared to a threshold ratio based on the outlier distance.
The matching feature points between two images are marked below with "X"s.
Part 4: Using 4-point RANSAC to compute a robust homography estimate.
Random Sample Consensus: select one match, count inliers.
Over many iterations: one selects 4 random feature pairs. The homography H is computed exactly as in Part A of this project. Then, one computes inliers on all pairs of interest points, where SSD(pi', Hpi) is less than some threshold error. Then, after many iterations, one takes the iteration that generated the largest set of inliers, and re-computes the least squares H estimate on all of those inliers.
Below I show the "inliers" that resulted from running RANSAC 10,000 times for a few pairs of photos. They are labeled with numbers so one can see the correspondences better.
For the flags images:
For the hill images:
For the piano images:
Part 5: Stitching into mosaics
Using the H computed from the automatically detected corresponding feature pairs, I warp the images and stitch them together into mosaics.
Below I will show the side by side results of auto-stitching versus manual stitching (part A).
Flag mosaics:
Hill mosaics:
For the piano images:
Manually stitched piano image n/a since I did not manually stitch one in part A.