Overview
As in Part A of this project, we continue to create our own panoramas. However, instead of selecting our own correspondences with which to compute the homography, we will create a system for automatically stitching images together.
Starting Images
We will use the same starting photos as in Part A (click to enlarge):
Lassen Volcanic National Park
Lassen Volcanic National Park
Crater Lake
Crater Lake
NYC
NYC
Harris Corner Detection
We select corners for our possible corresponding feature points, because they are easy to pinpoint. To detect corners automatically, we use the Harris Corner Detection algorithm. Resulting corners shown below (click to enlarge):
Lassen 1
Lassen 2
Lake 1
Lake 2
NYC 1
NYC 2
Suppressing Corners
We want to choose the best corners for our feature points, and also, have them spread across the image, to produce a better panorama. Therefore, we run Adaptive Non-Maximal Suppression on our Harris corners: choosing corners that are the "best" corner in their neighborhood (within a radius r).
Decreasing r increases the number of corners that pass the suppression test, so one must balance the radius, and the number of desired corners, appropriately. Below are results in the neighborhood of 50 corners and radius of 100 pixels (click to enlarge):
Lassen 1
Lassen 2
Lake 1
Lake 2
NYC 1
NYC 2
Feature Matching
In order to find correspondences between features (corners), we first must be able to describe each feature. We use the Feature Descriptor as described in Section 4 of this paper: taking 40x40 patches around each feature, sampling into 8x8 patches, then bias/gain normalizing each descriptor.
Then, to find matches between features, we compute the distance between descriptors. A smaller distance indicates a better match. However, "incorrect" matches can also have low distances, and an objective threshold on the distance does not work well in practice. Therefore, we threshold on the ratio between the first and second nearest neighbor distances (lovingly referred to as the Russian Granny Algorithm). If the ratio passes the threshold, we consider that feature and its nearest neighbor to be a match.
Below are the resulting matches (click to enlarge):
Lassen 1
Lassen 2
Lake 1
Lake 2
NYC 1
NYC 2
Computing Homographies
Even with the Russian Granny Algorithm, we might still have false matches, which would really screw up our homography calculations. Therefore, we calculate our homography only with inliers.
We use 4-point RANSAC: choose 4 correspondences at random, compute the homography, then apply this homography to every one of our matches. If the calculation equals the supposed match (within some margin of error), the correspondence is considered an inlier. We repeat this process many times, and keep track of the largest set of inliers. After n iterations, we take the largest set of inliers, and use them to compute a robust homography.
From that homography, we then warp the images, stitch them together, and blend to ease out the artifacts, just as we did in part A. The results are shown below (click to enlarge), next to the corresponding results from part A (created with manually defined correspondences). All parameters have been kept the same.
Lassen Volcanic National Park B
Lassen Volcanic National Park A
Crater Lake B
Crater Lake A
NYC B
NYC 2
You can see that the automatically stitched panoramas are more aligned and crisp.
Summary
I thought this was really, really cool. A lot of work, but really, really cool. The fact that this algorithm can choose correspondences bettter than I can, despite the fact that it can't even see or understand the images, is absolutely insane. Humans are really amazing to have created all this, but I can see why people are afraid of the computers taking over.
My thoughts from part A still stand: I'm really impressed by the people who create these algorithms, and also all the impressive technology we take for granted now. My code, notably the corner suppression, took a long while to run. But my smartphone, with way less muscle than my laptop, can stitch together a panorama in a matter of seconds. So to all the people out there who make these modern marvels: you are amazing.