Overview

For this part of the project we explore automatically stitching images into a mosaic.

1. Detecting corner features in the image.
2. Extracting a Feature Descriptor for each feature point
3. Matching these feature descriptors between two images
4. Use a robust method (RANSAC) to compute a homography
5. Proceed as in Part 1 to create Mosaic

Harris Interest Points

We use Harris corner detection where the basic idea is that when looking at a small window in a region a shift in any direction will cause a large change in intensity, a very simple yet powerful idea. We apply Adaptive Non-Maximal Suppression to get interest points that are local maxima within a radius. Note, the points are actually obtained from a blurred version of the image to reduce noise.

Feature Extraction

For each point of the previous part we extract a feauture, an image patch around the area (in a blurred version of the image) of the point. We then decimate it to get a smaller patch.

As one can see, blurring before decimating helps reduce aliasing.

Feature Matching

We now compare ALL patches to each other and use L2 difference as a measure of similarity between patches (after normalizing by subtracting the mean and dividing by the standard deviation). This yields a very rough estimate of the matches and has a lot of error. We used 1-NN/2-NN thresholding for this. This means we picked points that had a low ratio of L2 similarity of the nearest neighbor over L2 similarity of the second nearest neighbor. This intuitively makes sense to me because we do not want to pick points that can be mapped well to two different points. For example if you have a brick wall a single point can very well be matched to another point in the wall. In this case it could also be very well matched to a second point in the wall which means we get a high ratio 1-NN/2-NN and discard that point which makes sense because we do not want to try to match points in the brick wall since we probably pick different points that look similar.

RANSAC

We now use the previous matching and use RANSAC to get a better matching. Here the idea is that we randomly pick 4 points from our matchings and use the to compute a homography (exact for the 4 points since we have 8 DOF). We then count the number of other points that when applied this transformation get mapped to the matched harrison points. We do this iteratively and chose the homography that maximizes the number of such points and use those points to calculate the final homography matrix using least squares.

MOSAIC

Now that we have the transformation matrix we follow the same steps as for part 1 to get the mosaic.

Note, in this example the final result look very similar but the auto result looks better when especially when you look at the lower right corner.

More examples

The most interesting thing about this part of the project was learning the harris interest point detection method (including ANMS) and RANSAC. It was cool to see all the points the detector selected and how RANSAC improved the selection of matches drastically and very quickly while being very simple and intuitive.