Project 6B: Homographies and Image Stitching

Utkarsh Singhal

Introduction

The goal of this part of the project is to do automatic image stitching by detecting Harris corners, filtering them using adaptive non-maximum supression, matching them using oriented patches, filtering outliers using RANSAC, and then finally computing homography. The end result is surprisingly good, and can even perform better than humans (me).

Detecting Harris Corners

The first part is very simple: Harris Corner detection. I found that I got much better results when I thresholded the corener strength. Here is an example of what these detected corners look like:

Without Harris Corners

With Harris Corners (thresholded at 4*mean)

Adaptive Non-Maximum Suppression

Since Harris Corner extraction returns too many points, one needs to select a useful subset of them. ANMS tries to find a subset of points with high corner function value and even spacing by suppressing points that do not correspond to local maxima in an adaptively defined radius. I also made it robust by only suppressing a point when its neighbors have significantly higher corner function value. This process results in a much better set of points.

Without ANMS

With ANMS

Feature Extraction and Matching using SSD

Once we have the keypoints in an image, the next step is to match the keypoints in each to their corresponding points in the other image. However, since we want this matching to be invariant to perspective transformations, we have to use certain transformation-invariant features for the matching. The literature on these features is very rich, but for this project we decided to use downsampled patches around the keypoints. Matching was done by comparing SSD of these patches, and I used Lowe's 1NN/2NN error ratio threshold to get the final result. This worked suprisingly well, as most of the matched points were correct. However, there were some important corners (like some mountain peaks) that were missed in this process.

Image 1

Matched Points

Image 2

RANSAC

Regardless of its simplicity, RANSAC was the most powerful part of this project, and it helped improve my result quality by a huge factor. The basic idea behind RANSAC is to discard outlier points by choosing random subsets of the data to compute homogrpahies, and only accepting the inliers. This process significantly improved the quality of the SSD-based matches. This algorithm is my favorite part of this project.

Results

In general, I found automatic stiching to be better than when I marked the keypoints manually. There were some cases where the manual method completely failed, and only this method worked.

A house on Leconte Street

A house on Leconte Street (Manual)

A house on Leconte Street (Auto)

Memorial Glade

Memorial Glade (Manual)

Memorial Glade (Auto)

A sunny day in Berkeley

A sunny day in Berkeley (Manual)

A sunny day in Berkeley (Auto)

Summary

This was an exciting project, and it was great to see how well this works. Even though the underlying ideas are simple, their final effect was quite mind blowing. I think that RANSAC was the star of this project, and it is unbelievable how much one can get away with in SSD feature matching with RANSAC in place.