1. Recover Homographies

\[ \begin{bmatrix} x' \\ y' \\ w' \end{bmatrix} = H \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} \]

\[ H = \begin{bmatrix} h_{00} & h_{01} & h_{02} \\ h_{10} & h_{11} & h_{12} \\ h_{20} & h_{21} & 1 \end{bmatrix} \]

\[ (h_{20}x + h_{21}y + h_{22})x' = h_{00}x + h_{01}y + h_{02} \] \[ (h_{20}x + h_{21}y + h_{22})y' = h_{10}x + h_{11}y + h_{12} \]

Image 1	Image 2
Image 1 (labeled)	Image 2 (labeled)

2. Warp the Images

Image 1

Image 1 Warped

3. Image Rectification

The Economist

The Economist (frontal-parallel)

Costco

Costco (frontal-parallel)

4. Blend the images into a mosaic

Image 1

Image 2

Mosaic

Example 2

Fire Trail 1

Fire Trail 2

Mosaic

Example 3

Tyndall Effect 1

Tyndall Effect 2

Mosaic

Feature Matching for Auto-Stitching (Part II)

• Interest Point Detection (with single-scale Harris corners)
• Adaptive Non-Maximal Suppression (ANMS)
• Feature Descriptor Extraction
• Feature Matching
• Random Sample Consensus (RANSAC) for estimating homography

1. Interesting Point Detection

Using the harris package provided by staff, I was able to find the Harris corners for each image.

Indoor 1 (Harris Corners)

Indoor 2 (Harris Corners)

One immediate thing we can notice here is that there are way too many Harris corners. Ultimately, we are looking for invariant and distinctive point descriptors, so we need to get rid of the noisy corners and only keep the best results.

2. ANMS

To this end, we use the adaptive non-maximal suppression (ANMS) method discussed in the original paper. Here are the results of both images after we've applied ANMS to suppress the less optimal corners.

Indoor 1 (ANMS)

Indoor 2 (ANMS)

3. Feature Descriptor Extraction

Next, we extract the descriptions of local image structures to support the reliable and efficient matching of images. As decscribed in the MOPS paper, we start with a 40 x 40 window, and downsample it to 8 x 8 (this is performed at a higher pyramid level, i.e. with Gaussian filter applied). The descriptor vector is then normalized to mean 0 and standard deviation 1.

4. Feature Matching

With the descriptors extracted, we can move to the feature matching stage. We are essentially finding nearest-neighbors for the descriptors. It works as follows:

• For each descriptor in the first image, we compute the distances to all descriptors in the second image.
• Then, we identify the 1-NN and 2-NN and calculate the ratio of the distance of the 1-NN to the 2-NN.
• If this ratio is below the specified threshold (0.7, as shown in the paper), the match is considered good and is added to the list.

Here are the matched features:

Indoor 1 (ANMS)

Most matches are good, although there are a few outliers, which we will address using the RANSAC algorithm.

5. RANSAC

The RANSAC loop works as follows:

repeat {
Select four feature pairs at random.
Compute homography H.
Compute inliers where \[ dist(p_i',\textbf{H}p_i) < \epsilon \]
}
Keep largest set of inliers.
Re-compute least-squares H estimate on all of the inliers.

Here is the mosaic generated using homography H computed this way:

Indoor (Mosaic)

Let's compare it to the previous version, whose homography was computed using hand-labeled points:

Indoor (Auto-Stitching)

Indoor (Manual)

Auto-stitching provides us with a more stable and reliable way to create photo mosaics.

More Examples

Example 2: Trail

Trail 1 (Harris Corners)	Trail 2 (Harris Corners)
Trail 1 (ANMS)	Trail 2 (ANMS)

Trail (Matches)

Trail (Mosaic)

We can compare and see the improvements:

Trail (Auto-Stitching)

Trail (Manual)

Example 3: Sky

sky 1 (Harris Corners)	sky 2 (Harris Corners)
sky 1 (ANMS)	sky 2 (ANMS)

sky (Matches)

sky (Mosaic)