CS194-26 Fall 2017 Programming Project #6B
Betty Chang

FEATURE MATCHING for AUTOSTITCHING

Objective

The goal of this project is to create a mosaic from images of the same scene from different perspectives. We accomplish this by using correspondence points between the images to recover the perspective transformation matrix. Then, we warp all the images to the same perspective, and blend the images together to create a mosaic. The second part of the project is where we automate this process by using automatic feature matching. We first find the Harris corners of the images, filter the points using ANMS, perform feature matching, and use RANSAC to recover the best homography. Then, we use the same warping and blending process as in the first part to create the mosaic.

Image Rectification

Here are some examples of rectifying images:

Laptop

Laptop Rectified

Wallet

Wallet Rectified

Image Warping

We start out with three images of my living room couch:

Image 1

Image 2

Image 3

We then warp images 1 and 3 to the perspective of image 2 by computing their respective homographies. To do this, we find correspondence points between the images (I used the edges of the tables and couch), and find the perspective transformation matrix from image 1 to image 2, and from image 3 to image 2. Then, we find the bounding box of our resulting image by applying the transformation to the corners of the images. We then inverse warp each image to image 2's perspective of this new size:

Image 1 Warped

Image 3 Warped

Blending

Once we have the warped images and image 2 resized appropriately, we can blend them together to create our mosaic. For blending, I used an alpha mask for each image to determine where there are pixel values. For each pixel of the resulting image, I average together the images that have a value at that pixel.

Mosaic of My Couch

As we can see, the images do not line up perfectly. This is likely because I only used 4-6 points to recover the homography for each image. If I had used more points, the transformation would have been better.

Additional Examples

Image 1

Image 2

Image 3

Chapman University Library Mosaic

Harris Corners

To implement automatic feature matching, we first use the provided Harris Interest Point Detector to find the corner points of the image.

Image 1

Image 2

Image 3

As we can see, we have a lot of interest points (on the order of 1000) and a lot of the points are closely clumped together.

Adaptive Non-Maximal Suppression

To extract the best points for the Harris corners we have above, we will perform ANMS to choose only 250 points per image. For each corner point, we will calculate its radius - defined as the distance from the nearest corner that is significantly stronger than this corner - and take the 250 points with the largest radius.

After performing ANMS, we have significantly fewer points that are better spaced out.

Image 1

Image 2

Image 3

Feature Descriptor Extraction

Next, we extract the feature vector for each corner we have left. For each corner, we find the 40x40 patch around that pixel, then low-pass filter and subsample it to be an 8x8 patch. Then, we flatten that to a length-64 feature vector.

Feature Matching

To perform feature matching, we find the distance between each feature vector in the first image and each feature vector in the second image. Then, we only keep the features where the 1-NN/2-NN ratio of that feature is below a certain threshold (to ensure that we minimize the number of outliers we include).

Image 1 Feature Points

Image 2 Feature Points

Image 3 Feature Points

Image 2 Feature Points

RANSAC

For each pair of images, we perform RANSAC to compute the best homography. For each iteration, we randomly pick 4 matching feature points, and recover the homography from those points. Using that homography, we compute the number of inliers it produces (where the projected point is within 10 pixels of the actual point). After performing 1000 iterations, we pick the homography that produced the largest number of inliers. Lastly, we take that set of inliers and compute an approximate homography using least squares as our final H matrix.

Once we have the H matrix, we perform the same warping and blending steps as we did with manual stitching to output the mosaics.

Manual vs. Automatic

Overall, the automatically stitched mosaics turned out better. This could be because there were more points used in computing the homography, and because automatic stitching eliminates the error of clicking points manually.

Chapman U Hand Stitching

Chapman U Automatic Stitching

Unit 1 Hand Stitching

Unit 1 Automatic Stitching

My House Hand Stitching

My House Automatic Stitching

Summary

The coolest part of this project for me was seeing how mosaics and panoramas are made, and learning how to automate that process using automatic feature matching.