Image Warping & Mosaicing

Nancy Li

Overview

The goal of this project is to get familiar with image warping, and a cool application of it: iamge mosaicing! Part A of this project involves computing a homography between a pair of images, to project one image into the geometry of the other. Then, we are able to blend warped images together to create a panorama image.

Computing the Homography Matrix

After asking for at least 4 user defined input points, we compute the variables of the homography matrix using linear algebra. We solve for Xh = X', where h is an 8x1 vector represeting the 8 unknowns of the homography matrix, X is the user defined points, and X' are rectified points. Images are then warped using the homography matrix.

Mosaic

To create a mosaic or panorama, first choose one image as the base image. The other image is taken and warped to the base image's coordinates. Then the two are blended. After aligning the images, I take the maximum pixel value between both images to create the stiched mosaic.
I took pictures of the interior of First Presbyterian Church in Berkeley, and a view of San Francisco from a Salesforce office building. The pictures were taken on a phone, and you can see some image artifacts due to different lighting.

Part 2: Autostitching!

Harris Corner Dection

Previously, corresponding feature points in each pair of photos are picked by hand. In this section we use the Harris corner detector to automatically pick out features. I used the sample code from harris.py to extract feature points.

Adaptive Non-Maximal Suppression

Harris points returned in the previous section aren't very helpful; using ANMS, we can choose feature points more evenly distributed across the photo. For each Harris point, we find the distance to the closest feature point. Then map each point by this distance, and return the top 500 points from a list of descending value.

Feature Descriptor

For each harris corner point, we create a feature descriptor by sampling from a 40x40 pixel region around the point; each sample pixel is spaced by 5 pixels. This creates a 8x8 descriptor vector, which is normalized by subtracting out the mean and dividing by the standard deviation.

Feature Matching

Now we match points across a pair images by finding the SSD of their feature descriptors. Two points are considered similar if their features' SSD is small. We take the two smallest SSD patches, and only flag for a match if the ratio of the SSDs between the smallest SSD (1-NN) and the second smallest SSD (2-NN) is less than a threshold.

RANSAC

Now that we have a set of matching features across two images, we want to calculate the homography and warp one image to the other for the mosaic. We use the RANSAC method to find the best homography. The algorithm works by picking 4 random features, computing the H matrix, and computing the inliers after applying this H to all feature points. We keep the largest set of inliers, and recompute the homography matrix from the inliers. After finding the best homography, we can then stitch the images together using the same methods as part 6A. Shown below are the results, which are on the left, compared to the mosaic from the manual alignment on the right!

What I've learned

This is how phones automatically make panoramas! I learned that a simple 3 by 3 homography matrix is enough to warp an image into a whole new geometry, so powerfully that I can even read a sign that I could not read in the original image. I also learned the importance of taking good photographs with good alignment; sometimes image artifacts are unavoidable no matter how good the mosaic/blending algorithm is, if your images don't align well.

HTML/CSS for this page was taken from https://v4-alpha.getbootstrap.com/components/card/