Project 4A: Image Warping and Mosaicing

CS 194-26 Fall 2021

Bhuvan Basireddy



Shoot the pictures

I took pictures with my phone of places outside and some scenes indoors. Then, I labeled the correspondences.

Recover Homographies

To compute the homography between 2 images, I passed in 2 sets of correspondence points. Denote these as pts1 and pts2. We want to compute the homography matrix that transforms pts1 to pts2. I constructed the following matrix using pts1 and pts2, where pts1 = (x, y) and pts2 = (xˆ, yˆ). I solved for the h vector, appended a 1 to it since we can just set h33 to 1, and reshaped it into the 3x3 homography matrix H.

Warp the Images & Image Rectification

We warp the images by selecting a target image that we want to warp to. We do this by selecting points shared with 2 images and computing a homography then warping one image to the other with the homography by applying it to that image's coordinates. Then, for each pixel in our final image, we find where it comes from in the source image using the coordinates and interpolate to get the value. For rectifying, we can just select 4 points to be a bounding box on a plane in the picture, such as the 4 corners of a table, and we can warp to that plane to get the top-down view shown below.
Initial
Rectified
Initial
Rectified

Blend the images into a mosaic

Now, we can blend images. I selected a couple of images, where I warp one of them to the other using the homography, and stitched them together into a mosaic by doing a weighted average of the overlapping region of the images, which I found by creating boolean masks of the images and seeing where the product is 1.
First Image
Second Image
Mosaic
First Image
Second Image
Mosaic
First Image
Second Image
Mosaic

Project 4B: Feature Matching for Autostitching

CS 194-26 Fall 2021

Bhuvan Basireddy



Detecting Corner Features

For detecting the corner features, we used a Harris Interest Point Detector that we were given. I had to change the radius for peak_local_max to get the local maximums in a 3x3 neighborhood as in the paper. I used a threshold, if needed, to reduce runtime. There's a 20 pixel margin around the points, so we can do the feature extraction with a 40x40 area around each interest point.
Now, we can run adaptive non-maximal suppression (ANMS) on the interest points to get strong interest points that are also spatially distributed. For each interest point I, I found the minimum suppression radius, which is the minimum distance to a point P such that the corner strength of I is less than the corner strength of P * c_robust. I defined c_robust to be 0.9 as in the paper. After finding the radii for every point, I took the top 500 points with the highest radii as potential features.

Extracting and Matching Feature Descriptors

For each candidate feature point, we find the 40x40 patch around it. Then, we subsample every 5 points to get an 8x8 patch. We bias/gain-normalize by subtracting the mean from the patch and dividing by the std. This patch can then be flattened into 64 dimensional descriptor vector.
8x8 Feature Descriptor

We do this for the 2 images that we want to stitch together, so we have descriptor vectors for each candidate feature point of both images. The 2 sets of vectors are now used for feature matching. We use the dist2 function to compute a matrix of distances between 1 set of vectors to the other set of vectors. We want the nearest neighbor (1-NN) and next nearest neighbor (2-NN), which are the 2 lowest values in each row of this matrix. If the ratio of 1-NN/2-NN is less than or equal to some value (which we set for different images), we keep the feature match.

RANSAC

Using the matching features, we do 4-point RANSAC to get rid of any false matches. We randomly choose 4 matches and compute a homography H with them. Then, for each of the feature matches, we find the distance between H * p1 (point for 1st image) and p2 (corresponding match in 2nd image). If dist(H*p1, p2) is less than some epsilon (I used 1.0), then it is a good match and we count the number of good matches. This is done for 1000 iterations, and the homography with the most good matches is returned as the final homography. This can get rid of correct matches, but overall, it keeps most of the correct matches and gets rid of false matches.

Mosaics

Now, we can compare the manually and automatically stitched mosaics.
Manual
Auto
Manual
Auto
Manual
Auto