This portion of the project builds on top of the concepts of stitching learned in project 6a. Instead of manually selecting transform points, however, we compute points through various techniques.

We start by detecting the Harris corners of each image, which are simply "pointy" points in an image, hence the "corner" in Harris corner. However, the points returned in the provided Harris corner function is superfluous, so I started by taking the top 2000 points ranked by Harris strength. Below is an example for my room.

Next, we refine our selection using ANMS. ANMS is important because we don't want redundancy in the information our points from the Harris corner provides, and at the same time we want an even distribution of points from the images. We can compute the minimum suppression radius for each point, which is essentially the smallest distance to another point in which its Harris strength is significantly greater than the original point's. I sorted the points by their suppression radii and selected the top 500.

After significantly reducing our interesting points by ANMS, we can begin to match points between images. We can define the feature descriptor for a certain point as a 40x40 patch of the image with the point in the center, which is then downsized to 8x8, and finally bias/gain normalized. In my implementation, I applied a Gaussian filter (sigma=2) and just took every 5th sample of the 40x40 to get my 8x8 feature. Then, for each point in the first image, I looked at every point in the second image and looked for which pair produced the smallest SSD (sum of squared differences) between the 2 feature descriptors. After finding the smallest SSD, to make sure that these points were valid, I checked if (smallest SSD) / (nest smallest SSD) < 0.5, where 0.5 is an arbtirary threshold, to make sure that the pairing I found was valid as a transformation point. Below is an example of the resulting matched pairs between the left and center images.

Because feature matching still doesn't guarantee every matching is valid, we can use RANSAC as a technique for computing an effective homographic matrix H. For some number of iterations (I used 1000), we can select 4 random sets of points between the two images. We can compute a temporary H, and then for the rest of the point pairings we compute the error between our projected point and our true point. I used SSD again to calculate the error. For projections that fell underneath a certain error threshold (I used 800), we keep track of a running list of these pairs, called "inliers", during each iteration. After going through all 1000 iterations, I looked at which iteration yielded the largest number of inliers, and recomputed H using ALL the inliers as sample points instead of just the 4 randomly selected points during that iteration. RANSAC was very effective in computing an effective H and ultimately making an effective stitching. Below are the results:

I found it really cool how we were able to detect points of interest and actually interpret and process them automatically. Harris corners reveals a lot about what goes on during image processing and how images could even find features in the first place, and feature extraction / matching shows me the beauty in simple heuristics.