Homography.

We can transform a set of points by using homography, such that p' = H * p. To recover the homography from two sets of points, we set up a system of linear equations. The two sets of corresponding points had to contain at least n = 4 points each since we have eight unknowns to solve for. Once we find the homography matrix, we can use it to warp our images. To rectify an image, I chose rectangular shaped objects, such as a wall, from the image and warped them to rectangles in order to make them front facing.

Here is a picture of the wall art from a coconut/sugarcane juice bar in my hometown.

Here is a picture of a blanket that my cousin crocheted for me. I focused on rectifying the waterbending symbol.

Mosaic

To create a mosaic image, I chose one image to be the center image, and warped images to the left and right of the center image towards the center image one at a time. To combine the images together, I used the multiresolution blending technique (Laplacian stacks) from project 3, creating the appropriate masks for each pair of images that were blended together.

My Room: Leftmost Image

My Room: Center Image

My Room: Rightmost Image

This image was blended with the already blended leftmost and center images.

My Room: Panorama

The top image represents all 3 images stitched together. There is some misalignment due to the masks I used and the points that I chose. The bottom image is a cropped (cleaner) version of the panorama.

My Living Room: Leftmost Image

My Living Room: Center Image

My Living Room: Rightmost Image

My Living Room: Panorama

Campus: Leftmost Image

Campus: Center Image

Campus: Rightmost Image

Campus: Panorama

Harris Interest Point Detector

I use the code provided in harris.py to find the Harris Interest Points. When using the peak_local_max function inside of harris.py, it generated a lot of points. I reduced this by using corner_peaks. Below are the images used to make a panorama of my room with the Harris points overlaid on top. The left image uses the peak_local_max function and the right image uses the corner_peaks function.

Leftmost Image: Harris Points using peak_local_max

Leftmost Image: Harris Points using corner_peaks

Center Image: Harris Points using peak_local_max

Center Image: Harris Points using corner_peaks

Rightmost Image: Harris Points using peak_local_max

Rightmost Image: Harris Points using corner_peaks

Adaptive Non-Maximal Suppression

Because the Harris Point Detector produced so many points, this causes the rest of the code, such as feature matching, to run for a very long time. We want to reduce the runtime by reducing the amount of points we use. We can pick points that are more spatially distributed across the image by using adaptive non-maximal suppression (ANMS). We use the equation below to find which points we want to keep.

Because I used corner_peaks for finding the Harris corners, I didn't have many points to deal with from the start. I ran ANMS for reducing the amount of points from using the peak_local_max function to about 500 points and the results are shown below. However, I essentially used the points detected from the corner_peaks detection. If there were less than 500 points, I decided to use all of the points detected.Otherwise, I used the first 500 points with the highest r value.

Leftmost Image: Harris Points using peak_local_max

Leftmost Image: Harris Points after ANMS

Center Image: Harris Points using peak_local_max

Center Image: Harris Points after ANMS

Rightmost Image: Harris Points using peak_local_max

Rightmost Image: Harris Points after ANMS

Feature Descriptors

To extract the feature descriptors, I found the 40 x 40 patch around each Harris corner that was selected from the previous part (ANMS). I then downsampled the patch to an 8 x 8 patch by applying a Gaussian filter and taking every 5th pixel. Then, I did bias/gain normalization: (patch - mean(patch))/std_dev(patch).

Feature Matching

To match the features between two images, I found the SSD of each feature in one image to every feature in the second image. We then found the two points in the second image with the smallest SSD to the feature in the first image. These are the "nearest neighbor" points, where 1-NN has the smallest SSD and 2-NN has the second smallest SSD. If 1-NN / 2-NN is less than a threshold (I used 0.4), then I considered 1-NN to be a match to the feature in the first image!

Leftmost Matched to Center Image

Center Matched to Leftmost Image

Center Matched to Rightmost Image

Rightmost Matched to Center Image

RANSAC

It's still possible for us to have noisy points that may mess with the homography calculation. RANSAC takes four random points and computes the homography using those points. Then, we apply this homography matrix to every point in one image and compare the result to the actual points in the second image. We pick the homography matrix that resulted in the most amount of inliers, where inliers are designated if the difference between the actual point in the second image and p' (H*p) was less than a certain threshold (I used 2). We returned the best homography matrix after 2000 iterations. To reiterate, here are the following steps for RANSAC (taken from lecture):


1. Select four feature pairs (at random)
2. Compute homography H (exact)
3. Compute inliers where SSD(pi’, H*pi) < threshold
4. Keep largest set of inliers
5. Re-compute least-squares H estimate on all of the inliers

Below, On the left, we have panoramas created using manual corresponding points and on the right, we have panoramas created using Harris detection.

Here are the panoramas of my room. They look pretty much the same here. There are some spots visible from the rightmost image in the autostitching panorama since I changed the mask a bit. However, overall, the panoramas look the same.

Here are the panoramas of my living room. We can see that some of the features look different between the two. The blinds aren't as warped in the autostitched panorama and actually look a bit more natural/realistic.

Here are the panoramas of campus. We can see that some of the features, specifically VLSB, look different between the two as well. The Campanile is also bigger in the autostitched image!

Conclusion

It's crazy to think that such a small matrix can let us stitch images together! It was also super cool to be able to change the "perspective" of a picture.