Tejas Thvar, Fall 2021
3 sets of panorama photos are shown below, along with their correspondences. Correspondences were recoreded by using correspond.py, a script wrote using plt.ginput. Note that 7 correspondences were used to create a more pleasing panorama result. AE/AF exposure was locked between takes.
Homographies are calculated by computing the pointwise projection matrix of the respective sets of correspondences. This ends up taking the form of a matrix problem where the homography is the solution for the projection. The homography matrix was set up as according to the image below (source: https://towardsdatascience.com/estimating-a-homography-matrix-522c70ec4b2c). We set the last coordinate of the homography matrix, h33 to 1 and solve using OLS (overdetermined system).
-> 
Warps were done by calculating the homographies for the correpondence point values and warping the source image to the target. The canvas dimensions in the background were generated by calculating the warped coordinates and then creating the largest possible canvas necessary to hold all warped coordinates. The offsets to translate each image were similarly calculated by looking at the min warped coordinate and max warped coordinate (min was negative, indicating need for offset), and adjusting respectively. The warped images for the Berkeley landscape are shown below.
In order to test that homography and warping was working correctly, the below images were rectified by seeing them in a front-view.
In order to blend the images, a simple alpha blending procedure was used. Shown below are the masks used for the left and right image of the Berkeley landscape example - the gray part is representative of the overlapping sections of each image. I used a distance transform to obtain the individual masks and used a bitwise and to obtain the intersection. The results of said alpha blending is shown below on the Berkeley Example, as well as the other two examples.
In this project, I learned that homographies are really just simple projections and are super interesting in the world of art/street painting. I also learned how critical labelling accurate correspondences is, as it significantly affected the quality of my end results (trees and other objects with finer details ended up not aligning as well/blurring slightly when computing the final mosaic.)
To compute the Harris Corners in the images, the starter code was used. Due to the size of the image, the min_distance parameter = 10 and sigma = 7. Results are shown below on the right and left images.
To compute the adaptive non-maximal suppression the suppression radius formula as described in the paper was used [included below]. I experimented with a couple of crobust values but also ended up getting the best results with 0.9. We then take the top 500 points returned from anms ranked by suppression radius/SSD. Results are shown below.
Once anms points were obtained, we then need to extract 40 x 40 patches around each point. We then blur this patch and downsampled/normalized to 8 x 8. The 8 x 8 patches for Berkeley are shown below.
We then compute the nearest and second nearest neighbor for each 8 x 8 patch using SSD. We then used a threshold of SSD[NN1] / SSD[NN2] < 0.3 to count as a match using Lowe's Trick/the graph in the paper to obtain threshood.
We then run RANSAC using the following general algorithm: 1) get 4 random pairs of points, compute homography on those 2) compute homography * point for all feature points in img1 3) compute ssd on result from 2 and the corresponding points from img2 4) if the ssd for any feature pair is less than 2 add to inliers 5) if the number of inliers is greater than the prior max inliers, set to the max inliers set, repeat for N - 1 times. I chose to run RANSAC N = 4000 times to obtain optimal points. The RANSAC'd feature set is shown below, it is evident that outliers from anms have been eliminated.
Below results are shown for the manually stiched [L] and auto-stitched [R] panoramas. For the first Berkeley panorama, we can see that the auto-stitched is superior/less blurry, especially when looking at the sharpness of the darker brown house to the right. This is similarly true for the mosaic'd apartment [notice Amazon box in cabinet's bottom left]. I tried one more example to compare autostitching versus manual for a panorama of my bedroom, which had similar results (note how much blurrier the left side of the desk is on the left image).
In this part of the project, I am super glad to have learned Lowe's trick which offers a really intuitive method of pairing features. It was also particularly interesting comparing the auto-stitched and manually stitched images as I was surprised how much better auto-stitched panoramas were.