Richard Liu
For Part A of this project, I'll be focusing on manually establishing correspondences between two images and then programmatically stitching together into a mosiac by calculating a homography, warping, and alpha blending.
Our first step is to take some pictures! These are the two images I'll be warping into a mosaic.
And here are the keypoint pairs I selected on the two pictures!
After loading the points from our cameras, we compute the homography vector $\vec{h}$ by solving for a system of equations in which $A\vec{h}=\vec{b}$.
Thus, we get our $3\times3$ homography matrix as follows:
[[ 2.24239286e+00 -1.12331295e-01 -4.90250482e+03] [ 4.78055033e-01 1.73964822e+00 -1.13386337e+03] [ 3.38357385e-04 -6.44486286e-05 1.00000000e+00]]
Now, we move to warping the images with our given homography matrix. We will use inverse warping, first forward warping to get the corners of our warped image, then iterating through each pixel location in the warped image and using interpolation to find the correct color value. We use cv2.remap to help parallelize this.
We can also apply these homographic transformations to warp the perspectives of images. One such application is to rectify images that are taken from a slanted perspective. Here is an image of the floor from an angled view, where I've selected four keypoints as corners of a square tile on the floor.
The goal is to warp this to get a top-down, birds-eye view of the floor. So we define a correspondence of four points of a square and warp accordingly.
Here's another example, this time using an example of a sidewalk near my house.
Now finally for the big reveal of part A! We'll combine the warped left image and the right image with $\alpha$-blending to create a mosaic!
Here are two additional mosaics I created from pictures along University Avenue and a mirror in my apartment!
The coolest thing I learned in this part was how applicable my linear algebra classes were. When I was taking the EE16-series, there was frequently heavy layers of abstraction behind the concepts we were taught. Some of the problems were genuinely interesting, but it always felt very isolated from the end result. With this project though, I clearly saw the usefulness of linear algebra as we used the homography matrix to warp our images.
After successfully stitching together our mosaic by hand-selecting the correspondents, we move into the more difficult task of automating this entire process using the findings from the Multi-Image Matching using Multi-Scale Oriented Patches paper and RANSAC.
First, we'll use corners_harris to detect the corner features within both our rooftop images. The deteected points are shown below.
This is far too many points to handle efficiently, so we can prune these down by using Adaptive Non-Maximal Suppression (ANMS) to select the top 500 points. The top 500 points for each image are shown below.
Now, we need a clear way to programmatically match points from the left image to the right image in orde to establish our homography matrix. For each of our points, will take the $40\times40$ window surrounding that point and resize it to an $8\times8$ patch to create a feature descriptor describing each point.
Here are the first three features we found.
Now, all the pieces are set in place for matching! We can use the dist2 function and the 1-nearest neighbor algorithm to compute correspondences between the left and right features. We search through distances that match our threshold (currently set to $0.35$), then add that point and its nearest neighbor on the other image to our correspondences list. Here are the correspondences we found for the rooftop photos.
Finally, we need our homography matrix that will warp our left image to our right image before blending. To do this, we use the Random Sample Consensus (RANSAC) algorithm described in class to randomly sample 4 points, calculate a homography from those sample points, and retain the largest set of "inliers" produced where each inlier point $p_i$ satisfies $\text{dist}(p'_i, Hp_i) < \epsilon$. Then, we calculate the homography from the inliers using our computeH function from part A. I repeated this over 1000 iterations with $\epsilon=0.5$.
Now comes the big reveal of part B! We warp our left image using the homography matrix and stitch together using our blend function from part A.
Comparing our autostiched rooftop with our previously manually stitched one from part A, we see a pretty similar result. And the autostitching required no work on our part!
The University avenue mosaic autostitching also works similarly well. It's hard to tell the difference between the manual and automatically stitched image.
The mirror is probably the most noticeable of the three. In the autostitched version, the line for the tilts off the vertical axis and becomes a bit strange. This lack of clear, defining corners in the pair mirror images compared to the last two pairs may have been the culprit.
I learned a lot of cool things throughout this project. I had never given much thought to how mosaics and panoramoics were made before, but this project really gave me insight into how it works behind the covers. It's amazing how far a simple SSD between feature descriptors can really go in terms of matching correspondences. I have much deeper appreciation for the software behind photography now.