In this project, we are going to create panoramas and mosaics from individual snapshots. To do so, we first take two separate images of a scene from the same position. The images are not taken from the same angle, so there are some differences in the images, but there will usually also be some overlap. We then label some overlapping points between the images. I will demonstrate this primarily with two images taken from the balcony of my Las Vegas Airbnb. Below are the two photos and some associated points.
Now, after manually labeling corresponding points, we want to create a homography, or a mapping between the first set of points to the second set of points. This way, we can transform the second image to have its corresponding points match up with the first image's points and can later overlay them to create a mosaic. To compute an exact homography, you only need 4 points, but to account for slight human error, I decdided to use more corresponding points and compute the least squares estimate. In order to ensure the invertibility of this matrix, we format it as so on the right side.
We then use this homography to forward warp images from one set of coordinates to another. To test out my warping, we go on a little detour where we try to rectify images. This means that we take an angled picture of an object and try to change the perspective of the picture so that the object is center aligned and the corners are axis aligned. To do this, we select the corresponding points of the original image to be 4 corners and the transform points to be some arbitrary rectangle; thus, the original image gets morphed so that the 4 corners match up with the rectangular points.
Blending Into Mosaic
We see that we can morph images from one set of points to the next. I first have to warp the second image so that the points match up with the first one. Below, I'll show the individual steps. We start by padding the images, warping the images, and then overlaying them.
We see that simply overlaying them, even with color matching, has the potential to create a big seam in the middle, making the image look unnatural. To fix this, I used a gradient mask to help stitch the two images together. The idea is, for example, if the panorama is left-right combination, we should weight the left image more closer to the left, and the right image more closer to the right so that the images look like they flow together. Below, I show the steps to create the two masks for each image that we later combine.
We combine them each with the non overlapping mask for each individual picture to get. The right mask has the gradient in the opposite direction to signify the weighted averages when we add the two together.
Now, we combine the two masks with the two images to get the resulting picture. I tried to add Laplacian blending on top of this gradient blend to see if we can make this even more smooth, but I don't really see too much of a difference between the non-laplacian blended ones and the laplacian blended ones. Thus, to save some computation time, I blended the rest of the images without laplacian blending.
Below are 2 more examples of Mosaics Created. The first is from a local high school (Irvine High) and the second is from a local park (Beacon Park)
Before, we had to manually select corresponding points, which is kind of annoying and less accurate. In the following parts, I will automatically stitch images through several parts.
Locate Harris Corners
We first have to locate corners to help us with detection. We use the Harris Corner Detector, which essentially looks through the gradients to find local maximas, which help detect corners. We first detect some of the harris corners on the balcony images.
Adaptive Non Maximal Suppression (ANMS)
We see that in the above pictures, the Harris Corner detection algorithm has a high threshold and accepts many corners. We then use adaptive non maximal suppression, which computes a fixed number of features per image, and it similar to k-means in that the algorithm finds the best N corners that encompass the most information.
We now have the best corners, so we want to extrapolate some features from around the area. We take each corner and take a bounding box around this corner and extrapolate the features around it. We do this so that we can find features that match between two images. Below is an example of features found.
Now that we have the features from both separate images, we match the features. Essentially, we keep the corners whose features are within a small threshold of each other. Below are the matching corners.
Now that we have the set of matching corners from both images, we perform Random Sample Consensus (Ransac). To do this, we follow the loop below to compute the best homography. This homography matrix returned by RANSAC should be better than the one before.
Completing the Mosaic
We now have the points, and the homography matrix. We can proceed as we do above by warping the image and showing the result. Below are the three images above compared side by side.
We see that overall, the automatic stitching produces sharper images and should be used instead of manual stitching. Some examples to verify this are the gray car in the balcony and the treehouse in beacon park.
Coolest Thing About Project
The coolest part of this project for me was the automatic stitching part. I didn't realize how we can simply find corners and compare features, and then we can match images. This makes it seem like we can match many panoramas well.