CS194-26 Project 5 Image Warping and Mosaicing

Daniel Lin

Initial Images

In this project, we are going to create panoramas and mosaics from individual snapshots. To do so, we first take two separate images of a scene from the same position. The images are not taken from the same angle, so there are some differences in the images, but there will usually also be some overlap. We then label some overlapping points between the images. I will demonstrate this primarily with two images taken from the balcony of my Las Vegas Airbnb. Below are the two photos and some associated points.
Balcony Left View
Balcony Right View
Balcony Left View With Keypoints
Balcony Right View With Keypoints

Homographies

Now, after manually labeling corresponding points, we want to create a homography, or a mapping between the first set of points to the second set of points. This way, we can transform the second image to have its corresponding points match up with the first image's points and can later overlay them to create a mosaic. To compute an exact homography, you only need 4 points, but to account for slight human error, I decdided to use more corresponding points and compute the least squares estimate. In order to ensure the invertibility of this matrix, we format it as so on the right side.
Base Homogrpahy Calculation between points
Least Squares Homography Estimate

Image Rectification

We then use this homography to forward warp images from one set of coordinates to another. To test out my warping, we go on a little detour where we try to rectify images. This means that we take an angled picture of an object and try to change the perspective of the picture so that the object is center aligned and the corners are axis aligned. To do this, we select the corresponding points of the original image to be 4 corners and the transform points to be some arbitrary rectangle; thus, the original image gets morphed so that the 4 corners match up with the rectangular points.
Original Image of Dictionary
Rectified Dictionary
Original Image of Monitor
Rectified Monitor

Blending Into Mosaic

We see that we can morph images from one set of points to the next. I first have to warp the second image so that the points match up with the first one. Below, I'll show the individual steps. We start by padding the images, warping the images, and then overlaying them.
Balcony Left Pad
Balcony Right Pad and Transform
Balcony Overlayed
We see that simply overlaying them, even with color matching, has the potential to create a big seam in the middle, making the image look unnatural. To fix this, I used a gradient mask to help stitch the two images together. The idea is, for example, if the panorama is left-right combination, we should weight the left image more closer to the left, and the right image more closer to the right so that the images look like they flow together. Below, I show the steps to create the two masks for each image that we later combine.
Balcony Raw Overlap Mask
Balcony Gradient Overlap Mask
We combine them each with the non overlapping mask for each individual picture to get. The right mask has the gradient in the opposite direction to signify the weighted averages when we add the two together.
Balcony Left Mask
Balcony Right Mask
Now, we combine the two masks with the two images to get the resulting picture. I tried to add Laplacian blending on top of this gradient blend to see if we can make this even more smooth, but I don't really see too much of a difference between the non-laplacian blended ones and the laplacian blended ones. Thus, to save some computation time, I blended the rest of the images without laplacian blending.
Final Balcony Mosaic
Final Balcony Mosaic with Laplacian Blending
Below are 2 more examples of Mosaics Created. The first is from a local high school (Irvine High) and the second is from a local park (Beacon Park)
Irvine High Left
Irvine High Right
Irvine High Mosaic
Beacon Park Left
Beacon Park Right
Beacon Park Mosaic

Automatic Stitching

Before, we had to manually select corresponding points, which is kind of annoying and less accurate. In the following parts, I will automatically stitch images through several parts.

Locate Harris Corners

We first have to locate corners to help us with detection. We use the Harris Corner Detector, which essentially looks through the gradients to find local maximas, which help detect corners. We first detect some of the harris corners on the balcony images.
Balcony Left Harris Corners
Balcony Right Harris Corners

Adaptive Non Maximal Suppression (ANMS)

We see that in the above pictures, the Harris Corner detection algorithm has a high threshold and accepts many corners. We then use adaptive non maximal suppression, which computes a fixed number of features per image, and it similar to k-means in that the algorithm finds the best N corners that encompass the most information.
Balcony Left Corners after ANMS
Balcony Right Corners after ANMS

Feature Detection

We now have the best corners, so we want to extrapolate some features from around the area. We take each corner and take a bounding box around this corner and extrapolate the features around it. We do this so that we can find features that match between two images. Below is an example of features found.
Example Features
Example Feature 1 (Blue Dot)
Example Feature 2 (Orange Dot)
Example Feature 3 (Green Dot)

Feature Matching

Now that we have the features from both separate images, we match the features. Essentially, we keep the corners whose features are within a small threshold of each other. Below are the matching corners.
Balcony Left Matching Corners
Balcony Right Matching Corners

RANSAC

Now that we have the set of matching corners from both images, we perform Random Sample Consensus (Ransac). To do this, we follow the loop below to compute the best homography. This homography matrix returned by RANSAC should be better than the one before.

Completing the Mosaic

We now have the points, and the homography matrix. We can proceed as we do above by warping the image and showing the result. Below are the three images above compared side by side.
Balcony Automatic Stitching
Balcony Manual Stitching
Irvine High Automatic Stitching
Irvine High Manual Stitching
Beacon Park Automatic Stitching
Beacon Park Manual Stitching
We see that overall, the automatic stitching produces sharper images and should be used instead of manual stitching. Some examples to verify this are the gray car in the balcony and the treehouse in beacon park.

Coolest Thing About Project

The coolest part of this project for me was the automatic stitching part. I didn't realize how we can simply find corners and compare features, and then we can match images. This makes it seem like we can match many panoramas well.