Overview

---

In this project, we learned to use points of correspondence (both manually and automatically selected) to auto-stitch panoramas.

Recovering homographies

---

I used this matrix equation to solve for the homography between photos.

Rectification

---

In this part of the project, I manually selected the corners of a rectangular object and solved for the homography that mapped the corners of a perspective view of the object to a frontal-parallel view of the object.

Blending Images Into a Mosaic

---

Manual Registration

I started by manually picking points of correspondence between the two images. This ensured I was always picking matching features, since I actively picked the points as accurately as I could. Next, I solved for a homography that related the two different views, which, after applied, would allow me to stitch together the two images.I was able to get decent results only picking 4 points per picture, but I later realized that more points (or just more accurate points) would make a dramatic difference in the quality of the result. I blended the images using a Gaussian filtered binary mask.

Automatic Registration

To begin automatic registration, I used code provided by Prof. Efros to identify Harris corners in just one color channel - that is, areas in the image where moving in any direction would cause a large change in intensity. In the example below, we can see that there are many "corners" in the parts of the image with the building, and very few detected corners in the sky.
Next, I implemented Adaptive Non-Maximal Suppression from the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al. The idea of this algorithm is that we can sort corners by greatest distance to a corner with a greater corner strength. Then, we can take the top n points, which means we take the points that are local maxima within a certain radius. Essentially, ANMS allows us to identify strong and maximally spaced corners. Here are the results of applying ANMS to the Harris corners shown above.
Once I had sufficiently spaced corners, I matched features between the images by taking a 40x40 square of pixels around the ANMS corners and downsampling and normalizing this square to 8x8 pixels to form a descriptor for the corner. Here, I calculated a value based on the similarity of the best match compared to the second best match for the feature. If the best match (in the right side image) for a feature in the left image was sufficiently better than the second best match, the feature was kept. Lastly, I ran RANSAC on the discovered features to calculate the homography that "pleased" the most correspondences. Here are the resulting correspondences after RANSAC.
The automatic registration process produced a homography that I then applied to the image on the left. I blended the images using the same process that I did for the manually aligned images (using a binary mask smoothed with a Gaussian filter). The result was actually much better than my manually picked result!

Results

---

Haas School of Business

---

International House

---

East Asian Library

---

Plaza in front of Memorial Stadium

Summary

---

We have implemented a way to automatically warp images with the same center of projection to a common projective plane and stitch together panoramas. The most interesting part of the project to me is how much better the automatic alignment was than my manually defined points. I also liked how simple and robust the RANSAC algorithm was for finding the best homography. It is super fast and a very clever way to solve the problem.