In this project, I will take two or more photographs and create an image mosaic by registering, projective warping, resampling, and compositing them. Along the way, I will learn how to compute homographies, and how to use them to warp images.
I shot pairs of photos of the same things and with the different points of view. I overlapped about 40% of the images to make further steps easier. Also, I rotated the camera without changing position.
| Image 1 | Image 2 |
|---|---|
|
|
H is the projective transformation matrix here. The degree of freedom for H is 8. What we need to do now is to project one picure into the other, and H matrix can help. One way of solving this problem is by solving the least square problem below.
 |
Here is an example of defining homography with 12 points in each picture.
| Homo-pic1 | Homo-pic2 |
|---|---|
 |
 |
In order to warp the images, we use inverse warping here. The main idea is to get coordinates from skimage, and apply inverse warping to all coordinates. For each pixel point in the second image, the algorithm find the corresponding pixel value in the first image.
Below are three of my rectified images. They are bottom-to-middle, left-to-middle, and up-to-middle respectively, which shows that my rectification is working well.
| Original Image | Output Image |
|---|---|
 |
 |
 |
 |
 |
 |
This is a way further than the rectification part. I warped both images and put them together on a canvas. For overlapping, I calculated the average of intensity from corresponding pixels in both images. Below are three examples.
| Image 1 | Image 2 | Mosaic |
|---|---|---|
|
|
 |
 |
 |
 |
 |
 |
 |
Part A: With homography as the connector, everything is connected with each other.
The goals of the second part is to create a system for automatically stitching images into a mosaic. A secondary goal is to learn how to read and implement a research paper. The paper is "Multi-Image Matching using Multi-Scale Oriented Patches" from https://inst.eecs.berkeley.edu/~cs194-26/fa20/hw/proj5/Papers/MOPS.pdf.
I use Harris Corner Detector to find all seemingly likely corners in the images. the results are shown below:
| Original Image | Harris Point Detector |
|---|---|
|
 |
|
 |
Harris produce too many points than we actually needed. Therefore, I want to make sure that fixed amount of points are produced for each picture and they are spreading out evenly. Therefore, we implement ANMS here, we first calculate the min_suppression_radius for each corner point, and set certain number of points and certain robust buffer. The results are shown below:
| Original Image | ANMS |
|---|---|
|
 |
|
 |
In order to generate patches around each feature, we first need to find the 40 x 40 square of pixels around each coordinate. After that, I downsample the square to an 8 x 8 patch. We also calculate the SSD between each pairs of coordinates. In order to match features, I keep the pair if its nearest neighbor devided by distance to its second nearest neighbot is smaller than some threshold. The result is the following.
| Original Image | Feature Patch | Features Matched |
|---|---|---|
|
 |
 |
|
 |
 |
With so many matching points, we need to identify the good matching and the bad ones. Also, we need to produce homography from the good matching. By using RANSAC, I get a homography based on four randomly chosen pairs, and calculate the Euclidian distance between projected points and the correspondence. SSD = 1 is set as the cutoff. By iterating many times and compute homography on the largest set of inliner, we get the final auto-stitching. Part 5 are the results.
| RANSAC1 | RANSAC2 |
|---|---|
 |
 |
| Image 1 | Image 2 | Manually Stitched | Auto Stitched |
|---|---|---|---|
|
|
|
 |
|
|
|
 |
|
|
|
 |
It is obvious that the manual results with 12 predetermined points and the auto-stitched results are pretty alike and almost identical. The auto-stitched ones seems a little bit smoother. For such reason, the implementation is working correctly.
Part B: It is important to filtered out the outliers and focus on the most important things.