Project 5: [Auto]Stitching and Photo Mosaics
Aivant Goyal
Part 1
This part is about manually stitching together two images to form a mosaic.
Computing Homography
First, we must write an algorithm to compute a homography: a perspective warp matrix that turns one set of points to another. A Homography matrix has 8 degrees of freedom (with a 1 in the bottom right corner).
We can solve for an equation with 4 correspondence points, but it’s best to use more and use Least Squares to solve (in order to reduce noise).
I used the following formula (taken from http://www.cse.psu.edu/~rtc12/CSE486/lecture16.pdf)
Rectification
As a sort of sanity check for our homography, we test it out on an image taken of a book. With our homography, we can cast the picture of the book from it’s current position to a birds-eye view!
We do this by performing an inverse warp from the desired plane to the original image. In rectification, we set manual bounds for the warped image.
Before:
After:
Mosaic
Now, we can take two images and try to stitch them together like a mosaic. The goal here is to warp the left image onto the plane of the right image. My images were pictures of my bedroom, with the correspondence points shown:
| Left |
Right |
 |
 |
First, we send the corners of our original image through the homography, to get a sense for the warped shape. Then we use an inverse warp to fill in that shape with points from the original image. This uses the same code as rectification, just with warped bounds.
When I warped my left image, it turned out like this:
Putting it side by side with the right image, we can see that they definitely are in the same plane. However, there was an issue with the bounds, which made it difficult to properly stitch it together into a mosaic.
Conclusion
While I couldn’t finish part 1 to it’s completion, I thought the rectification and general warping calculations were really cool. It’s crazy to think that we can use rectification to view a lot more details in images that we are able to otherwise
Part 2
This part of the project was focused on doing part 1 automatically. This involved using a Harris corner detector algorithm to find feature points, finding the most relevant features, matching feature points across pictures, and then using those to calculate the homography from part 1.
I wasn’t able to complete this part due to some difficulties with numpy, stress and lack of time. But I still found the process incredibly interesting and am glad I studied it!
The algorithm is adapted from this paper: “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al."
Step 1: Harris Corner Detection
First we used the classic Harris Corner Detection algorithm on a scaled down, black and white version of the image. Before we scaled the image down, this led to 200,000+ feature points. Scaled down, we could work with ~4500. Either way, there was a lot of noise:
| Left |
Right |
 |
 |
Step 2: Adaptive Non-Maximal Suppression
In order to find the 500 most relevant corners, we use an algorithm called “Adaptive Non-Maximal Suppression” which selects the top 500 points with the highest suppresion radius (minimum distance to nearest higher-value corner). The idea is to find the most distinguishing and distributed features in the image.
There is less noise, and we can notice some distinct features like the corners around the light switch. That said, the feature detector didn’t seem to detect a lot of the more obvious corners on the bed, which was a little confusing to me
| Left |
Right |
 |
 |
Step 3: Matching Features
The next step is to turn each corner into a “feature” by expanding out into an 40x40 square and downsampling. We can then compare features across images and try to find meaningful correspondence points. I display 10 features from each image here
Left Image Feature Sample
Right Image Feature Sample
Conclusion
This is as far as I got in this project. While I was able to extract these features, I kept having trouble trying to match them in order compute a useful homography. Given more time, I would have fixed my issues in part 1 and hopefully seen the project through to completion by implementing the RANSAC algorithm and completing the stitching process.
Regardless of not finishing, I think the thing I most enjoyed about this project was understanding the power of an image warp and how it can be used to imagine an image in different geometrical spaces!
Project 5: [Auto]Stitching and Photo Mosaics
Aivant Goyal
Part 1
This part is about manually stitching together two images to form a mosaic.
Computing Homography
First, we must write an algorithm to compute a homography: a perspective warp matrix that turns one set of points to another. A Homography matrix has 8 degrees of freedom (with a 1 in the bottom right corner).
We can solve for an equation with 4 correspondence points, but it’s best to use more and use Least Squares to solve (in order to reduce noise).
I used the following formula (taken from http://www.cse.psu.edu/~rtc12/CSE486/lecture16.pdf)
Rectification
As a sort of sanity check for our homography, we test it out on an image taken of a book. With our homography, we can cast the picture of the book from it’s current position to a birds-eye view!
We do this by performing an inverse warp from the desired plane to the original image. In rectification, we set manual bounds for the warped image.
Before:
After:
Mosaic
Now, we can take two images and try to stitch them together like a mosaic. The goal here is to warp the left image onto the plane of the right image. My images were pictures of my bedroom, with the correspondence points shown:
First, we send the corners of our original image through the homography, to get a sense for the warped shape. Then we use an inverse warp to fill in that shape with points from the original image. This uses the same code as rectification, just with warped bounds.
When I warped my left image, it turned out like this:
Putting it side by side with the right image, we can see that they definitely are in the same plane. However, there was an issue with the bounds, which made it difficult to properly stitch it together into a mosaic.
Conclusion
While I couldn’t finish part 1 to it’s completion, I thought the rectification and general warping calculations were really cool. It’s crazy to think that we can use rectification to view a lot more details in images that we are able to otherwise
Part 2
This part of the project was focused on doing part 1 automatically. This involved using a Harris corner detector algorithm to find feature points, finding the most relevant features, matching feature points across pictures, and then using those to calculate the homography from part 1.
I wasn’t able to complete this part due to some difficulties with numpy, stress and lack of time. But I still found the process incredibly interesting and am glad I studied it!
The algorithm is adapted from this paper: “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al."
Step 1: Harris Corner Detection
First we used the classic Harris Corner Detection algorithm on a scaled down, black and white version of the image. Before we scaled the image down, this led to 200,000+ feature points. Scaled down, we could work with ~4500. Either way, there was a lot of noise:
Step 2: Adaptive Non-Maximal Suppression
In order to find the 500 most relevant corners, we use an algorithm called “Adaptive Non-Maximal Suppression” which selects the top 500 points with the highest suppresion radius (minimum distance to nearest higher-value corner). The idea is to find the most distinguishing and distributed features in the image.
There is less noise, and we can notice some distinct features like the corners around the light switch. That said, the feature detector didn’t seem to detect a lot of the more obvious corners on the bed, which was a little confusing to me
Step 3: Matching Features
The next step is to turn each corner into a “feature” by expanding out into an 40x40 square and downsampling. We can then compare features across images and try to find meaningful correspondence points. I display 10 features from each image here
Left Image Feature Sample
Right Image Feature Sample
Conclusion
This is as far as I got in this project. While I was able to extract these features, I kept having trouble trying to match them in order compute a useful homography. Given more time, I would have fixed my issues in part 1 and hopefully seen the project through to completion by implementing the RANSAC algorithm and completing the stitching process.
Regardless of not finishing, I think the thing I most enjoyed about this project was understanding the power of an image warp and how it can be used to imagine an image in different geometrical spaces!