Introduction
If a picture is taken of a single object from 2 different perspectives, the projective transform between these two points of view can be calculated from the images. 4 correspondence points are needed to be able to recover this projective matrix. If we use more points, we can use linear least squares to get a better predicion of what the projection matrix H is. In this project we explore a few uses of this type of algorithm.
Image Rectification
If we want to see what a planar surface woud look like if we were looking at it strait on, we can rectify that surface into a square shape using the calculated matrix H. Here are two examples of that.
Building Mural
This building mural painting was taken in south Berkeley.
Original image.
Same image with the painting rectified
Cropped view of the painting
Video Game
This image was taken in my living room and shows an example of rotating in multiple axes.
Cover of a video game.
Same image with the cover rectified
Cropped view of the game
Mosaics
Using correspondance points, we can determine the projection from one image to another image in the same area. We can then project one of the images using H and blend the two images together using a linear blend to create a Mosaic Panorama.
Golden Gate Fields
These images were taken at the Golden Gate Fields horse race track.
Right image
Center image
Right image
Mosaic image from the above
Hearst Memorial Mining Building
These images were taken in the lobby of the HMMB building.
Right image
Center image
Right image
Mosaic image from the above
Cesar Chavez Park
These images were taken Cesar Chavez Park at the Berkeley Marina. I believe the quality of these is lower because it was outside and the plants and trees were moving slightly. Also there is a remarkable amount of detail on the ground, so any slight offset will be clear..
Right image
Center image
Right image
Mosaic image from the above
Feature Matching and Autostitching
We can auto stitch the mosaics by detecting feature points in the images, matching features across images, and computing the homography matrix H from the auto matched points. We use the Harris Corner Detector to find keypoints in the images. On average, there are several thousand corners detected in the images, so we need to find a way to filter them out.
Adaptive Non-Maximal Suppression
Adaptice Non-Maximal Suppression (ANMS) is an algorithm that can be used to get strong points that are spread out evenly in the image. The algorithm works by suppressing points based on the strength of the corner and only accepts those that are maximal in a small neighborhood. We can choose how many final points n that ultimately want. Several choices for n are shown below. An n of 400 was chosen for the rest of the project.
No suppression
n = 100
n = 400
n = 1000
Feature Descriptors and Matching
Corner features can be described by the pixels that surround it. We calculate a descriptor patch for each feature by looking at a subsampled 8x8 area around each corner. These descriptors are then compared to all other descriptors on the second image, and the best two matches are chosen. The ratio of the SSD of the top two matches is calculated and only points whose first match is significantly better than the second match are kept. Results from matching between two images from Golden Gate Fields are shown below.
Feature matching between the left and center images of the Golden Gate Fields mosaic.
RANSAC
As can be seen above, some features are matched incorrectly. We cannot estimate the homography matrix from these points because Linear Least Squares is not robust to outliers. Random Sample Consensus is an algorith that can efficiently reject outliers so that we only use good matches (inliers) to calculate H.
Results from RANSAC show a much better matching between features.
Auto-stitched Mosaics
Now that we can automatically detect a set of correspondence points between two images, we can return to the previous problem of creating mosaic panoramas. Results are shown below of 2 of the previous panoramas and also a new one.
Golden Gate Fields
Mosaic from hand picked points.
Automatically stitched mosaic.
Hearst Memorial Mining Building
Mosaic from hand picked points.
Automatically stitched mosaic.
Berkeley Institute of Design
These images were taken inside of the Berkeley Institute of Design lab.
Right image
Center image
Right image
Mosaic image from the above. There is no hand stitched mosaic for this scene.
Summary
The main thing that I learned from this project is that it can be hard to come up with a set of hyperparameters that work for all scenes. For each scene, I had to hand tune the parameters in some way to produce good results. I think some of these tuning procedures could be automated.