Shooting The Pictures, Recovering Homographies
As the project mentions, it's best to take photos that have a lot of overlap. It's also to important to take images that simply rotate the center of projection, so that outer images aren't heavily warped.
I used ginput for users to define common points. Attached below is the user interface that users can define the points needed. Users would click a point in the left image, then click the corresponding point in the right image.
Below are the points for the example warps. Notice how the points are at corners, and line up between the images.
Warping Images (and Rectifying them)
Using the points defined above, we can use least squares to compute a good matrix to compute a projective transformation of the points.
To do this, I set up a matrix to determine what the 8 unknown value of the H matrix were–we use least squares so that we can use the data from more than 4 correspondences, and so that we're resistant to error.
Below are two examples of warped images for panoramas
We can also use this method to "rectify" images. This includes transforming images of the sides of buildings to make it look like the photo was taken edge-on.
Below you can see some examples of the images as well as the GUI built for the warping. Note: the recified images are cropped portions of the full image.
Stitching and Blending Panoramas
Bringing everything together
Time to stitch the images together! I took the approach to add the images together, blend the overlap, and then warp the combined image to the third image's points. I did linear alpha blending to blend.
Overall strategy: image -> warp -> add next image and blend -> warp -> add next image and blend
For blending, I calculated where the overlap between the two images started to happen, then I progressively increased alpha for the new image, and decreased alpha to 0 for the warped image.
Below you can see the original images, and the final panorama.
Harris Corner Detector
I used a min_distance of 3. As you can see in the example images, the harris corner detector detected lots of corners–way too many corners to actually use!
Feature Descriptor Exctraction and Matching
We run through all the points found above and extract the 40x40 grid that surrounds it, then rescale down to a 8x8 grid. We then normalize by subtracting the mean and dividing by the standard deviation
Below you can see what these (non-normalized) windows look like
We can now use these "windows" to compare points. For each window, we compare to all other windows and find the windows that match the best (using a simple SSD metric). If the closest neighbor in the other image is much better than the second closest match, we call the closest neighbor and the current point "matches"
Below you can see of matches between two images that are a part of a panorama.
RANSAC
Obviously, the result above isn't great. We see multiple "matches" that aren't true, or that aren't true matches. (above, you can see that the street curb looks the same at multiple points)
Hence, we use RANSAC to choose a subset of matches. We randomly choose a set of four points, compute the transformation matrix, and see how of our existing points fit withing that matrix. We do this many times, and choose a final set of points that yields the most number of "inliers"
This is quite good! We save these points to a file and use them to stitch the panorama.
Examples!
Automatically found correspondences
Manual Alignment | Automatic Alignment
Automatically found correspondences
Manual Alignment | Automatic Alignment
Automatically found correspondences
Manual Alignment | Automatic Alignment
Reflections
This project was cool! I was very surprised at how good RANSAC was at finding good points, and how useful it is. I learned a lot about how to take photos in order to stitch them, as well as how to do projective transformations for a certain goal.
Reflections
This project was cool! I was surprised at how simple it is to warp images–all you need is four points, and you can compute transformations between two images. The coolest thing about this project was the image warping. I found the actual positioning of images, and the rectifying of images, to be a hassle.