To merge the photos together, I calculated a homography by first selecting corresponding points on each image.
Then, I calculated a homography matrix that warped the points from one image to the other using least squares.
I used more than 4 points such that there would be more equations and reduced noise by using least squares fit.
This was the equation set up.
By substituing in coordinates from one image for x and the other image for x', we can compute the homography by solving for h.
Set of correspondence points using to compute the homography matrix using least squares.
Warped image
A mosaic of the two images, with the warped image over the first view.
As you can see, there is more inconsistancy with stitching due to changing depth in the objects.
Homography + Blending
We can use laplacian pyramid blending from project 2 on two images to correctly blend high and low frequencies of images together.
View from my staircase with two vertical photos. Due to automatic white balance, the board becomes different colors.
Let's see how well blending deals with it.
The allignment does a good job however the blending completely fails.
This is because the black edges of the image get blended in and it creates an even sharper edge.
Photos of my background with a bike and random garbage.
The warped, merged, and merged + blended photos.
Once again, the blending creates an unnatural black border due to images having black padding around them.
Mosaics
More backyard photos with a bunch of random tables and objects to align to.
First I warped the leftmost photo, then joined it with the center.
Then I morphed the rightmost photo and once again joined it with the center to obtain the final image.
Here is another attempt in blending photos.
I improved the technique by increasing the sigma however the warp border remains around the top.
Automatic Image Stitching
Let's see how we can automatically stitch the above image using the material we learned in class.
Harris Interest Point Detector
I used the given implementation of the harris interest point detector to identify all of the possible edges and corners within an image.
This detector proved to be very sensitive, which meant that we had to use filtering to obtain a diverse set of corners to the run the computation on.
Adaptive Non-Max Supression
I used adaptive non-max supression to reduce the number of interest points.
Specifically, I used a radius of 80 and threshold of 0.9 when filtering out nearby points (if a points h value was less than 0.9 of the max within the radius, it out be filtered out)
This made the matching problem much more tractable and reduced false matches.
Feature Descriptor
I extracted 8x8 feature descriptors from each interest point above. I did so by extracting a 40x40 chunk around each point, blurring, and downsampling it into an 8x8 feature.
One downside was that a lot of image features were lost during downsampling (right example). When smoothing and downsampling it often lost a lot of the sharp edges.
Additionally, I normalized each feature descriptor by subtracting by its mean and dividing by the standard deviation of its pixels.
Feature Matching
I performed feature matching by calcuting the SSD error for one feature against all the features in the other image. I then sorted the SSD errors.
If the ratio of the smallest SSD error to the 2nd smallest SSD error was less than 0.4, then I labeled the features as matching.
Intuitively, if there was a feature that much almost twice as well as the next best feature, then it makes sense to call it a match those features match a lot better than the next best.
Robust Homography and Stitching
I used a 4-point RANSAC to compute a robust homography matrix. I took a random size 4 sample of the matching points, computed the homography matrix, and counted the number of inliers with threshold of 3.
The random sampling and fitting process was repeated 1000 times to get the best random sample.
On the above image, there were 13 matching points, and the best homography had 11 inliers and 2 outliers.
The homography stitching was much better than computing a homography against the entire 13 points.
More Mosaics
My hallway.
My yard with Rishabh Gupta's bike.
Outside of class. This stitching actually fails to find a robust homography.
The interest point matching algorithm only finds 5 matches, and RANSAC is only able to find 4 inliers (which means that no set of 4 points predict the 5th well.)
This could be amended by filtering out less points in the initial stages of the process.
What I learned
I learned that manual point homography is extremely costly from a human perspective.
Automatic homography estimation algorithms are able to find interest points in corners of images and find matches across images.
Using unstable least squares techniques also leads to highly variable homography matrix which can affect your results a lot.
I also learned that my blending code does not generalize to images with black borders. I hope to improve that in the future.