In this project we took pictures of the same scene from multiple angles and recovered a transformation between the two views so that we could stitch the images together into a single mosaic. This was done by identifying corresponding points between the two images and recovering a homography that transformed points in one coordinate frame to the other. Please see the code for this project for an example of how this homography was calculated.
For the first part of the project we "rectified" single images by identifying areas of the picture that we knew were rectangular in real life but did not appear to be so in the picture because of the perspective. We then warped these pictures into a coordinate frame where the object did appear rectangular, thus "rectifying" it to a rectangular frame. Two examples of this - a window and a poster - are included below
the rectified version, with the window now rectangular:
the rectified version, with the poster now rectangular:
For the next part we stitched together multiple images taken at different angles into a single coherent image. This was challenging because modern cameras automatically change the focus and exposure according to the environmental condiditons so it was difficult to get two images in which the lighting and other conditions matched up well. We partially ameliorated this issue by performing a blending of the images so that the seam between them did not appear as prominent but it did not always work correctly. Included below are the source images and resulting blended mosaics for three examples, showing that blending does not always work correctly.
For the next part of the project we wanted to extract matching points between the pictures and find a warping homography automatically, instead of manually defining point correspondences. To do this we used the harris corner feature to find features that potentially matched between the two pictures. We then used RANSAC to sample from the potentially matching features and compute a homography warping images in one coordinate frame to the other.
I used the provided sample code to extract harris corners from the images, with a sigma of 5 and the min distance set to 10. Inlcuded below is a picture of the raw harris corners for the left image in the plant mosaic above. Below that is an example of the harris corners after ANMS. Also included below that is the strongest harris corners in the same image. We can see that ANMS spreads the points better along all parts of the image, particularly visible on the sidewalk between the plants and the fence.
After pruning the number of harris corners down to a reasonable number I extracted 8x8 patches surrounding each corner and stored them for the left and right images in the mosaic.
I used the extracted features from each image and a similarity metric to find matching feature descriptors among the two images. To ensure that the features were a good match I used the thresholding technique described in the paper in the provided spec: for each feature descriptor I found the ratio of the similarity score between it and it's closest neighbor over the similarity score with the second closest neighbor. This ensured that the matches were "good" and were not a coincidence. Included below is a visualization of the matching features for the two images. (Thanks Ajay for the visualization code!)
After finding the matching coordinates between the images, we proceeded to estimate a homography between the two using RANSAC. In RANSAC we first choose four points from the matching sets, use them to estimate the homography, and then calculate the number of points that "agree" with this homography. This procedure is then repeated for a number of iterations, replacing the current estimate of the homography if it produces a mosaic with more pixels that "agree". In this way we can run RANSAC for a number of iterations and find a good homography with high probability even if some of the matching points are not actually matching
as can be seen when comparing this automatically produced mosaic with the one produced manually above, the RANSAC step clearly does not work well in my implementation. It seems to find a homography that puts the right picture in roughly the right place but a little off horizontally.
I greatly enjoyed this project, as it showed me how much processing goes on when I take a panorama with my phone. Although I wasn't able to get all of the parts working correctly I still liked understanding all the concepts and seeing the intermediate results. The coolest thing I learned from this project was definitely how to use a homography to warp the perspective of an image, which was used really well in the rectification step.