In this project, we compute homography matrices and use them to rectify images as well as create mosaics using multiple photos of the same scene. This part of the project focuses on warping the images into a mosaic using correspondence points found manually.
To compute the homography matrix, I find correspondance points in two images. I then use the equation p'=Hp, with p' being the points for one image and p being the corresponding points on the other, to find H.
To warp one image to another, I compute the homography matrix using their correspondance points. I find the size of the final image and create an empty image with just the image I'm warping to in the middle. I then do inverse warping for the other image by finding the locations on the new image where it will warp to and finding the corresponding points on the original.
Using the above technique, I can rectify an image by finding the corners of the object and setting its correspondance points to be the corners of the desired shape. I then find the homography matrix and warp the image to the desired shape.
Creating a mosaic of multiple images is similar to rectifying an image, with the difference being that instead of warping to a shape, I'm warping the image to another image. Now that I have both images placed appropriately, I can blend the images together to create the final mosaic. In the overlap region, I found the maximum pixel value between the two images and used that for the final mosaic to blend them together.
It is pretty amazing what we can do with multiple images and a bit of math. I think it's super cool that by just computing a homography matrix I can change the perspective of images as well as stitching images with different perspectives together into one mosaic.
As seen above, manually finding the correspondence points can lead to a homography matrix that is a little bit off, so the warped images do not fit perfectly together into a completely blended mosaic. This part of the project works to automatically find matching correspondence points given two images.
Using the Harris Interest Point Detector, I was able to automatically find the corner points of the images. As you can see below, there are many corner points detected.
We want to reduce the number of corner points for computational efficiency while keeping them well-distributed instead of focusing on a certain area. We can achieve this using Adaptive Non-Maximal Suppression. This allows us to select the top however many, in this case 500, points based on their minimum suppression radius:
Using ANMS also allows me to keep the points well-distributed across the image, as shown below.
Now that we have our points, we can extract feature descriptors for each of them. I got a 40x40 window around each of the points and downsampled them so they became 8x8. I then normalized them by subtracting the mean and dividing by the standard deviation.
Now I want to find the correspondence points, aka features that match in both images. I do this by computing the distances between each of the feature descriptors. I then find the nearest neighbor and the second nearest neighbor and compute the ratio of distance of first nn over the distance of the second nn. I then consider the feature pair if this ratio is under a certain threshold. Below are examples of the features that were matched together in this fashion.
As you can see, not all the features match up correctly, so in order to weed out those outliers, we use RANSAC to find the best possible set of feature matches. RANSAC involves iterating a large number of times and picking a random subset of 4 features and computing a homography matrix with those. I then compute the errors for all the features with this homography matrix and select those that are under a certain threshold. I count how many and keep track of the largest set of inliers. After all the iterations, I keep the largest set of inliers and use that as my final set of matching features and use them to compute the homography matrix used to warp the images. Below are the results after RANSAC.
I thought autodetecting the corners would be the hardest part of this project, but that was actually the easiest part. It was how we manipulated the corners to find matching features that took a while. However, the end result is cool and it's worth not having to manually find all the correspondence points.