Here are all the pictures I took for the project:
Photo of a window and restaurant board that I will rectify into a frontal view
Photos of my living room that are used for a panorama
Photos of outside of Haas Pavilion that are used for a panorama
Photos of my kitchen that are used for a panorama
In order to recover homographies we need to set up atleast 4 correspondences since the perspective warping matrix has 8 degrees of freedom, but to reduce noise we can use least squares if we find more correspondences. Lets define the homography matrix as [[a,b,c], [d,e,f], [g,h,1]]. By multiplying it by [x,y,1] we can find [z*x_prime, z *y_prime, z]. When we divide by z we can obtain the points after they are transformed by the homography. We want to the get the problem in the form A * [homography parameters] = [warped_xCoords, warped_yCoords]. We can expand the original equation and get the following equations: d*x + e*y + f = z * y_prime and a*x + b*y + c = z * x_prime. We can simplify w to g*x + h*y + 1. Thus giving us: d*x + e*y + f - g*x*y_prime - h*y*y_prime = y_prime and a*x + b*y + c - g*x*x_prime - h*y*x_prime = x_prime. We can use these 2 equations to construct our A matrix and recover the homography through least squares. If we have 8 correspondences for example we will get 4 equations in each of the forms shown in the previous sentence.
With the recovered homography we can now warp the perspective of an image. I first found all the coordinates of the destination image (where I will warp to) and then used the inverse homography to get the coordinates in the source image to interpolate color from. I warped the corners first to check if I need to increase the size of the destination image and used translation shifts in case parts of the picture were in negative coordinates.
Below are the results of warping planar objects into their frontql view, they originally start out slanted
Here I frontal warped the a window in my apartment.
Here I frontal warped the a restaurant board.
Instead of a frontal view we can wrap one image into the perspective of another to form a panorama. We can average the points that overlap to blend them together. Below are three of my results.
Panorama to the side of the Haas Pavillion.
Panorama of my kitchen.
Panorama of my living room.
We can detect corners automatically by looking for points in the image where there is both large change in X and Y. I set the threshold to 0.2 so I could focus my search on points with stronger corner strengths. Below are the harris corners detected for the two images I am trying to automatically match.
In the previous step found thousands of harris corners, but we want to limit that to 500 before we start seaching for correspondences to save computational power. Instead of just picking the 500 strongest corners, we can use ANMS to find 500 corners that are well distributed through the image. We keep a list of selected points and we compare unselected corners and see the distance to the closest selected corner that would suppress them. If the distance is less that the current radius we add it to the list. At the end of every iteration the radius threshold will decrease. Below are the 500 selected points in both images.
Since we now have our 500 selected corners from each image we can match corners from both the images. From each corner we can extract a 40 by 40 patch, down sample to 8 by 8, and then normalize. I used SSD as my metric to measure similarity in patches and used the Lowes algorithmn. We compare the SSD to best match and the second best match. The more different these two scores are the more likely it is a proper match. I used the threshold that SSD best match / SSD second best match < 0.6. Below are the selected lowes corners plotted on each image. Most selected corners exist in the overlapping area, but there are still some outliers.
To get rid of the outliers we can use RANSAC. We recover homographies using 4 random corners and check which of the corresponding points match. The set of points that agree the most will be used as the final set of inliers. I used 10,000 iterations in my implimentation of the RANSAC algorithmn. Below are the set of inliers that were found in the image. All these points exist in the overlapping region.
We can use the correspondences found in RANSAC to recover a more refined homography using least squares. Below are the 3 panoramas built using 8 manual correspondences and 3 automated panoramas. The first image in each pair is the manual one and the second is the automated one. The kitchen one improved the most as it much less blurry now in overlapping regions.
Panoramas of Haas Pavillion
Panorama of my living room
Panoramas of my kitchen
The coolest thing I learned is the RANSAC algorithmn. It feels so intuitive and it also very effective in removing all the outlier corresponcdeces that were selected by Lowes alogirthmn.