Dining Room Image
Rectified Image
Lobby Image
Rectified Image
In this project, I shot and digitized photos, recovered homographies, warped the phots, and blended them into mosiacs using linear blending.
After shooting and digitizing a set of photos, I tried to figure out a way to recover a homography matrix between two sets of point correspondences. To do this, I take Hp = p' where p = [x, y, 1] and p' = [wx', wy', w] and convert it to Ah = b where h is the 8 variables in H and b = [x', y']. I add more rows to A and b depending on the number of point correspondences we are using to compute a Homography. Then, I use least squares to get the 8 variables in h and convert into a 3x3 H matrix. I then use the inverse of H to warp my images. Shown below, I have "rectified" two images by warping them so that certain planes are frontal-parallel. To do this, I chose a set of points on the original image and warped them to a set of points that I manually set. For the first set of images, I rectified a dining room image so that the purse is in the front view of the picture. For the second set of images, I rectified a lobby image so that the fire alarm box is in the front view of the picture.
|
Dining Room Image |
Rectified Image |
|
Lobby Image |
Rectified Image |
For making mosaics, I took my first image and placed it into a bigger black image. I then warped my second image to this bigger black image. To combine this warped image with the bigger image, I tried to use linear blending. I created a mask that fell off linearly from 1 to 0 in the overlap region between the bigger image and my second image. This seemed to do a good job getting rid of some but not all artifacts. The results are shown below for three mosaics. The first is a wall in my apartment's living room. The second is an intersection outside my apartment. The third is the lobby of my parents' house in Saratoga.
Left View Wall |
Right View Wall |
Wall Mosaic |
Left View Intersection |
Right View Intersection |
Intersection Mosaic |
|
Left View Lobby |
Right View Lobby |
Lobby Mosaic |
From this project, I learned that image warping and blending can create seamless-looking panoramas. In general, it is cool to see how a mix of processing techniques can create illusions that would potentially fool the average person. I am excited for the next phase of the project, where we do not have to manually click points to create mosaics.
For this part of the project, I implemented a system that automatically stiches images together into a mosaic without needing to manually establish point correspondences. First, I used a harris point detector to detect a set of potential corner features. Then, I implemented an Adaptive Non-Maximal Suppression algorithm to get a smaller set of evenly spaced corner features. From there, I extracted a 40x40 window around each corner feature and downsampled to make an 8x8 feature descriptor. Then, I implemented feature matching with the patches and ran a RANSAC algorithm to compute a Homography over matching points, which I used to make my mosaics.
First I ran the Harris Corner Detecting algorithm from harris.py on my images to find a series of potential corner points that could be used for feature matching. Below are examples of harris corners overlaid on some of my images.
Right View Lobby With Harris Corners |
Left View Lobby With Harris Corners |
After getting a series of harris corners, I implemented an Adaptive Non-Maximal Suppression Algorithm that picks out a series of harris corners with even spacing throughout the image. To do this, I calculated the min distance between each harris corner h and any other harris corner h' where Crobust * the corner strength of h' is greather than the corner strength of h. I set Crobust equal to .9 as in the given research paper to get an even spacing. Then, I returned the 250 harris points with the minimum distances. Below are examples of corners chosen by ANMS for some of my images.
Right View Lobby With ANMS Corners |
Left View Lobby With ANMS Corners |
After getting a series of points from ANMS, I take a 40x40 window around each point and imresize this window down to an 8x8 patch, which involves blurring the 40x40 window and sampling every fifth pixel. From there, I bias and gain normalize the patch by subtracting the mean and dividing by the standard deviation. Then, I reshape this 8x8 patch into an array of 64 elements. This makes a 250x64 matrix for all the features in a given image. Given, two images that I want to combine and their feature matrices, I implement Feature Matching. To do this, I go through every feature in one image, which is represented by going through the rows in the 250x64 matrix, and find the nearest neighbor and second nearest neighbor from the features in the other image. To eliminate false matches, I only accept nearest neighbor matches between the two images if the ratio of the distance to the nearest neighbor and the distance to the second nearest neighbor is less than 0.4. This gives us the highest number of good feature matches between the two images. However, there will still be some false positives, so I have to implement RANSAC to compute proper homographies for the mosaics.
For RANSAC, I use 5000 iterations, and in each iteration, I select 4 points from one image and take the corresponding points from the other image. I compute a Homography and use this Homography to tranform all the points in the first image and see the number transformed points that are close to the corresponding points in the second image within a certain threshold distance. I used a distance of 1 for my threshold. I call these points inliers, and I keep track of the highest number of inliers throughout the 5000 iterations. Then, I take the highest number of inlying points after the iterations have run and calculate a final Homography matrix between these inliers and their corresponding points in the second image.
With these homographies from RANSAC, I can automatically stitch together mosaics that I had previously manually stitched together. I show the results of manual vs automatic stitching below for all three of my mosaics. The automatically stitched mosaics seem to be blended together more smoothly.
|
Wall Mosaic Manually Stitched |
Wall Mosaic Automatically Stitched |
|
Intersection Mosaic Manually Stitched |
Intersection Mosaic Automatically Stitched |
|
Lobby Mosaic Manually Stitched |
Lobby Mosaic Automatically Stitched |
From this part of the project, I learned that you can use feature descriptors to try and match points in two images. It is cool that you can use properties in images that would be invariant to transformation in order to connect similar images together automatically.