I took some pictures using my phone, in both portrait and landscape mode.
Given a set of corresponding points, we'll have to find the homography matrix that will allow us to warp one image to another. The homography matrix has eight degress of freedom, so we need at least four pairs of points. If we have more pairs of
points, we can find an even better homography matrix using least squares. To do so, we use the equation below, where (x1,y1) is our point in the first image and (x'1, y'1) is our point in the second image.
Original Vase Image
|
Vase Image to Warp to
|
Warped Vase Image
|
Original Tower Image
|
Tower Image to Warp to
|
Warped Tower Image
|
Image Rectification
Using the homography and warping, we can rectify images by passing in coordinate points of a rectangle. (Notice the chess image is distorted because I rectified to the chessboard, without accounting for the pieces).
Original Book
|
Rectified Book
|
Original Book
|
Rectified Book
|
Original Chess
|
Rectified Chess
|
Mosaic
Using the previous steps, I took two images, warped one onto the other, and then combined the two. By calculating the overlapping bounds, I was able to blend the overlapping area of the two images.
Rectified Book
Part B
This part of the project follows the paper, Multi-Image Matching using Multi-Scale Oriented Patches.
Pictures Taken
First Sofa Image
|
Second Sofa Image
|
Harris Interest Point Detector
We were given the code for this portion, but in this step we are finding interest points using Harris Corner Detection.
Harris Points on First Sofa Image
|
Harris Points on Second Sofa Image
|
Adaptive Non-Maximal Suppression
In this part, we want to restrict the number of generated interest points. So we use the following equation, where xi is our point of interest, and xj are the other points, and f(x) returns us the strength of the point.
Points after ANMS on First Sofa Image
|
Points after ANMS on Second Sofa Image
|
Feature Detection
To get the feature descriptor around our points of interest, we take a 40 by 40 patch around the point and then downsample it into a final 8 by 8 feature descriptor patch.
Feature from First Sofa Image
|
Feature from First Sofa Image
|
Feature from First Sofa Image
|
Feature Matching
To match the features, we compare each point against the other points. We see whether the ratio of the norms of the two closest patches is below a certain ratio, and if such, there is a match.
Points matched by feature on the First Sofa Image
|
Points matched by feature on the Second Sofa Image
|
RANSAC & Results
To find the best set of points to compute the homography matrix for, we loop (in my code, about 500-1000 times) -- where in each loop, we randomly select four feature pairs, compute the homography matrix, and then compute the inliers of that
transformation. Our goal is to find the homography that gives us the most inliers. Then we'll use those inliers to compute our final homography matrix we use to transform our images.
Hand Warped Image of Sofa in Sitting Room
|
Automatically Warped Image of Sofa in Sitting Room
|
Other Examples using this Process
Apartment Buildings
First Apartment Image
|
Second Apartment Image
|
Harris Points on First Apartment Image
|
Harris Points on Second Apartment Image
|
Points after ANMS on First Apartment Image
|
Points after ANMS on Second Apartment Image
|
Feature from First Apartment Image
|
Feature from Second Apartment Image
|
Points matched by feature on the First Apartment Image
|
Points matched by feature on the Second Apartment Image
|
Hand Warped Image of Apartments
|
Automatically Warped Image of Apartments
|
Roads
First Road Image
|
Second Road Image
|
Harris Points on First Road Image
|
Harris Points on Second Road Image
|
Points after ANMS on First Road Image
|
Points after ANMS on Second Road Image
|
Feature from First Road Image
|
Feature from Second Road Image
|
Points matched by feature on the First Road Image
|
Points matched by feature on the Second Road Image
|
Hand Warped Image of Road
|
Automatically Warped Image of Road
|
Garages
First Garages Image
|
Second Garages Image
|
Harris Points on First Garages Image
|
Harris Points on Second Garages Image
|
Points after ANMS on First Garages Image
|
Points after ANMS on Second Garages Image
|
Feature from First Garages Image
|
Feature from Second Garages Image
|
Points matched by feature on the First Garages Image
|
Points matched by feature on the Second Garages Image
|
Hand Warped Image of Garages
|
Automatically Warped Image of Garages
|
What I've learned from Project 5
This was a really interesting project! While there was a lot of time spent debugging (especially indexing, as I tried to use a mix of regular lists and numpy arrays at first), I really enjoyed getting to see how all this warped together. I think the coolest thing I learned was Lowe's threshold for finding feature matching, as well as getting to implement RANSAC.
