Part A

Pictures Taken

I took some pictures using my phone, in both portrait and landscape mode.

Recover Homographies / Warp the Images

Given a set of corresponding points, we'll have to find the homography matrix that will allow us to warp one image to another. The homography matrix has eight degress of freedom, so we need at least four pairs of points. If we have more pairs of points, we can find an even better homography matrix using least squares. To do so, we use the equation below, where (x1,y1) is our point in the first image and (x'1, y'1) is our point in the second image.

 Original Vase Image Vase Image to Warp to Warped Vase Image Original Tower Image Tower Image to Warp to Warped Tower Image

Image Rectification

Using the homography and warping, we can rectify images by passing in coordinate points of a rectangle. (Notice the chess image is distorted because I rectified to the chessboard, without accounting for the pieces).

 Original Book Rectified Book Original Book Rectified Book Original Chess Rectified Chess

Mosaic

Using the previous steps, I took two images, warped one onto the other, and then combined the two. By calculating the overlapping bounds, I was able to blend the overlapping area of the two images.

Part B

This part of the project follows the paper, Multi-Image Matching using Multi-Scale Oriented Patches.

Pictures Taken

 First Sofa Image Second Sofa Image

Harris Interest Point Detector

We were given the code for this portion, but in this step we are finding interest points using Harris Corner Detection.

 Harris Points on First Sofa Image Harris Points on Second Sofa Image

In this part, we want to restrict the number of generated interest points. So we use the following equation, where xi is our point of interest, and xj are the other points, and f(x) returns us the strength of the point.

 Points after ANMS on First Sofa Image Points after ANMS on Second Sofa Image

Feature Detection

To get the feature descriptor around our points of interest, we take a 40 by 40 patch around the point and then downsample it into a final 8 by 8 feature descriptor patch.

 Feature from First Sofa Image Feature from First Sofa Image Feature from First Sofa Image

Feature Matching

To match the features, we compare each point against the other points. We see whether the ratio of the norms of the two closest patches is below a certain ratio, and if such, there is a match.

 Points matched by feature on the First Sofa Image Points matched by feature on the Second Sofa Image

RANSAC & Results

To find the best set of points to compute the homography matrix for, we loop (in my code, about 500-1000 times) -- where in each loop, we randomly select four feature pairs, compute the homography matrix, and then compute the inliers of that transformation. Our goal is to find the homography that gives us the most inliers. Then we'll use those inliers to compute our final homography matrix we use to transform our images.

 Hand Warped Image of Sofa in Sitting Room Automatically Warped Image of Sofa in Sitting Room

Other Examples using this Process

Apartment Buildings

 First Apartment Image Second Apartment Image
 Harris Points on First Apartment Image Harris Points on Second Apartment Image Points after ANMS on First Apartment Image Points after ANMS on Second Apartment Image Feature from First Apartment Image Feature from Second Apartment Image Points matched by feature on the First Apartment Image Points matched by feature on the Second Apartment Image Hand Warped Image of Apartments Automatically Warped Image of Apartments