Shooting the Pictures

For this project, I used three pictures taken outside of Lewis Hall. These pictures were taken by rotating the camera from the same spot, so that the transforms between the image are projective. I used the exposure and focus locking (AE/EF) on iPhone so that the image/camera settings don't change between images.

Recover Homographies

Next, I recovered the parameters of the transformation between two images (an input and a target) at a time, using 16 corresponding points each. Using these points, I can recover a 3x3 homography matrix H, where p' = Hp. The matrix can be found via a linear system of n equations of the form Ah=b (where h is a vector holding the unknown entries of H). While the system can be solved using only 4 points, I use 16 correspondences, solving the overdetermined system with least-squares, to ensure a stable homography recovery.

Image Warping

Once I found the parameters of the homography, I used it to warp my images. First, I transformed the corners of the input image to find the bounds of the resulting warp, then expanded the output size to these bounds. I applied interpolated colors from the original image to the warped points.

Image Rectification

Once I was able to warp an image to another image's homography, I can apply the same warping method to "rectify" an image, where I warp planar surfaces to be frontal-parallel. This can be done by corresponding the image input points to a points fit to a plane/rectangle/square.

Here are some examples of some more extreme rectifications:

Mosaic

Now that I was able to individually warp images, I can create an image mosaic, where I warp an input image towards a target image using computed homography, then combine them together via multi-resolution blending to create a smooth, seamless blend.

This process can be repeated for any number of images to build upon the mosaic. Here is my mosaic continued with a third:

What I've Learned

This project allowed me to apply my previous understandings about image frequencies from proj 2 (for multi-resolution blending) and geometric warping from proj 3 (applied to warp to homographies). The most important/coolest part about this project was the demonstration of the various applications of using homography - my favorite part was rectification, which revealed interesting details about an image from different perspectives.

---

Auto Feature Matching

---

Detecting Corner Features

Harris Corners

I start by using Harris Interest Point Detector to find potential correspondences. Here are the results of using the harris point detector sample code without any thresholding/suppression:

Adaptive Non-Maximal Suppression

Now, I apply ANMS to select a fixed number (500) interest points that are spatially well distributed over the image (to allow for greater area of overlap between pairs of to-be-stitched images). The points chosen were those that had maximum corner strength relative to surrounding points.

Here are the interest points after ANMS is applied:

Extracting Feature Descriptors

In order to match interest points between different images, I extract feature descriptors for each interest point. This is done by selecting 40x40 patches from each point, downsampling it down to 8x8 windows, normalizing them to have a mean of 0 and a standard deviation of 1. These feature descriptors are then used in Feature Matching.

Feature Matching

Once I've extracted feature descriptors, I can use these to match corresponding interest points by comparing the feature points from image 1 with image 2 via SSD. However, I need to filter out the points that don't have a match with the other photo. I do this by taking the ratio between the smallest SSD (deemed the optimal match) and the second-smallest SSD (deemed the sub-optimal match). If this ratio (optimal / suboptimal) is small (lower than a threshold of 0.27), that means that the optimal match is much better than the suboptimal one, and we can use this pairing as a matched feature point.

RANSAC

Finally, to compute a good homography, I randomly select 4 feature pairs, compute the homography H with them, then compute the inliers that result from applying H on all of the points. Inliers are chosen based on whether the distance between the transformed points and the target points are less than a threshold epsilon (2 pixels). The homography that produces the largest set of inliers is our chosen homography.

Here is a demonstration of RANSAC finding the best homography after 1000 iterations, and the resulting point matches:

Results

Input images:

Input images:

Input images:

What I've learned

This project was a very cool experience, and learning how to automatically choose points was really interesting and fun to implement, especially since point selection was my least favorite part of working on part A. I thought that feature matching via Lowe's algorithm was a very simple and clever way to identify matching keys.