Background:

This project is aimed to help us understand the basics of image warping and mosaicing.

Part 1: Shoot the Pictures

I chose to take a few pictures in VLSB libraries. This particular sequence has a few outstanding advantages: there are a lot of rectangular objects which could be used to align, and the color difference is obvious enough to tell the quality of the alignment.

q1.JPG

q2.JPG

q3.JPG

Part 2: Recover Homographies

Part 3: Warp the Images

I apply the forward Homography to the four corners to get the a sence of the location that the source image will be mapped to the destination image. Then I expend/shrink the window of the destination image to make sure the transformed source image fit snug. I record the offset I used, to help the mosaicing process in the subsequent section. I spent a lot of time on this part, mainly because Matlab uses relatively inconsistent indexing conventions that I was not fully aware of.

q1.jpg

q1.jpg warpped to q2.jpg

q3.jpg

q1.jpg warpped to q2.jpg

Part 4: Image Rectification

In this part, I selected a area of vent hole in q3.jpg as area of reference. The 9x9 vent hole area is supposed to be square. I map it to a 200x200 pixel square.

vent hole area

rectified q3, with respect to 9x9 vent hole

Part 5: Blend the images into a mosaic (manual point selection)

Mosaic without blending

Mosaic with alpha blending, alpha = 0.5

Part 6: Detecting corner features in an image

For this part I first ran the included Harris Corner Detector on the image

q1.jpg with Harris Corner Detector

q2.jpg with Harris Corner Detector

Then I follow the ANMS formula as descriped in the paper. To improve efficiency, I decided to manipulate the order of the formula, and calculate a threshold for each point x_i, and compute the minimum for all the points that meet the threshold. In the end, I sort the points by radius and took the top 500 for matching

q1.jpg with Harris Corner Detector

q2.jpg with Harris Corner Detector

Part 7: Extracting a Feature Descriptor for each feature point

For this part I first simply implement the 8x8 feature descriptor as mentioned in the paper, by finding a 40x40 window and downsampling with matlab's built-in indexing function.

q1.jpg feature at index 5

q2.jpg feature at index 5

q1.jpg feature at index 10

q2.jpg feature at index 10

q1.jpg feature at index 15

q2.jpg feature at index 15

Part 8: Matching these feature descriptors between two images

For this part I used the provided dist2.m file to measure the Euclidean distance between the pair features across two images. The metric applied was the ratio (Euclidean distance to the first best match)/(Euclidean distance to the second best match) AKA the Russian Granny Method. After some test, I fixed the cut threshold to be 0.3, which is slightly lower than 0.4 from the paper)

Mathced points from q1.jpg

Matched points from q2.jpg

Part 9: Use a robust method (RANSAC) to compute a homography

For this part I randomly select four points (out of roughly 30 points) at a time and compute the best fit Homography for these four points. Then I apply the found homography to all the top matched points, and count how many agree with the homography within Euclidean distance of one pixel. Then I repeat this process for 5000 times, and pick the homography with the most number of consents. For q1.jpg and Q2, I was able to find a Homography that more than 70% of the points consent, and the reqult indeed look very good.

q1.jpg

q1.jpg warpped to q2.jpg, resized for display

q2.jpg, for reference

Part 5: Auto Mosaic Results

Mosaic (Manual, from Part A)

Auto Mosaic 1: VLSB Library

Auto Mosaic 2: kitchen

Auto Mosaic 3: living room

Part 6: Tell us what you've learned

Matlab is tricky. X and Y coordinate can cause many troubles if one is not careful. In addition, selecting points that spread across the image will result in better homography. Last but not least, keeping track of extra information about offset when shifting is the way to go.

However, auto-mosaicing is so much cooler! No more hand selection needed! Wahoo!!!