Overview

First, manually pick corresponding points for pictures. Then use least square solution to solve and find the homography H. Use inverse warping similar as the one in proj3, warp the pictures to desired appearance.

After building this Ah = b equation, solving using least square solution where h = (A^TA)^-1A^Tb

Rectify

Mosaicing

Takeaway

Part B

1. Detect Corner Features

In part A, we define correspondence between pictures manually by hand. Here in part B, the correspondence is found automatically.

First, we get the corner_strength of a picture through function corner_harris. We consider points with corner strength that is a local maximum in a 3*3 square and above chosen threshold as interest points. Here marked by red points in the graph. We can see that interest points now are spatially unevenly distributed with many point squashed together.

Then we use Adaptive Non-Maximal Suppression to make the interest points more spatially evenly distributed. For every interest point pi(red), find minimum radius of pi to every other interest point pj which satisfy that corner strength of pi is smaller than c_robust * corner strength of pj. Then select new interest point in the order of max minimum radius.(marked in blue)

2. Extracting Feature Descriptors

For every interest points(blue), we extract a 40*40 pixel square and downscale it to 8*8 descriptor patch.

3. Matching Feature Descriptors

Calculate the ssd between every feature descriptors in two pictures, find the smallest ssd(1_NN), and second smallest ssd(2_NN), and find the ratio 1_NN/2_NN. If the ratio if smaller than 0.4, then set feature correspondence.

Here we see blue interest points and white matching features:

The correspondence is really good.

4. RANSAC

After having correspondence from step3, we use RANSAC to estimate the homography H.

1. We randomly choose 4 feature pairs and compute the exact homography H_4 based on the four.

2. Using H_4 to computer inliers, where dist(p, Hp) < epsilon

3. keep largest set of inliers

4. Calculate least-square estimate for all inliers

After having the RANSAC estimated H, repeat the same process of warping and mosaicing as in part A, and get following results:

Takeaway

The automatically found correspondences are really good and fast! It saves lots of time manually picking and selecting points. And when it is hard for human to manually set corresondence, such to stitch seashore images, automatic feature matching works very well. The result of automatically found matching are amazingly good. When implementing the algorithms, I found lowe's trick really helpful. Only selecting matching features on 1_NN give plenty bad matches. Using 1_NN/2_NN ratio instead greatly improve the accuracy.

Bells & Whistles

Some experiments with cylindrical mapping

panorama recognition

I randomly selected some other picture as shown below, and add one set of panorama pieces in. For every picture, calculate the number of matching feature descriptors. Pieces of panorama have much bigger matching counts than others, and thus can easily picked the pieces of panorama out. Then according to the position of matching feature on the picture(large sum of x axis mean more matching on right, means the picture is left of the panorama), to get a ordered left to right sequence of the piece, and then warp it as usual. Now it can do all the desk, living room and windows panorama automically from set of other pictures. It currently only support stitches of 3 picture.