image warping

We'll have to figure out how to warp images and effectively stich them together, with minimal effort beyond taking pictures. For most of this project, we'll be following “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al. We'll start with rectification, as this is easy enough being simply finding the homography matrix necessary to map the four corners of the planar object onto the four corners of a axis-aligned image.

mosaic

With that under our belt, we start on more complex stiching. We take some pictures, and manually label correspondence points. We use least squares on these pairs of correspondence points to figure out the best homography matrix to warp and create a mosaic.

harris and suppression

While it's seeming to be doing okay with quickly done hand picked correspondence points, it's not a nicest since you have to manually pick those points. Let's try and use automatic correspondence point pickers. Harris points seem promising.

Ok, that's a lot. Let's cut that down a lot more with automatic non-maximum suppresion detailed in the paper. We want to make sure each correspondence point is the strongest in its local area. So for each point, we calculate the minimum distance to a point with a significantly higher energy. We formalize this distance as

$$ r_i = \min_j |x_i - x_j|, s.t. f(x_i) < c_{robust} f(x_j)$$ $$ c_{robust} = 0.9 $$

We select the lowest 300 points to be our new set of correspondence points.

match and ransac

We cut down on the amount noise, but we still have many points that don't necessary correspond to each other. Let's try and sample each point's local neighborhood and compare it to every other point in another image. We expect matching points to have similar local samples. However, we add another constraint that we believe the second nearest neighbor shouldn't look anything like the feature. Therefore the ratio of distances between the closest point and second closest point should be significantly low (< 0.5).