Mosaics and Image Rectification with Kevin

The process of creating a mosaic involves selecting corresponding coordinates to define a transformation matrix H, which we then use to transform the perspective of one image to that of the other. This process is called rectification. We then blend the two images using alpha blending and continue the process again with the next image we plan to add to the mosaic.

Image Rectification

We check our H matrix by doing a couple image rectifications. First we select a set of points p in on our original image and then a set of target images p'. We then just solve the system of linear equations Hp = p'. We can then use the warp function using the H that we calculate using least squares on the set of points p we selected. The output is our rectified image.

Bags of Coffee at Sightglass SF

Blue Bottle Latte Art

Macbook

Manual Mosaics

Let us now go through the process of developing a mosaic. We start by placing each of the images on a large canvas. Then we select a series of corresponding points in the images as shown below.

Now that we have the corresponding points, we warp the left and right images into the perspective of the center image.

Then we blend the images together using alpha blending. First I blended the left warped image to the center and then I blended the right warped image to the blended image, producing our mosaic.

Here is our final image!

Another Example

Automatic Image Stitching

Now the question is can we do better? Actually we can. We do better by automatically detecting features, giving us a more precise features that even humans can't detect. We outline the process below and show our results.

Step 1: Harris Corners

We want to determine potentially good features. Based on human intution when picking features manually, we notice that corners are good features to select. We use the Harris Corners detection to select the potential corners in our image. We run the Harris Corner detection algorithm which results in the image above.

Step 2: Adaptive Non-Maximal Suppression

Unfortunately, it's very difficult to work with such a large number of points just like those plotted above so we selected a subset of the best points. We select the best points using Adaptive Non-Maximal Suppression. Running ANMS gives us the minimum distance r_i for each given point x_i to the next closest point x_j such that x_j meets the criteria that its harris corner strength is at least 0.9 times the harris corner strength of x_i. We return then return the points with the top 500 r_i values. We notice the points are nicely distributed in the image above.

Step 3: Feature Descriptor Extraction

We now define a feature vector for each point we selected through running the ANMS algorithm. For each point, we define a 40x40 axis-aligned box which we then downsample to an 8x8 box and convert it into a 64x1 vector. This gives us a feature vector for each image which we will use for comparisons. We normalize each feature vector to be zero mean and have variance 1.

Step 4: Feature Matching with RANSAC

We still have a bunch of points to choose from so now we need to find good correspondences. We do this using nearest neighbors. For each pair of features, we compute the difference between the two. Then we take the min and the second min. Since a good match would have the greatest point be much better than the second best option, we only consider correspondences for points with a ratio of nn-1 / nn-2 that is below a given threshold.

Now that we have narrowed down our set of points and mapped correspondences, we use RANSAC to determine a set of inliers, or best correspondence points. To implement RANSAC, we randomly select 4 of the correspondences, compute a homography and then determine the error of the homography for each point. For the points that fall under the threshold, we add them to our set of inliers. We continue this for 10000 iterations and then compute a final homography over all the points.

We show the final warping results for the left and right images below.

Step 5: Warp & Blend into Mosaic

The results of the three mosaics are shown below. We compare the results with the manual image stitching and automatic image stitching. In the SF mosaic, we see a better alignment especially along the edges and less ghosting of the overlapping images. This is especially true for the Wayne mosaic, where the human standing is much more clear and not blurry, a definite improvement.

Manual SF Mosaic

Automatic SF Mosaic

Manual Coffee Mosaic

Automatic Coffee Mosaic

Manual Wayne Mosaic

Automatic Wayne Mosaic

Final Gallery

Overall, I really enjoyed doing this project. I learned a lot implementing AMNS which was probably my favorite part because it was cool seeing how well it chose well distributed points. Also it was cool using the NN trick to find good correspondences and then finding an even better correspondence matching using RANSAC.