Overview

For this project, we experiment with homographies and then warp images taken from same point of view but from different view directions to blend them into a paranoma mosaic. After playing with finding transformation using manually selected correspondance points, we implement feature matching with feature patches for auto-stitching and produce mosaic automatically.

Part A IMAGE WARPING and MOSAICING

Part 1 Shoot the Pictures

It's hard to go out given the current pandemic situation, so all the pictures are taken from back home or found online.

Part 2 Recover Homographies

For perspective transformation, there are 8 degrees of freedom, so at least 8 correspondance points are needed for calculating the matrix; if there are more points, then we can use lease squares to solve the overconstrained system of equations. For this part, 30 correspondance points are used for more accuracy.

Part 3 Warp the Images

Using the perspective transformation matrix, we can warp the left image into right image's projection.

Part 4 Image Rectification

Using the same method to find the perspective transformation matrix, we can warp any planar surfances into front view projection. Here are few examples.

Part 5 Blend the images into a mosaic

When images are registered, we can blend them to form a wider-angle-perspective mosaic picture. First approach I try is simplying adding them together, which creates a wierd picture with high brightness in the middle, then I use a simple mask to weight average the middle part. Using this method will create a line of subtle edges between where images are blended together.

Part B FEATURE MATCHING for AUTOSTITCHING

Part 1 Harris Interest Point Detector

For finding features automatically, we first find the harris corner points for the image. Since we will run ANMS later, it is ok to have a large number harris corner points at this step. For a 3000 x 4000 image, with 20 min_distance, it will produce around 5500 interest points.

Part 2 Adaptive Non-Maximal Suppression

After finding harris corner points, since there are too many unrelated points and not spatially distributed evenly, we run ANMS to find the 500 dominant feature points that distributed across the image.

Part 3 Feature Descriptor extraction

Then for each points, we sample a 8 x 8 patch from a 40 x 40 window to produce a feature descriptor, and bias/gain-normalize the descriptors. This will allow us to use these feature descriptors for feature matching and they are photometrically invariant to affine changes in intensity.

Part 4 Feature Matching

Then we will match the feature descriptors by finding their nearest neighbour in the other picture. To ensure that they are pretty close match, they have to be mutual nearest neighbour and the ratio of distance between 1-NN / distance between 2-NN is small, to guarantee that the first choice is much better than the second choice and not an ambiguous selection. We use 0.3 for thresholding. Here are the matched points.

Part 5 4-point RANSAC

Lastly, with the matched feature points, we run RANSAC to find the homography transformation matrix with the largest group of inliners. I run the iteration for 100 epochs and use 2 as the ssd strength limit. It turns out to have a largest group of 9 inliners out of the 50 feature matches. Then we will go on warp the first image and blend it with the second image to produce the mosaic.

The results are pretty much the same! We did it! It is fun to manually select the correspondance points, but it will be much more convenient to produce the mosaic automatically. Perspective transoformation is so powerful that we can so so much image processing application with it. Definitely want to do one with my final project!