I took the pictures on my iPhone camera with the exposure and focus locking feature turned on. The set of images were obtained by rotating the field of view while keeping the camera position fixed.
At least four points of correspondence are taken to formulate a least square problem and solve for the homography transformation.
Select 4 points on the image that should form a rectangle if viewed from the right orientation. Warp the image to the supposed rectangle using the homography transformation matrix.
Here is an example:
I defined 8 points of correspondence for each mosaic. I first implemented naive blending which is just stacking the warped images together. The edge of the warped images are apparent. I tried many ways to blend them more smoothly, and I ended up with a nice blending scheme that is inspired by the SoftMax function. For every overlapping pixel, I calculate its Euclidian distance to the nearest edge of each image, take the exponential of that distances with some tuned temperature and divid the distance to the first image by the sum of the two distances.
exp(distance1) / (exp(distance) + exp(distance2))
The original images and the mosaic are as follows:
Harris corner detection works by computing the x and y image gradiant, taking the outter product to get a matrix, and computing a score by dividing the determinent of that matrix by the trace. In my implementation, I set a threshhold of 10% of the highest score to filter out low score points.
Below is the top 1000 harris interest points for a set of images:
We notice right away that the distribution of interest points with large scores are highly skewed, which introduce us to a improvement trick in the next section.
In order to ensure we have well spatially distributed interest points for better correspondence matching, Adaptive Non-Maximal Suppression(ANMS) is used to select interest points from the set of points given by the harris detector.
The idea of ANMS is to select points that have large distances to their nearest point with high score. In my implementation, I rank each interest point by the said distance and take the top 1000.
Here are the same image but with ANMS points selection:
For feature extraction, I first blur the image by a gaussian filter with sigma = 5, then take a 40 by 40 patch around each interest point, and finally downsample it to a 8 by 8 patch.
Given the feature patches of our interest points, we first try to match corresponding features of each image by calculating the 1-NN/2-NN of that feature where 1-NN is the nearest neighbor error and 2-NN is the second nearest neighbor error. The intuition behind this is that when we have a matching pair of features, the error distance of the right match is smaller than the second close match by a lot. I limit this ratio to 0.5 to get the set of candidate matches.
Next we use the Ransac algorithm to select match points that satisfy spatial constraints the most, which is to say that the recovered transformation aligns more candidate points than others.
In my implementation, two points are considered aligned if the euclidian distance between them is less than 1. Usually, 1000 Ransac iteration is enough, but to avoid unlucky situations, I raise the number of iterations to 5000. With my vectorized implementation, this is still pretty fast.
Below is an example of the Ransac auto-selection on the same set of image:
Here are mosaics created by hand-picked correspondence(left) and autostitching(right):
We can see that the mosaics on the right, the autostitching results, looks better than hand-picked results. The reason for this is that autostitching can select more points on each image such that the least squares algorithm can recover better transformations. In addition, autostitching select points on a pixel by pixel level which is more acurate by nature.
I designed a non-conventional blending technique that is inspired by the idea behind SoftMax smoothing. My blending technique works well in practice and is really fast to do.
I learned how to manipulate images with sophisticated coordinate transforms. This project integrated many aspects learned in previous projects. Autostitching is a powerful and fun algorithm. I learned a lot about feature engineering. The resulting mosaics are beautiful and the algorithm enables many artistic possibilities to try out in my spare time.