CS194 Project #6 [Auto]Stitching Photo Mosaics
Part I: Image Warping and Mosaicing
Recover Homographies
H is a 3x3 matrix with eight degree of freedoms. I solved this problem with at least four corresponding points between two images and model the problem as solving a linear system of equations using linear algebra. For the image warping, I used inverse warping with value interpolation.
In this example, I defined ten pairs of corresponding points.
There are more corresponding points than required to solve this linear equation because the linear system was solved using least square with overdetermined matrix.
Warp the Image
The image was warped using inverse warpping function similar to that in project three. However, prior to warping, the exact corresponding was automatically tuned to give the best alignment result using SSD.
target image homography based on the matrix H
Image Ractification
These are some of the image ractification resluts. Notice that the results might not be ideal since the perfect corresponding points are hard to get and I am still in the process of trying to solve this issue.
Although the target was a ractangle with correct aspect ratio for the Mona Lisa example, the reslut was still not perfect as the top and bottom edge of the painting were not parallel. Upon checking with the buil in cv2 function reveals that there might be some bugs in the warping function.
Blend Images into a Mosaic
After the image ractification, it is possible to blend multiple images together to create a mosaic. The part of the project is still actively deveoping and the reslut here are just some of the progerss so far.
Below are the source images for the final mosaic.
ractification target
ractification source
There is still a lot of work left to do in this first checkpoint of the larger project five. So far I am already amazed by how good a simple projective matrix will do to produce the mosaic.
Part II: Feature Matching For Autostitching
Extending from the project 6 part I. After successfully implemented the function to find the homography and the function to warp the image, we want to get rid the last bit of the manual labor to find a way for automatic feature matching and autostitching. This project was largely based on and inspired by the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al. with some significant simplification.
Data
Harris Interest Point Detector
Harris corner detection was used to find interest points for the feature. The Harris corner detector uses the ratio of eigenvalues of a masked region to find interest points where a corner might exists. I used the build-in corner_harris and corner_peaks functions in skimage to implement the function. Below are the outputs from the sample image.
Adaptive Non-Maximal Suppression
As shown above, sometimes the Harris corner detector finds too many interest points that are similar and repetitive. The computation needed to perform feature matching on such a data set is simply too much. Adaptive Non-Maximal Suppression is a way to find the interest points that are more evenly spread out in the image and reduce the amount of interest points. In order to do that, I first sort all the interest points by their corner strength from the strongest to the weakest, then find the distance of the nearest interest point that is just slightly stronger for each interest points, and then sort the interest points based on the distance. The idea being if one point is the strongest in a given radius, then it must also be the strongest in a smaller radius as well.
Feature Descriptor Extraction
Normalized features were extracted from each of the interest points output from the adaptive non-maximal suppression algorithm. The feature was extracted from a 40x40 pixels patch of the grayscale image about each interest points and down-sampled to 8x8 pixels. Additionally, each feature can be made rotation-invariant by using a octagon mask to calculate the angles of the corner and axis-aligned each feature.
Above shows the features extracted from each pair of images. For each image, first and thrid columns are the features extracted from the same image and second and fourth columns are the features extracted from the same image. The features are also matched using the feature matching algorithm described below. Each row are two pairs of matched features.
Feature Matching
Each pair of features is matched using Euclidean distance to give a ranking of feature matching scores. Then set of features that are used for RANSAC was then selected using the ratio of the strongest matched feature distance over the second strongeset matched feature distance with the threshold of 0.7 to try to minimize the number of incorrect matches in the set.
For each image above, all the points are the output of ANMS algorithm and the blue points are the threshold matched features in each pair of images.
RANSAC
Finally, after we have found the matching pairs, we can use RANSAC algorithm to find the best set of inliers where we can compute homography. The following images demostrated one example of four random pairs of matching points that can be used to calculate the inliers using RANSAC. Each color correspond to one matching pair.
For the homography from image 2 to image 1, there are 37 pairs of points in the inliers with SSD less than 3. For the homography from image 3 to image 2, there are 27 pairs of points in the inliers with SSD less than 3. The ouput homograph matrices calculated using the largest set of the inliers are reported below.
After find the homography the image was then warped using warp function in Part I of the project and using alpha blending with laplacian pyramind as oraple example from project 2 to yield the final panorama that was produced automatically.
Overall this is a really fun and challenging project. One of the coolest thing I learned from this project is using features to match the image together. It is really mind blown to me to see how well it works.