CS194 Project #6 [Auto]Stitching Photo Mosaics

By: Scott Shao

Part I: Image Warping and Mosaicing

Recover Homographies

H is a 3x3 matrix with eight degree of freedoms. I solved this problem with at least four corresponding points between two images and model the problem as solving a linear system of equations using linear algebra. For the image warping, I used inverse warping with value interpolation.

In this example, I defined ten pairs of corresponding points.

Source image with corresponding points
Target image with corresponding points

There are more corresponding points than required to solve this linear equation because the linear system was solved using least square with overdetermined matrix.



Warp the Image

The image was warped using inverse warpping function similar to that in project three. However, prior to warping, the exact corresponding was automatically tuned to give the best alignment result using SSD.

Source image warped into
target image homography based on the matrix H
Target image


Image Ractification

These are some of the image ractification resluts. Notice that the results might not be ideal since the perfect corresponding points are hard to get and I am still in the process of trying to solve this issue.

Source image
Image ractification reslut one
Image ractification reslut two
Source image
Image ractification reslut

Although the target was a ractangle with correct aspect ratio for the Mona Lisa example, the reslut was still not perfect as the top and bottom edge of the painting were not parallel. Upon checking with the buil in cv2 function reveals that there might be some bugs in the warping function.

Blend Images into a Mosaic

After the image ractification, it is possible to blend multiple images together to create a mosaic. The part of the project is still actively deveoping and the reslut here are just some of the progerss so far.

Below are the source images for the final mosaic.

Image ractification reslut one
Image ractification reslut two
Image ractification reslut one
Image ractification reslut two
Second image
ractification target
First iamge
ractification source

There is still a lot of work left to do in this first checkpoint of the larger project five. So far I am already amazed by how good a simple projective matrix will do to produce the mosaic.

Part II: Feature Matching For Autostitching

Extending from the project 6 part I. After successfully implemented the function to find the homography and the function to warp the image, we want to get rid the last bit of the manual labor to find a way for automatic feature matching and autostitching. This project was largely based on and inspired by the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al. with some significant simplification.

Data

image_1
image_2
image_3

Harris Interest Point Detector

Harris corner detection was used to find interest points for the feature. The Harris corner detector uses the ratio of eigenvalues of a masked region to find interest points where a corner might exists. I used the build-in corner_harris and corner_peaks functions in skimage to implement the function. Below are the outputs from the sample image.



Adaptive Non-Maximal Suppression

Example output with too many Harris corner interest points.

As shown above, sometimes the Harris corner detector finds too many interest points that are similar and repetitive. The computation needed to perform feature matching on such a data set is simply too much. Adaptive Non-Maximal Suppression is a way to find the interest points that are more evenly spread out in the image and reduce the amount of interest points. In order to do that, I first sort all the interest points by their corner strength from the strongest to the weakest, then find the distance of the nearest interest point that is just slightly stronger for each interest points, and then sort the interest points based on the distance. The idea being if one point is the strongest in a given radius, then it must also be the strongest in a smaller radius as well.

Sample image Harris corner interest points.
Top 500 points from the output of ANMS algorithm.


Feature Descriptor Extraction

Normalized features were extracted from each of the interest points output from the adaptive non-maximal suppression algorithm. The feature was extracted from a 40x40 pixels patch of the grayscale image about each interest points and down-sampled to 8x8 pixels. Additionally, each feature can be made rotation-invariant by using a octagon mask to calculate the angles of the corner and axis-aligned each feature.

Sample features extracted from image 1 and image 2.
Sample features extracted from image 2 and image 3.

Above shows the features extracted from each pair of images. For each image, first and thrid columns are the features extracted from the same image and second and fourth columns are the features extracted from the same image. The features are also matched using the feature matching algorithm described below. Each row are two pairs of matched features.

Feature Matching

Each pair of features is matched using Euclidean distance to give a ranking of feature matching scores. Then set of features that are used for RANSAC was then selected using the ratio of the strongest matched feature distance over the second strongeset matched feature distance with the threshold of 0.7 to try to minimize the number of incorrect matches in the set.

Image 1 outputs from image 1 and image 2.
Image 2 outputs from image 1 and image 2.
Image 2 outputs from image 2 and image 3.
Image 3 outputs from image 2 and image 3.

For each image above, all the points are the output of ANMS algorithm and the blue points are the threshold matched features in each pair of images.

RANSAC

Finally, after we have found the matching pairs, we can use RANSAC algorithm to find the best set of inliers where we can compute homography. The following images demostrated one example of four random pairs of matching points that can be used to calculate the inliers using RANSAC. Each color correspond to one matching pair.

Image 1 randomly selected points for RANSAC.
Image 2 randomly selected points for RANSAC.
Image 2 randomly selected points for RANSAC.
Image 3 randomly selected points for RANSAC.

For the homography from image 2 to image 1, there are 37 pairs of points in the inliers with SSD less than 3. For the homography from image 3 to image 2, there are 27 pairs of points in the inliers with SSD less than 3. The ouput homograph matrices calculated using the largest set of the inliers are reported below.

Homography matrix calculated using inliers between image 1 and image 2.
Homography matrix calculated using inliers between image 2 and image 3.

After find the homography the image was then warped using warp function in Part I of the project and using alpha blending with laplacian pyramind as oraple example from project 2 to yield the final panorama that was produced automatically.

Final homography from stitching image 1 and image 2 together.

Overall this is a really fun and challenging project. One of the coolest thing I learned from this project is using features to match the image together. It is really mind blown to me to see how well it works.