Project 6: IMAGE WARPING AND MOSAICING

Mian Zhong cs194-26-acm

In the first part of this project, I captured some photos to stitch, defined correspondences, computed Homography matrix and warped images! The homography matrix calculation is very similar to what we have done for affine matrix in previous project. The most challenging part for me was warping function implementation. With proper understanding, I finally debugged all flipped coordinates issue about my warping function after two long nights!

Part A

Part 1: Compute Homography

The homography transformation is defined as p' = Hp, where H is a 3x3 matrix with 8 degrees of freedom (set H_33 = 1). To recover the matrix, we rely on two sets P, P' of correspondence points from two images. The set needs to have at least 4 pairs to proper recovery. For every (x,y) in P, and (x', y') in P', we have (wx', wy', w) = H*(x, y, 1). Therefore, we can set up 2n x 8 matrix A, and h = [a b c d e f g h].T where n is the number of correspondence pairs, and a-h are 8 variables in H we try to recover. Ah=b such that b is all unrolled coordinates from P'.

Part 2: Image Rectification

Here is a tapa bar menu I took in Barcelona, and rectified menu:

Here is a whiteboard lecture note on the left, and rectified notes on the right:

Part 3: Mosaicing

Special thanks to my caring friends who hosted me almost every weekend to stay loved, supported and cozy. Here is my first try of shooting out study/dining place using a phone camera:

I use the center one as objective projection, then I got the left and right warped images:

The final mosaicing with and without cropping:

I captured three pictures using a DSLR camera at a rooftop. To speed up, I resized the image by 0.25, and again warped the first(L) and the third(R) image to the center:

The final mosaicing with and without cropping:

Part B

Part 1: Harris Corners & Adaptive Non-Maximal Sampling(ANMS)

I used the skeleton code to detect harris corners on each image ready to be stitched. However, there are way too many corners detected by the algorithm. Thus we want to process the image such that good corner features are used. If we directly select the Harris corneres with the strongest score, for example, top 500. Corners are likely to cluster as below:

Therefore, we use ANMS to sample corners to be the local maximum in a certain neighborhood. Applying ANMS, we get corners more spreading on the entire picture. Examples of the same pictures are shown below:

Part 2: Feature Descriptor

I first used a 40x40 window around each harris corner, blurred the patch with a gaussian filter and then resized the patch to 8x8. Then I flattened the patch to be the feature descriptor of that harris corner.

Part 3: Feature Matching

I used the fast distance method to calculate the distance matrix between the feature descriptors of two images. Then I found the first and second nearest neighbor and calculated the ratio. I used a threshold 0.5 to select correspondence pairs. Below are examples of correspondence points on two to-be-stitched images. We can notice that there are some bad correspondence pairs. For example, there is no corresponding scene in the second image for the the rightmost three points in the first image. Thus we will try to eliminate those bad correspondence pairs in the next part.

Part 4: RANSAC

RANSAC is probablistic method. First, we randomly select 4 corresponding pairs, compute the homography matrix and then compute the set of inliers. I pre-defined the number of iterations, and repeated to update the inlier set to be largest. Lastly, we use the finaly inlier set to be the good correspondec pairs and calculate homography matrix for imaging warping.

Part 5: Final work

Here are three mosacis I generated. I found that the visual quality slightly worse than the imaging from part A with more noticeable stitches. However, the overall is pretty good and the correspondences are found automaticly!