Project 5 [Auto]Stitching Photo Mosaics

Lizhi(Gary) Yang

Part A (Part B is below)

Shoot and digitize pictures

Here are the photos I shot for the project:

Left Image Right Image

Recover homographies

First I define the 4 points in each image with the ginput function of matplotlib. Then I construct the A matrix and b vector in order to solve a least-squares problem with the form Ah=b for the unknowns in the homography matrix, where for one point:

A=[[x2, y2, 1, 0, 0, 0, -x1x2, -x1y2].
       [0,0,0,x2,y2,1,-y1x2, -y1y2]]

After I get h, I reshape it into a 3 by 3 matrix and the last element is 1. The homography matrix is as follows, where H is the result of the least squares.

[[H[0], H[1], H[2]],
 [H[3], H[4], H[5]],
 [H[6], H[7], 1.]]

Warp the images

After I have the homography matrix I am able to rectify images with inverse warping, here are 2 examples of rectification, left is original and right is result:
Before Rectification After Rectification

Blend images into a mosaic

Blending is done with alpha blending and a cosine function evaluated from 0 to pi/2 over half the image for the values of alpha - since we want to gradually move from the first image to the second. Here are 3 mosaics with the original images on the left and the result on the right:
Left Image
Right Image Pano Image

Bells and Whistles

Video mosaic

I used my phone and borrowed a friend's device to record the flow of the water fountain. On the left are the original videos and on the right is the mosaic. I handheld the devices so the video is not that stabilized.

Left Video
Right Video
Pano Video

Part B

Find matching feature points

I use the Harris corner detector to find corners, then apply ANMS to pick 100 points with highest suppression radius which is defined to be the minimum distance from that point to any point with corner strength than it. Then I extract a patch of size 40x40 around each point, apply a Gaussian filter to each patch and normalize it.  To find the feature matching, I calculate the ratio of 1-NN squared error with 2-NN squared error, and designate a match if the ratio is less than 0.2 in accordance with the paper. From left to right below are the Harris corners, the ANMS result and the Feature Matching result:

Harris corners ANMS result Feature Matching result

4-point RANSAC

After getting the feature points, I apply 4-point RANSAC by randomly choosing 4 pairs at a time, calculate the homography matrix, and  find the number of correctly mapped points. I do this for 1000 steps and choose the homography that generates the most number of correctly mapped points, for which I apply part A's warping and mosaic to get the following images. The second example is more "straight" while the third example has much less ghosting effect than the result from part A's hand-picked points.

RANSAC Hand-picked