CompSci 194 - AutoStitching Photo Mosaics

" This project is all about stitching together many photographs to create larger composite images. "

Part A. IMAGE WARPING and MOSAICING

Shoot the pictures. The pictures are taken of a playground from exactly the same points of view, but with different view directions. For the right image, the camera is slightly tilted to the right.

Recover Homography.

[[ 2.14469011e+00 -8.18563958e-03 -1.62752628e+03]
[ 3.90044474e-01 1.73674619e+00 -2.87985995e+02]
[ 8.56283396e-04 -1.59979507e-04 1.00000000e+00]]

I manually selected 8 points from each of the pictures, all of which are distinct features such as the corners of the slide, the edges of the stairs, as shown above.

In order to recover the homography matrix, since there are more data (16) than unknowns (8), I rearrange p'=pX equation and solved the overdetermined system by least square method np.linalg.lstsq().

To make sure the warped image does not go beyond the canvas, I used forward warping to compute where the image corners land in the destination direction to see if they go overboard. If so, I pad the image with 0 (black borders) beforehand.

With the homography matrix H, I am able to warp the left image to match the view direction of the right image. Since inverse warping works better, I compute the corresponding coordinates of pixels in the source image of each pixels in the destination image. I used scipy.ndimage.map_coordinates() to map the corresponding pixels.

With the left image warped to match the view direction of the right image, I stitch them together to make a mosaic. The results are shown below.

--- other examples ---

--- other examples ---

Image rectification. I took a few sample images with some planar surfaces (notebook, cupboard). The goal is to "rectify" their tilted surfaces so that the plane is frontal-parallel.

Since the objects are rectangles, I selected only 4 points from each image (4 corners). Then I defined 4 points that form a perfect rectangle shape in the center of the image canvas, and perform a warping on the original image. The process is straightforward and simple, and all of my rectification attempts are successes.

Part B. FEATURE MATCHING for AUTOSTITCHING

Step 1-3: For this part, I implemented the algorithm described by the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al.

Step 1: Harris Interest Point Detector (Section 2). I used the given sample code harris.py to extract harris corners points from each images. I had to adjust the min_distance parameter otherwise there would be too many interest points detected.

Step 2: Apply Adaptive Non-Maximal Suppression, in order to suppress the number of interest points. For each image, I pick the top 500 points only.

Step 3: Implement Feature Descriptor extraction + Feature Matching. For each 5x5 square within the 40x40 window, I picked the center pixel, which gives me an 8x8 patch. After getting descriptors, I applied feature matching technique, meaning to find pairs of features that look similar and are thus likely to be good matches: I reject the point if it is an outlier and return the points only if they are good matches. After a few experiment runs, I set the threshold to 0.3.

Step 4: RANSAC. The purpose of this step is to get ground truth correspondence points: to eliminate matched feature descriptor points returned from last step that are not actually corresponding pairs. For each iteration (500), I randomly select 4 points from the set to compute a homography matrix, assess whether or not it gives a good warping transformation and exclude it if not.

Warp. With the correspondence points produced by previous steps, I am able to create warping transformations ···

··· and successful Mosaics.



--- other example ---

Harris Points

ANMS Points

Matched Descriptor Points

RANSAC Points

Harris Points

ANMS Points

Matched Descriptor Points

RANSAC Points

What I learned from the project

For part A, a significant part that I learnt is that after multiplying the homography matrix (or its inverse, if doing inverse warping) with the coordinate matrix (x, y, 1), it is important to divide the resulting matrix warp by warp[-1], the bottom row, to ensure homogeneity. If not, the even though i could get correct homography matrix, my warping transformation can be a mess.

For part B, I learnt that each method/package (e.g. matplotlib.pyplot, skimage.feature) could interpret coordinates differently: some see it as (column, row), while some see it as (x, y), which are contrary. It caused me a significant amount of time to debug my code because I failed to incorporate the coordinate representation at first.