[Auto]Stitching Photo Mosaics


In part of the project, I took three photos and implemented programs to recover homograhies, warp the images, rectify the images, and blend the images into a mosaic.

1. Shoot and digitize pictures (20 pts)

In this part, I shoot 3 images for rectification, and 4 pairs of images for stitching. These photos are taken with my smart phone camera.

Photos for Rectification

Photos for Mosaicing

2. Recover Homographies (20 pts)

This part of the program produces a 3x3 matrix H with 9 degrees of freedom (lower right corner is a scaling factor that equals 1) that recovers the homography between each pair of images - p' = Hp. It first uses ginput to collect a set of (p', p) pairs of corresponding points taken from the two images. Then I set up A and b to compute the entries of matrix H. The correctness is verified by the following parts that are based on homography computation.

3. Warp the images (20 pts) [produce at least two examples of rectified images]

In this step, I used the homography computed in the previous part to warp my images. The warpImage function takes an image and a matrix H, and warp the image in the direction represented by the homography. The size of the resulting image is passed in as an argument, and cv2.remap is used to do the interpolation. By passing in four coordinates that form a rectangle (based on my observation) and selecting the four corresponding points in the planar surface images, I warped those planes to be frontal-parallel.


4. Blend images into a mosaic (20 pts) [show source images and results for three mosaics.]

In this part I created image mosaics by warping the right images and leaving the left images intact. The overlapping parts of the two images are weighted averaged to avoid strong edge artifacts.

Intact Warped (Manual) Mosaic

What I've Learned

I learn that it is important to set up and transform matrices and equations and through this project I am getting more comfortable coding matrices. I get a better idea about homography, and learned the cv2.remap approach to interpolation (from Piazza).


In part of the project, I used four pairs of photos and implemented program to automatically finding correspondances and create panoramas. The techniques refers to "Multi-Image Matching using Multi-Scale Oriented Patches" by Brown et al.

1. Detecting Corner Features in an Image (10 pts)

With the Harris Interest Point Detector sample code provided, I set reltive thresholds (for the "Desk" image) and min distance through arguments for each pair of images. The Harris Interest Point detector finds the corners because corners can indicator some features of an image.

In the next step I implemented Adaptive Non-Maximal Suppression (ANMS) to reduce the number of corners. This works by computing the radius from a point to other points and keep k points with the largest radius value while suppressing the weakers points.

Living Room Front Yard Back Yard Desk

2. Extracting a Feature Descriptor for Each Feature Point (10 pts)

3. Matching These Feature Descriptors between Two Images (20 pts)

In the second and third steps, I extracted feature descriptors and find matches between two images. I sampled 8*8 patches from 40*40 window, and normalize by subtracting mean and dividing by standard deviation. Then I computed squared distances and find a ratio of 1NN and 2NN, and compare with the threshold.


4. Use a Robust Method (RANSAC) to Compute a Homography (30 pts)

As introduced in class, the RANSAC loop starts by randomly sampling 4 points, and compute homography. Then I compute and count the number of inliers relative to an epsilon value. I keep the largest set of inliers and output the cooresponding points.


5. Produce a Mosaic [produce at least three mosaics] (30 pts)

The outputs of RANSAC are used as inputs to recompute homography, warp image, and make mosaic. The manual and automatic results are shown as the following.

Manual Auto

What I've learned

The techniques introduced in this part of the project are amazing - they are effective and the intuition makes a lot of sense to me. Some of the automatic output actually masaics better than manual input, and the process is really convenient and powerful.