CS194-26 Project 4A: Image Warping and Mosaicing

Matthew Tang

Part 1: Shoot and digitize pictures

The first step was to take pictures. I used my iPhone7 to take the photos, trying my best to only rotate the angle and not translate the reference point (to be able to accurately reconstruct the homographies later). I took photos for 3 sets of panoramas.

Campanile*

Soda Hall Entrance*

Apartment Hall*


* Note: images displayed are compressed to fit the 25MB max size for upload utility

Part 2: Recover Homographies

We will need to compute a homography to warp the photos to be able to stitch them together. I chose to do an inverse warp to avoid aliasing problems (using similar interpolation code to proj3). We can fix the bottom right element of H to be 1 (the scale factor) and solve for the rest of the parameters. Although 4 points uniquely identify a homography, I chose to take more than 4 points, resulting in an overconstrained system to reduce the effects of error in point selection. For this reason, I solved for the H matrix using least squares using the following formula*:

* credit to this website for the derivation.

To more accurately choose points, I zoomed in and selected them in Photoshop.

Part 3: Warp the Images

Now that I had a homography matrix H, I could warp the photos. To stitch 2 photos together, I would inverse warp the second one (on the right) to the first one (on the left). For an arbitrary number of photos n, I would finish the process of stitching the first n - 1 photos together, and then stitch that result with the nth photo.

Part 4: Image Rectification

As an aside, we now have the tools to rectify images. That is, given an image of an object (taken at an angle) that we have outside knowledge to be rectangular, we can warp the image to be rectangular.

Painting 1 (Original)

Painting 1 (Rectified)

Painting 2 (Original)

Painting 2 (Rectified)

Part 5: Blend the images into a mosaic

Now that we have the images warped, we want to stitch them together. An easy way to do this is to select the left image to be the shared canvas. The right image will have precomputed left padding since I am doing an inverse warp of points to the left image. I add right padding on the left image to make the two images the same size. Then I create a mask and do laplacian blending (code taken from proj2). See part a for the intermediate images (removed to fit size limit)

Campanile Panorama


This panorama looks pretty good. You can see a slight seam in the sky since unfortunately the exposure/camera settings seems to have changed between photos.

Soda Panorama


This panorama looks good too. The seam is a little more obvious this time unfortunately due to different exposure/camera settings (sigh "idiot cameras")

Apartment Panorama


This panorama looks good too. Seam is less visible than the other indoor one.

The coolest important thing I learned from this project was accomplishing my homography and warping without for looping over pixels. I forced myself to think in terms of vectorized functions and use numpy functions to efficiently code.

CS194-26 Project 4B: Feature Matching for Autostitching

Part 1: Harris Interest Point Detector

I integrated the Harris point detector with my code to obtain corners, overlaid on the images below.

Part 2: Adaptive Non-Maximal Suppression

To avoid clustering of feature points, we can use non-max suppression to remove some of them. The results of that are shown below. I used a c value of .9 as suggested by the paper. This resulted in around 50-70 interest points remaining (fine tuned based on the total number of points)

Part 3: Feature Descriptor extraction

Now that we have interest points, we need to extract patches around it. I extracted a 40x40 window and downscaled it (with anti aliasing) to 8x8.

Part 4: Feature matching

With the patches, I was able to match pairs of features. I used Lowe's threshold between the 1NN and 2NN to throw out false matches. The results are shown below.

Part 5: RANSAC

The final step is to implement RANSAC, as shown in class. Running RANSAC for 1000 iterations with epsilion = 10 worked well (Soda ran more consistently with 10000 iterations since it had more points for RANSAC). The results for the mosaics are shown (Auto on left, manual on right). The results look near identical to the human eye.

Campanile Panorama

Soda Panorama

Apartment Panorama



The coolest important thing I learned from this project was that this stuff actually works. I was skeptical that these methods would really throw out all outliers, since the homography calculation is very sensitive to outliers. I thought that this would be a really hard computer vision problem to solve, but it seems like it is actually a solved problem.