CS 194-26 Project 5: (Auto)Stitching Photo Mosaics

Shreyas Patankar

Part A: Image Warping and Mosaicing

Shoot the Pictures

For the purposes of Part A, I have shot the following photos of my living room to ultimately stitch together:

Left Shot

Right Shot

I have also shot the following photo to test image rectification--I'll do a front perspective of the painting on the left.

Painting

Then the homography matrix was computed using least squares to solve for H in the equation p' = Hp, where p represent the points in the original image, and p' represents the transformed points. See section below.

Recovering Homographies

The next part in the project was to compute the Homography matrix to create the "perspective change." I first began by selecting 5 keypoints in each image. I selected the keypoints as follows:

Room Keypoints

Painting Points

Then the homography matrix was computed using least squares to solve for H in the equation p' = Hp, where p represent the points in the original image, and p' represents the transformed points. I ended up only using 4 out of the 5 points in the homography computation.

Image Rectification

The homography matrix H was then used to "perspective change" the photos. In my case, I chose to change all images to match the perspective of the middle image. Below, I have shown 1 example of my homography translation for the room change as well as straightening of the painting.

Room after Homography

Painting after Homography

Blending the images into a Mosaic

I then was able to take the room after homography & blend that with the second picture of the room. The result is shown below:

Room Stitch from Manual Keypoints

The result is not perfect--I presume this is because I slightly shifted my hands while taking the photos; in a small space, the differences can lead to a homography that isn't exactly possible to match. This is why homographies work particularly well in panoramic photos, since slight shifts in hand position affect the resulting photo minimally.

Part A Final Thoughts

After completing this part of the project, I am quite impressed by the amount of precision with simply selecting the keypoints that I did. Although I did not 100% get the mosaicing part, I was excited to see that my images fit together nicely even while manually completing the mosaic! I am excited to move to the auto-stitching portion of the project where I can continue to learn about these concepts.

Part B: Feature Matching for Autostitching

Detecting corner features in an image

For the purposes of showing the different parts of part B, I will be using images of Telegraph Ave which I found on a previous semester's 194 project here: https://inst.eecs.berkeley.edu/~cs194-26/fa20/upload/files/proj5B/cs194-26-acg/ . I found that the images on my room did not perform very well with the Harris corner detector & couldn't figure out exactly why; other images performed well. The original images are shown below:

Telegraph 1

Telegraph 2

I then used the provided Harris Corner Detector code to identify the corners in the image. I replaced the original peak_local_max with corner_peaks, which performed much better for my purposes. The results of corner detection are shown below:

Telegraph with Harris Corners

Then, in order to uniformly distribute keypoints and prepare the next part with the generally "stronger" corners, I implemented the adaptive non-maximal suppression (ANMS) algorithm to detect points that fit these characteristics. I then chose the top 300 points as generated by ANMS:

Telegraph after ANMS

Feature Descriptor Extraction & Matching

Next, my goal was to find points that matched best between these two images. In order to do this, I iterated through all pairs of points generated by ANMS, extracted an axis-aligned 40x40 patch centered at the keypoint, scaled it down to 8x8 and compared using a simple SSD comparison. I found that I was able to get more success on matches by applying a 5x5 Gaussian filter before running the matching algorithm. The results of matching are shown below:

Telegraph Matched Points

RANSAC Homography

While the matches wer quite good in general, there are bound to be outliers sometimes. I used RANSAC to a) remove such outliers, and b) choose only the best 4 matches to compute the homography. This will help the homography be as realistic and well matched as possible. My RANSAC algorithm came out with the following results:

Telegraph after RANSAC

Final Mosaicing + Other Examples

Once these points were determined, we could use the mosaicing method developed in part A of the project. My final results for Telegraph, as well as other results, are shown below. My results aren't 100% perfect, but I am proud of my work!

Car 1

Car 2

Car with Manual Stitching

Car with Auto Stitching

Telegraph 1

Telegraph 2

Telegraph with Manual Stitching

Telegraph with Auto Stitching

Desk 1

Desk 2

Desk with Manual Stitching

Desk with Auto Stitching

Part B Final Thoughts

I thought the coolest thing was when RANSAC was perfectly able to detect matching points!