Project 6: [Auto]Stitching Photo Mosaics

Yena Kim - cs194-26-aax




Part A: Image Warping and Mosaicing

Overview

In this project, we compute homography matrices and use them to rectify images as well as create mosaics using multiple photos of the same scene. This part of the project focuses on warping the images into a mosaic using correspondence points found manually.

Homography

To compute the homography matrix, I find correspondance points in two images. I then use the equation p'=Hp, with p' being the points for one image and p being the corresponding points on the other, to find H.

Warping Images

To warp one image to another, I compute the homography matrix using their correspondance points. I find the size of the final image and create an empty image with just the image I'm warping to in the middle. I then do inverse warping for the other image by finding the locations on the new image where it will warp to and finding the corresponding points on the original.

Image Rectification

Using the above technique, I can rectify an image by finding the corners of the object and setting its correspondance points to be the corners of the desired shape. I then find the homography matrix and warp the image to the desired shape.

poster original
poster rectified (cropped)
art original
art rectified (cropped)

Blending and Mosaicing

Creating a mosaic of multiple images is similar to rectifying an image, with the difference being that instead of warping to a shape, I'm warping the image to another image. Now that I have both images placed appropriately, I can blend the images together to create the final mosaic. In the overlap region, I found the maximum pixel value between the two images and used that for the final mosaic to blend them together.

winter original 1
winter original 2
winter mosaic
winter mosaic (cropped)
point reyes original 1
point reyes original 2
point reyes mosaic
point reyes mosaic (cropped)
hotel original 1
hotel original 2
hotel mosaic
hotel mosaic (cropped)

What I Learned

It is pretty amazing what we can do with multiple images and a bit of math. I think it's super cool that by just computing a homography matrix I can change the perspective of images as well as stitching images with different perspectives together into one mosaic.

Part B: Feature Matching for Autostitching

Overview

As seen above, manually finding the correspondence points can lead to a homography matrix that is a little bit off, so the warped images do not fit perfectly together into a completely blended mosaic. This part of the project works to automatically find matching correspondence points given two images.

Detecting Corner Features

Using the Harris Interest Point Detector, I was able to automatically find the corner points of the images. As you can see below, there are many corner points detected.

hotel harris corners 1
hotel harris corners 2

Adaptive Non-Maximal Suppression

We want to reduce the number of corner points for computational efficiency while keeping them well-distributed instead of focusing on a certain area. We can achieve this using Adaptive Non-Maximal Suppression. This allows us to select the top however many, in this case 500, points based on their minimum suppression radius:

Using ANMS also allows me to keep the points well-distributed across the image, as shown below.

hotel after anms 1
hotel after anms 2

Feature Descriptor Extraction

Now that we have our points, we can extract feature descriptors for each of them. I got a 40x40 window around each of the points and downsampled them so they became 8x8. I then normalized them by subtracting the mean and dividing by the standard deviation.

Feature Matching

Now I want to find the correspondence points, aka features that match in both images. I do this by computing the distances between each of the feature descriptors. I then find the nearest neighbor and the second nearest neighbor and compute the ratio of distance of first nn over the distance of the second nn. I then consider the feature pair if this ratio is under a certain threshold. Below are examples of the features that were matched together in this fashion.

hotel after feature matching 1
hotel after feature matching 2

RANSAC

As you can see, not all the features match up correctly, so in order to weed out those outliers, we use RANSAC to find the best possible set of feature matches. RANSAC involves iterating a large number of times and picking a random subset of 4 features and computing a homography matrix with those. I then compute the errors for all the features with this homography matrix and select those that are under a certain threshold. I count how many and keep track of the largest set of inliers. After all the iterations, I keep the largest set of inliers and use that as my final set of matching features and use them to compute the homography matrix used to warp the images. Below are the results after RANSAC.

hotel after ransac 1
hotel after ransac 2

What I Learned

I thought autodetecting the corners would be the hardest part of this project, but that was actually the easiest part. It was how we manipulated the corners to find matching features that took a while. However, the end result is cool and it's worth not having to manually find all the correspondence points.