Project 6: [Auto]Stitching Photo Mosaics

By James Smith

Introduction

If a picture is taken of a single object from 2 different perspectives, the projective transform between these two points of view can be calculated from the images. 4 correspondence points are needed to be able to recover this projective matrix. If we use more points, we can use linear least squares to get a better predicion of what the projection matrix H is. In this project we explore a few uses of this type of algorithm.

Image Rectification

If we want to see what a planar surface woud look like if we were looking at it strait on, we can rectify that surface into a square shape using the calculated matrix H. Here are two examples of that.

Building Mural

This building mural painting was taken in south Berkeley.

Original image.

Same image with the painting rectified

Cropped view of the painting

Video Game

This image was taken in my living room and shows an example of rotating in multiple axes.

Cover of a video game.

Same image with the cover rectified

Cropped view of the game

Mosaics

Using correspondance points, we can determine the projection from one image to another image in the same area. We can then project one of the images using H and blend the two images together using a linear blend to create a Mosaic Panorama.

Golden Gate Fields

These images were taken at the Golden Gate Fields horse race track.

Right image

Center image

Right image

Mosaic image from the above

Hearst Memorial Mining Building

These images were taken in the lobby of the HMMB building.

Right image

Center image

Right image

Mosaic image from the above

Cesar Chavez Park

These images were taken Cesar Chavez Park at the Berkeley Marina. I believe the quality of these is lower because it was outside and the plants and trees were moving slightly. Also there is a remarkable amount of detail on the ground, so any slight offset will be clear..

Right image

Center image

Right image

Mosaic image from the above

Feature Matching and Autostitching

We can auto stitch the mosaics by detecting feature points in the images, matching features across images, and computing the homography matrix H from the auto matched points. We use the Harris Corner Detector to find keypoints in the images. On average, there are several thousand corners detected in the images, so we need to find a way to filter them out.

Adaptive Non-Maximal Suppression

Adaptice Non-Maximal Suppression (ANMS) is an algorithm that can be used to get strong points that are spread out evenly in the image. The algorithm works by suppressing points based on the strength of the corner and only accepts those that are maximal in a small neighborhood. We can choose how many final points n that ultimately want. Several choices for n are shown below. An n of 400 was chosen for the rest of the project.

No suppression

n = 100

n = 400

n = 1000

Feature Descriptors and Matching

Corner features can be described by the pixels that surround it. We calculate a descriptor patch for each feature by looking at a subsampled 8x8 area around each corner. These descriptors are then compared to all other descriptors on the second image, and the best two matches are chosen. The ratio of the SSD of the top two matches is calculated and only points whose first match is significantly better than the second match are kept. Results from matching between two images from Golden Gate Fields are shown below.

Feature matching between the left and center images of the Golden Gate Fields mosaic.

RANSAC

As can be seen above, some features are matched incorrectly. We cannot estimate the homography matrix from these points because Linear Least Squares is not robust to outliers. Random Sample Consensus is an algorith that can efficiently reject outliers so that we only use good matches (inliers) to calculate H.

Results from RANSAC show a much better matching between features.

Auto-stitched Mosaics

Now that we can automatically detect a set of correspondence points between two images, we can return to the previous problem of creating mosaic panoramas. Results are shown below of 2 of the previous panoramas and also a new one.

Golden Gate Fields

Mosaic from hand picked points.

Automatically stitched mosaic.

Hearst Memorial Mining Building

Mosaic from hand picked points.

Automatically stitched mosaic.

Berkeley Institute of Design

These images were taken inside of the Berkeley Institute of Design lab.

Right image

Center image

Right image

Mosaic image from the above. There is no hand stitched mosaic for this scene.

Summary

The main thing that I learned from this project is that it can be hard to come up with a set of hyperparameters that work for all scenes. For each scene, I had to hand tune the parameters in some way to produce good results. I think some of these tuning procedures could be automated.