CS194 Project 5: Image Warping and Mosaicing

Cesar Plascencia Zuniga

Overview

At one point or another most of us have taken a panorama on our phone. We slowly pan the camera from side to side while a photobomber creeps into the shot and ruins it. The key to making panoramas work is homography. In this project, we will explore the various applications of homography and eventually develop our own panoramas! Here are example images that we will be working with.

Recovering the Homography

As it turns out, any transformation between two shots of a panorama can be related by a homography matrix. The relationship is p'=Hp As long as we have a minimum of four points, we can retrieve H.

We restructure the problem as Ah=b as seen below. Then we use least squares to recover H when accounting for more than four points.

Warping and Rectification

Now that we can recover homographies, we can construct a warp function that takes an image and a homography, then warps the image by that homography. We implement the function similarly to how we handled inverse warping in project 3. Afterwards, we can implement image rectification. This is when we have an image that is planar, but the image appears crooked in a way. We can 'straighten' out the image by warping it to be frontal-parallel. This is done by selecting points in the image that we know the shape of (such as squares/rectangles) and warping the image to those known shapes.

Original Image
Rectified View of Mission Front
Original Image
Rectified View of Deck Floor

Image Mosaicing

Let us now use the power of warping and homographies to stitch together image mosaics! The technique used here is to first take a "left-most" image and stitch images onto it one at a time. Each stitch creates a bigger image. A stitch involves selecting the corresponding points between two images, warping the "right" image onto the first, then adding the pixels onto the base image. We can see the results below.

Feature Matching and Autostitching

Previously, we had to select correspondence points between images by hand. This is less than idea since we do not have the best perception of sub-pixel accuracy. This led to slightly misaligned images that greatly affects our homography. To fix this, we will find correspondence points and homographies automatically. This way, we can achieve near perfect warps.

The first step is to automatically find corners using the Harris Interest Point Detector. At the high level what is going on is we are finding areas of the image that have drastic changes both vertically and horizontally. Here are the results from running Harris on our first picture of the Mission.

Santa Barbara Mission with Harris Corners

Once we have our Harris Corner Points we apply Adaptive Non-Maximal Suppression to extract the 'important' corner points from our image.

ANMS 250

Next, we need to take these corner points and extract the square of pixels around them called the feature descriptor. For example, if we have a point at the top of a pillar, we can make a feature descriptor for this patch of pixels and downsample the patch to be 8x8. We also normalize this patch.

Original Patch
Downsampled and Normalized

Once we have the feature descriptor patches, we can start finding corresponding patches in other images. Finding these corresponding patches equates to finding corresponding corner points among images. As we already know, once we have corresponding points we can find homographies and warp the images to create a mosaic!

Matching Corner Points

Afterwards, we simply need to send these points to RANSAC in order to find a robust estimation for our homography matrix. Essentially, we are trying many random points to feed to our homography function and selecting the best points. Once we have the homography, we create the mosaics as before and now we can compare our results!

Manual Mission Mosaic
Automatic Mission Mosaic
Manual Wharf Mosaic
Automatic Wharf Mosaic
Manual Seagull Mosaic
Automatic Seagull Mosaic

Reflection

As you can see, the results from our automatic stitching were quite improved. I do not believe humans were meant to have super-pixel accuracy. Overall, I enjoyed this project a ton! The coolest thing by far was learning just how much information you can extract from a 2d image. Whether it be from rectification or automatically finding corresponding points, there is a lot of power in a 2d matrix.