﻿

image warping and mosaicing

OVERVIEW

In this project, I explored different aspects of image warping, namely image mosaicking. I performed image warping by taking at least 2 photographs, registering, projective warping, resampling, and compositing them. A key step in performing mosaicing was computing homographies and using them to warp the images. Finally, I used corner detection and feature matching to perform automatic image stitching (as opposed to using selected matching points).

PART 1

RECOVERING HOMOGRAPHIES

Before warping images into alignment, I had to recover the parameters of transformation between pairs of images. This transformation is a homography, which relates the transformation between two planes (images are only related by a homography if they are viewing the same plane from a different angle). The homography matrix H is a 3x3 matrix with 8 degrees of freedom.

Given a point (x,y) in image 1, and its corresponding (x’, y’) point in image 2, we can find H using the following equation (attaining H using least squares):

WARPING AND RECTIFICATION

Warping an image with H transforms the image into the desired perspective, as the homography matrix maps the source points to the desired points. I used inverse warping with linear interpolation to avoid aliasing during resampling. Finally, I was able to rectify images. I chose two sample images with planar surfaces (one square and one rectangular) to warp so the plane is frontal parallel. I used ginput to select 4 points in each image, and defined the corresponding (x’,y’) corners by hand to be square and rectangular, respectively.

Here are the results of rectification:

 original rectified

BLENDING INTO A MOSAIC

We now have the capabilities to take 2 images, warp them, and blend them to create an image mosaic. Where the two images overlap, I used weighted averaging at every pixel, and the intersection linearly goes from being most similar to the left image to most similar to the right image (by linearly increasing the alpha value). Here are the results:

 Left image Right image

 Warped left image Warped right image Composite

Learning how to rectify images was my favorite part of this assignment because I’ve always wondered how it was performed in photo editors and document scanning apps like Scannable.

PART 2: AUTO-STITCHING

For the second part of the project, I used corner detection and feature matching to perform auto-stitching.

CORNER FEATURE DETECTION

To detect image corners, I used the provided Harris corner detection algorithm, using corner_peaks instead of peak_local_max. These are the results of corner detection:

Using Adaptive Non-Maximal Suppression, I selected the “strongest” corners to reduce the number of harris points s to 500 using the following equation from this paper:

For each point, I calculated the distance to all other points (with a corner strength <= 0.9 * corner strength of other points) and kept the points with the largest minimum suppression radius (ri). This means we have fewer clusters of points, and more evenly distributed points. Here are the harris points after applying ANMS:

FEATURE MATCHING

Next, I extracted feature descriptors to match across images. For each point, I looked at a 40x40 window surrounding it, applied a Gaussian blur, then down-sampled it to be an 8x8 patch. The patches were normalized to have a mean of 0 and standard deviation of 1. Here are a few example patches:

Next, I matched these feature descriptors across the two images. As in the paper, for each feature descriptor in the first image, I computed the distance between it and every harris point in the second image. I calculated the ratio of the closest point to the second closest point (least distance). If the ratio was below 0.275 (used this ratio for better predictions, otherwise a point from the two different tables in the left and right image was matched), the features are a match. This gave pretty accurate results:

RANSAC

Finally, to find the optimal homography between the two images, I implemented RANSAC (RANdom SAMple Consensus). Over 1000 iterations, the algorithm randomly selects 4 feature points from both images, computes the homography over those 4 points, counts the resulting inliers, and saves the longest list of inliers. The inliers are then used to compute the final homography matrix between the two images, into which they must be warped.

Now, our auto-mosaicing algorithm is ready.

AUTO-MOSAICING

Below are a few outputs of my auto-mosaicing algorithm: warped left images, warped right images, and the blended mosaic. I used the same warping and blending techniques as in part 1.

LEARNINGS

I really enjoyed this project because I learned a lot of useful new concepts and algorithms: from image rectification to homographies to ANMS, feature matching, and RANSAC. It also gave me more confidence in my ability to understand and implement cutting-edge algorithms from research papers.