Project 4a: Image Warping and Mosaicing

Overview

In part a of Project 4, we implemented mosaicing of images of a common scene. In order to do this, we establish keypoint correspondeces between images, and then solve for the homography, or homogeneous transform, in order to warp pixels from one image to another image. In order to solve the homography, we place the keypoint correspondences in a matrix, where every 2 rows corresponds to a keypoint pair. Our data matrix, P, has the dimensions (n*2)x9.

We can then solve for the homography vector H of dimensions 9x1 by solving for the null space of P (since we want the equation PH=0 to hold) and reshape it into our 3x3 homography matrix. We then perform mosaicing by morphing all images to a common plane Once we have these homographies, mosaicing involves several steps:

1. Forward warping the corners of each image to the common plane to find the bounding polygon from which to interpolate from
2. Inverse warping from inside the bounding box on the common plane to retrieve pixel intensities from the original images
3. Blending to reduce the harshness of borders separating mosaiced images

Improvements (Bells and Whistles)

Keypoint Refinement

The process underlying all the homography calculation is the keypoint selection and correspondence. Having keypoints that are slightly off results in misaligned images. One method I used to combat this was refinement of the keypoints after manual selection. I did this using a process inspired by keypoint descriptors such as SIFT, doing a grid search of patches around the manually selected keypoint, computing a descriptor, and then selecting the keypoint with the descriptor most similar to the reference point. The descriptor is generated by extracting a patch around the point, calculating the gradient magnitude and direction at each pixel in the patch, binning the pixels based on gradient direction, and then summing up the magnitudes in each bin. I also introduce orientation invariance by aligning each patch such that the average gradient is always pointing in the same direction.

Image Mosaicing Results

Below are results of performing image mosaicing.

Image A	Image B	Image Mosaic

Plane Rectification

One last thing we implemented was plane rectification, both to sanity check our warp as well as develop some interesting results. Below are a couple of my rectification results. The first result really displays the shortcomings of our homography approach. It requires a planar assumption, and when the planar assumption is violated, the warp becomes very odd.

Original Image	Rectified Image

What I learned

I think the thing that I've learned the most is a newfound respect for panoramas in cameras. It seems very difficult to properly blend the images together, and its very difficult to algorithmically determine if there are errors in our mosaic. I also experienced the difficulty of working with very large images and the restriction my computer's memory imposes.