[Auto]Stitching Photo Mosaics

In this project, we will explore estimating homographies in order to rectify images and produce mosaics. Check it out!

Calculating Homographies

The core of the magic that produces the outputs you are about to see is the Homography matrix, which maps between two projective planes with the same center of projection. This allows us to take an image with one perspective and warp to fit another perspective, which is why it is useful for rectification and mosaicing. A Homography matrix looks like the following:

In order to figure out what this matrix looks like between two images, we need to get a set of points that corresponds between two images. In the images below that I will use for mosaicing, there are lots of objects present in both scenes that correspond to each other and this Homography matrix should be able to tell us how.

There are only 8 degrees of freedom in this matrix, so we can solve for the matrix just using 4 points. However, it is better to use more points when possible to get a better estimate for the matrix. In order to do this, you can set this problem up as a least-squares problem and learn the matrix from all of your points. The formulation is as follows, where the final entry of the matrix will always be set to 1

Image Rectification

Let's apply this concept to be able to rectify images. What this means is taking an image with planar surfaces and warping them so that the plane is frontal-parallel. This is basically straightening out the planes in the image. Since there is only one image here, the way we compute the homography is by labeling the corners of the planar object and then mapping them into some defined shape of my choosing. In these images, I chose to warp them into squares. Let's take a look:

Before

After

Before

After

Mosaics

We can take the same method behind image rectification and apply it to create image mosaics. The idea here is that we can take corresponding points between images and warp all of the image into a single perspective and then blend them together to create one large image in a single perspective. Let's take a look at some results:

This process of selecting points can also be automated by combing harris corner detection, adaptive non-maximum supression to select points of interest in an image. We can then use feature matching to find coresponding points between images.

Harris Corners

Harris Corners

Adaptive Non-Maximum Supression

Adaptive Non-Maximum Supression

Matched Points

Matched Points

Not all of these matched points are perfect correspondences - some are outliers! Therefore, we need to use RANSAC to compute the ideal homography - the one that fits the most inliers. Let's look at the automated mosaics. They turned out pretty well!! Compared to the automaic, the stitching a bit cleaner. Check it out!

Manual

Automatic

Manual

Automatic

Manual

Automatic

What I learned

The coolest thing I learned was how powerful homographies were as a transformation. It almost feels like magic to see the nice alignment of the images in the mosaic. It all fit together so nicely - it made the project quite worthwhile!