[Auto]Stitching Photo Mosaics

In this project, we will explore estimating homographies in order to rectify images and produce mosaics. Check it out!

Calculating Homographies

The core of the magic that produces the outputs you are about to see is the Homography matrix, which maps between two projective planes with the same center of projection. This allows us to take an image with one perspective and warp to fit another perspective, which is why it is useful for rectification and mosaicing. A Homography matrix looks like the following:

In order to figure out what this matrix looks like between two images, we need to get a set of points that corresponds between two images. In the images below that I will use for mosaicing, there are lots of objects present in both scenes that correspond to each other and this Homography matrix should be able to tell us how.

There are only 8 degrees of freedom in this matrix, so we can solve for the matrix just using 4 points. However, it is better to use more points when possible to get a better estimate for the matrix. In order to do this, you can set this problem up as a least-squares problem and learn the matrix from all of your points. The formulation is as follows, where the final entry of the matrix will always be set to 1

Image Rectification

Let's apply this concept to be able to rectify images. What this means is taking an image with planar surfaces and warping them so that the plane is frontal-parallel. This is basically straightening out the planes in the image. Since there is only one image here, the way we compute the homography is by labeling the corners of the planar object and then mapping them into some defined shape of my choosing. In these images, I chose to warp them into squares. Let's take a look:

Before

After

Before

After

Mosaics

We can take the same method behind image rectification and apply it to create image mosaics. The idea here is that we can take corresponding points between images and warp all of the image into a single perspective and then blend them together to create one large image in a single perspective. Let's take a look at some results:

What I learned

The coolest thing I learned was how powerful homographies were as a transformation. It almost feels like magic to see the nice alignment of the images in the mosaic. It all fit together so nicely - it made the project quite worthwhile!