Project 5: [Auto]Stitching Photo Mosaics

Jason Zou

In this project, we use homographies to warp images onto the same projective plane and stitch them together to form a mosaic. We first do this by manually picking correspondences by hand and then blending, and then we try to do it automatically by matching image features.

Homographies

A homography is a projective image transformation that can be used to help warp images into a different perspective. This can be useful for getting a better look at some part of the image, such as was done for the b/w floor pattern by Criminisi, or for stitching together image mosaics. To recover a homography, we can manually define a correspondence of at least four points between the original image and its shape in the destination plane, then find the least squares solution to the linear system of equations derived from changing p'=Hp, where p' and p are the 3xN matrices of point correspondences, to be of the form Ax=b, where x is a 8x1 vector containing the 8 unknown coefficients of H (we can set the 9th to 1) and A and b are of the shape 2Nx8 and 2Nx1 respectively.

Rectification

To start off our exploration into homographies, we first try to use them to rectify an image. The following two pictures are of the interior of a room, the first taken off the web and the second taken of my personal kitchen. Below them are their rectifications, done using inverse warping.

... ...
... ...

Blending Mosaics with Manual Correspondences

Now we can take it a step further and try to blend two images, taken of the same object/structure but from different perspectives, into one single image mosaic. The images below are taken of a large window in my apartment.

... ...

Thankfully, there are nice square shapes to draw our correspondences between the two images. After projecting the bottom perspective into the top and using a basic alpha mask to blend the two together, we get the following result.

Automatically Detecting Correspondences

While manual correspondences provide nice results, they are pretty time consuming to determine. Luckily, we have a way of finding a good set of correspondences automatically. The results below are produced using a single-scale (but rotation invariant) version of the MOPS algorithm and manually defined alpha mask which attempts to smooth the transition between images. Each picture required slight adjustments to the parameters for Harris Corner detection, Outlier Rejection, and RANSAC. Amazingly, the automatic correspondences seem to be just as good as my manually defined ones!

Click on a picture to view it in full resolution.

Manual Automatic Harris ANMS
... ... ...
... ... ... ...
... ... ... ...

Bells & Whistles

A cool thing that I saw someone do as an extension of homographies was to warp a pedestrian crossing sign onto a sidewalk and blend it so that it looked like it was really there. Here I try to replicate that idea by creating wallpapers on my wall that I don't actually have by carefully using alpha masks along with the homographies.

Things Learned

From part A, by far the most important thing I learned was how to properly define projective transforms and set up the linear system of equations. I spent a few hours hardcore debugging my code before I realized what was wrong (I had failed to account for w properly), but in the process learned a lot about how each parameter in the warp affects the final image transformation. As for the coolest thing, I think seeing the results of homographies that you make yourself really shows how powerful such a relatively simple concept can be.

From part B, the entire algorithm was basically pretty cool to me, as it combined a bunch of techniques we've learned throughout this course and applied it to this specific problem. Harris Corners + ANMS were probably the cooler of the steps for me, as it was interesting to see how the suggested points varied as I tuned the various parameters of the Harris Corners method.