Part 1 - Image Warping and Mosaicing

The goal of this project is to create tools to combine multiple photographs taken from different perspectives into a single seamless image.

The main images for testing and demonstration in this images were these two images of the Channing-Bowditch Apartments:

To transform the images as if they were taken from the same perspective, we have to be able to calculate a homography from one set of points to another set of corresponding points. To test this we can first try using a homography to rectify the image on the right.

We define a set of points, in red, on the original image, and specify a set of blue points that we want the red points to go to. The 3x3 homography matrix, which defines a transformation from the blue points to the red points is then calculated. Using the homography matrix, we can then use an inverse warping and bilinear interpolation to produce our rectified image:

We can repeat this rectification on other images:

Next we can define a set of points on both of our images of the building:

Once again, we can calculate a 3x3 homography matrix that transforms the points from the right image to the points on the left, then use this matrix to warp the image on the left to the perspective of the image on the right. As a sanity check, we transform the points from the right onto the image on the left, and see they are more or less in the same positions:

We then use an inverse warping to transform the image on the left:

A naive overlaying (with non-right edges cropped from the left) of the two images, creates a clear line. (The non axis-aligned edges of the combined image to create a less stitchy-looking image)

To make the combination look nicer, we can blend the two together using alpha blending, creating a softer and less noticeable seam:

We can apply this to other images too:

Part 2 - FEATURE MATCHING for AUTOSTITCHING

Now that we can manually select points and stitch images together, we can now try and automate the process. We start by first selecting a large number of points of interest using a Harris Interest Point Detector, which (roughly speaking) detects corner-like features in the image.

The Harris detector returns an overwhelming number of points, many of which may have no significance or match in our other images. To selected the only strongest points of these, I implemented an Adaptive Non-Maximal Suppression (ANMS) algorithm which selects a set of strong points well distributed spatially across the image. ANMS performs better than other methods, such as max suppression, since it tends to not select clumps of points.

To match interest points between each image, each interest point is given a descriptor calculated by taking a 40x40 pixel patch around each point, low pass filtering each patch, then sampling every 5th pixel to create an 8x8→64 number long descriptor vector. To evaluate the strength of a match between two points we use the Euclidean distance between the vectors.

To eliminate weak or mistaken matches, we only accept a match if the ratio of the strength of the strongest match to the strength of the 2nd strongest match is below a specified threshold. The reasoning behind this is that true matches are significantly stronger than false matches, and if the strongest match is close in scale to the 2nd strongest, both are likely false matches. Iterating through all possible matches to select ones, create a set of likely matches between the two images.

However, at this point there might still be mistaken matches between our points. I used RANSAC, which repeatedly fits a homography to randomly 4 matched pairs then repeats keeping track of the best homography it encounters, to find a robust homography. For a sanity check, the matched points from the right image are tranformed using the homography matrix onto the left image. Clearly, the transformed points (blue) are close to the matched points (red).

Inverse warping the left image using the homography matrix, then blending the images together just like in part 1, we get the following:

Cropping, we can get a seamless image.

Mini Bell and Whistle

Extending the method, we can even combine 3 images!

Comparing to our manually generated mosaics, the quality of the autostitched mosaics are the same or even better.

Bell and Whistle: Purging Pesky People

To create our mosaics, we usually have to take pictures over a period of time over which pesky people, animals and cars may be moving around. This can influence the quality of our reconstruction (both by possibly confusing the algorithm and also by inherently being ugly stains on the landscape). Luckily humans are notoriously impatient and do not stay in a single place for long, and we can use this fact to remove them from the landscape.

Taking 5 photos over time from each perspective:

Taking the median over the 5 images of each pixel however, we can eliminate all of those pesky people!

We can now create a beautiful, human-free panorama: