Project 6: Autostitching Mosaics

Credits to Mumu Lin for photos of San Francisco

Segment 1: Stitching Photos with Manual Correspondence

Part 1: Photo Collection

In order to obtain obtain some interesting photos for this stitching project, my friend Mumu and I took a day in San Francisco with a Canon EOS Rebel T3i camera. To take the photos, we stood at a fixed point, capturing images while rotating. The rotation was not as precise as could have been achieved with a tripod and some exposure settings were not locked, so some of the pictures came out slightly differently. However, I believe that we have obtained enough results to do some really interesting stuff.

Here are some sample photos we obtained. Most of these are part of multiple sets of redundant photos:

Bubble Maker
Cable Car
Pier 39

For simplicity, the two photos I will start working with are taken from Berkeley BART. I already defined 18 points of correspondence for making a mosaic out of them, which I will use at the end of this segment.

Berkeley BART Left
Berkeley BART Right

Part 2: Recovering the Homography

The goal here (similar to project 4) involves finding a 3x3 transform matrix used to translate pixels across the canvas. Such involves solving a system of linear equations to find a set of variables [h11, h12, h13, h21, h22, h23, h31, h32], with h33 constrained to 1.0 for normalization. That means that a correspondence needs at least 4 (x, y) points.

Each source point (x, y) will be used to create a row in matrix A along with the destination point...

[x1, y1, 1, 0, 0, 0, x1 * px1, y1 * px1]

[0, 0, 0, x1, y1, 1, x1 * py1, y1 * py1]

... with b being [px1, py1, ...].

Since some systems might be overdetermined (i.e. more than 4 points) I use numpy's least square solver for this task.

Part 3: Rectifying by Warping

Now that the homography matrix can be retrieved, we have to define a transform that applies to every pixel of the image. To apply a homography, dot product H * [x, y, 1] to receive w * px, w * py, w, the operation I will define as warp(x, y) -> px, py.

  1. Find a bounding box to make the destination image by applying warp to the corners of the image.
  2. Map pixels in the destination image to pixels in the source image using inverse warp (H^-1). Using forward warp to map source to destination might cause black empty space.

Here is the result on a secret door from a shabu shabu restaurant in Anaheim, warped to a square.

Perspective Secret Door
Front Door (cropped)

If you know the proportions of objects, you can also rectify into other shapes. For example, this Vanguard card has ratio 4:3.

Draconic Overlord
Rectified (cropped)

Part 4: Photo Mosaics

This part is much the same as the previous, except the points will be overdetermined this time and we'll need to deal with blending multiple images.

Per some suggestions from other classmates, for intersecting portions of images, I used the max function to decide which pixel to use and it turned out pretty well.

Bart Left
Bart Right
Bart Mosaiced
Candy Shop Left
Candy Shop Right
Candy Shop Mosaiced

You'll notice the mosaic isn't quite perfect for the candy shop, due to the level of detail here. Better precision would require more correspondence points. Additionally, the guy in the background moved slightly, causing him to ruin part of the illusion. As you probably guessed, these mosaics are merely an approximation of the same scene, as there is a variable of time involved.

Again, defining correspondences that precisely transform the picture is hard:

Cable Car Left
Cable Car Right
Cable Car Mosaiced

The BART one is probably the most convincing due to the simplicity of the geometry. In the next portion of this project, we'll see how to automatically (and hopefully better) define the correspondence points to make a mosaic.