# Project 6: Autostitching Mosaics

Credits to Mumu Lin for photos of San Francisco

## Segment 1: Stitching Photos with Manual Correspondence

### Part 1: Photo Collection

In order to obtain obtain some interesting photos for this stitching project, my friend Mumu and I took a day in San Francisco with a Canon EOS Rebel T3i camera. To take the photos, we stood at a fixed point, capturing images while rotating. The rotation was not as precise as could have been achieved with a tripod and some exposure settings were not locked, so some of the pictures came out slightly differently. However, I believe that we have obtained enough results to do some really interesting stuff.

Here are some sample photos we obtained. Most of these are part of multiple sets of redundant photos:

 Exploratorium Bubble Maker Cable Car Pier 39

For simplicity, the two photos I will start working with are taken from Berkeley BART. I already defined 18 points of correspondence for making a mosaic out of them, which I will use at the end of this segment.

 Berkeley BART Left Berkeley BART Right

### Part 2: Recovering the Homography

The goal here (similar to project 4) involves finding a 3x3 transform matrix used to translate pixels across the canvas. Such involves solving a system of linear equations to find a set of variables `[h11, h12, h13, h21, h22, h23, h31, h32]`, with `h33` constrained to 1.0 for normalization. That means that a correspondence needs at least 4 (x, y) points.

Each source point (x, y) will be used to create a row in matrix `A` along with the destination point...

`[x1, y1, 1, 0, 0, 0, x1 * px1, y1 * px1]`

`[0, 0, 0, x1, y1, 1, x1 * py1, y1 * py1]`

... with `b` being `[px1, py1, ...]`.

Since some systems might be overdetermined (i.e. more than 4 points) I use numpy's least square solver for this task.

### Part 3: Rectifying by Warping

Now that the homography matrix can be retrieved, we have to define a transform that applies to every pixel of the image. To apply a homography, dot product `H * [x, y, 1]` to receive `w * px, w * py, w`, the operation I will define as `warp(x, y) -> px, py`.

1. Find a bounding box to make the destination image by applying `warp` to the corners of the image.
2. Map pixels in the destination image to pixels in the source image using inverse warp (H^-1). Using forward warp to map source to destination might cause black empty space.

Here is the result on a secret door from a shabu shabu restaurant in Anaheim, warped to a square.

 Perspective Secret Door Front Door (cropped)

If you know the proportions of objects, you can also rectify into other shapes. For example, this Vanguard card has ratio 4:3.

 Draconic Overlord Rectified (cropped)

### Part 4: Photo Mosaics

This part is much the same as the previous, except the points will be overdetermined this time and we'll need to deal with blending multiple images.

Per some suggestions from other classmates, for intersecting portions of images, I used the `max` function to decide which pixel to use and it turned out pretty well.

 Bart Left Bart Right Bart Mosaiced Candy Shop Left Candy Shop Right Candy Shop Mosaiced

You'll notice the mosaic isn't quite perfect for the candy shop, due to the level of detail here. Better precision would require more correspondence points. Additionally, the guy in the background moved slightly, causing him to ruin part of the illusion. As you probably guessed, these mosaics are merely an approximation of the same scene, as there is a variable of time involved.

Again, defining correspondences that precisely transform the picture is hard:

 Cable Car Left Cable Car Right Cable Car Mosaiced

The BART one is probably the most convincing due to the simplicity of the geometry. In the next portion of this project, we'll see how to automatically (and hopefully better) define the correspondence points to make a mosaic.