Project 6: Autostitching Photo Mosaics (Panoramics)

CS 194-26: Image Manipulation and Computational Photography, Fall 2018
Jonathan Fung, CS194-26-abs

In this project, we attempted to rectify images and mosaic images together by using projective warping. These homographies, as they are called, have 8 degrees of freedom. After choosing a few correspondance points by hand from the original photo to the desired perspective, we solve for the homography matrix, which allows us to warp all points from the original photo to the new perspective.

In order to rectify an image, I selected four points which form the corner of whatever I was trying to rectify into a rectangle. I then formed a rectangle with the other set of correspondance points, then transform the whole image.

In order to mosaic images together, we first define a handfull of correspondance points between the two images. We then solve for the homography matrix using least squares, and then warp one image into the perspective of another. Finally, we overlay the two images to create a mosaic'd image.


Part 1. Image Warping and Mosaicing

Part 1a. Image Rectification

For rectification, we select 4 points that are the corners of a rectangle plane, then map them into a rectangle. Note that after we solve for the homography matrix H, we have to take the inverse because we use inverse warping, cycling through the pixels of the destination and applying the inverse transform H_inv. Here are a few results.

A Paper.
A Poster.
A computer.
Paper is now a rectangle, but the words on the hair gel are distorted.
Easier to see the poster now!
Now we can see what's going on!
Windows.
Rectified the lowest window.

Part 1b. Mosaics

Now that we know our warping and homographies work, we move on to mosaic'ing together two images. We first take one image, warp it to the perspective of the second photo, and then blend the two together. I tried 3 methods of blending, 1) Overlapping, 2) Maximal regions, Linear Blending. Overlapping entailed overwriting existing pixels. Maximal regions entailed taking the maximum intensity of the pixels that overlapped, and linear blending involved scaling an alpha value at 0 towards the edge of the overlap and assigning the pixel value a weighted sum.

Left side of a room.
Right side of a room.
Left side of a room warped to perspective of right.
Merged with the overlap technique. 6 points of correspondances were used, leading to minor imperfections (chair leg)
Merged with the overlap technique. 12 points of correspondances were used, a much better result.
Merged with the maximal value technique (between overlaping regions)
Merged with mean-based alpha blending linear technique. Chair legs now look perfect, but the window has imperfections.
Merged with min-based alpha blending linear technique. Chair legs now look perfect, but the window has imperfections.




Left side of a pic of the Campanile by Bechtel.
Right side of a pic of the Campanile by Bechtel.
Left side of the scene warped to perspective of right.
Merged with the overlap technique
Merged with the maximal value technique
Merged with the linear blending technique




Left side of a Classroom, with me!
Right side of a Classroom, also with me!
Left side of the classroom warped to perspective of right.
Merged with the overlap technique. Border on the left side.
Merged with the maximal value technique. Lighting was different between the two, so this method was suboptimal, when it performed well in previous pictures.
Merged with the linear blending technique. Performed the best, because of different lightings.

Part 2. Feature Mapping for Autostitching

With the completion of Part 1, we know that we can stitch together photos by manually selecting correspondance points between the two images and warping them together. However, automating the generation of correspondance points would save us a lot of time! We accomplish this by use of the Harris Corner Detector algorithm on each image to find specific corner pixels of an image. Next, we use Adaptive Non-Maximal Suppression (ANMS) in order to obtain the corner pixels with the most strength. Then, in order to pair up correspondance points, we create feature patches around each corner, and use the nearest-neighbors technique to pair up feature patches. Finally, we use RANSAC to eliminate outliers and obtain a set of points for a stable homography.

Part 2a. Harris Corner Detection

For Harris Corner Detection, the general idea is that at any corner, moving a window in any direction will change the image a lot, whereas an edge will change one direction a lot, and a flat image will not change much in any direction. Here, we display some images with the harris points overlayed on top.

Mouse over for Original Image
Mouse over for Original Image
Mouse over for Original Image

Part 2b. Adaptive Non-Maximal Suppression

Wow, that's too many points to work with. We notice that if two points are close to each other, that is redundant information. Thus, ANMS works by selecting a uniform spread of points to eliminate redundant information by selecting the top points with the greatest suppression radius, or radius of points they are stronger than by a certain threshold. (strongest points in a neighborhood). We select the top 200 points in ANMS by a 90% threshold here.

ANMS points shown. Mouse over for Harris points.
ANMS points shown. Mouse over for Harris points.
ANMS points shown. Mouse over for Harris points.

Part 2c. Extract Feature Descriptors

Now, we need to pair up these correspondance points. We extract small patches around each correspondance point (40x40x3), gaussian low pass filter them, downsample to 8x8x3, and transform them to zero mean and stdev 1. These patches can now be SSD'd against each other to measure how simalar they are. Note that we did not account for rotational invariance in our patches because in panoramics, there are no rotations. Note how the general features are preserved, e.g the diagonal line.

Original Patch extracted from image
Downsampled and bias normalized.

Part 2d. Matching Correspondance Points with Feature Patches and Nearest Neighbors

To pair Correspondance Points, we take each point's feature patch, and find the closest 2 feature patches with respect to SSD. By examining the ratio of the the closest SSD value and the second closest SSD value, we can determine if the point is a possible correspondance. In this case, the cutoff I used was 0.4. If the ratio is lower than 0.4, we know that the both points match signifigantly stronger with each other than any other point, meaning that it is a possible correspondnace point. Here are the points visualized. Note that there are still a couple of outliers.

Part 2e. Random sample consensus (RANSAC)

Now, to discard the outliers, we perform RANSAC. Over a large number of iterations (in this case, 1000), we select 4 random correspondance point pairs, compute a homography H, warp every point in one image to the plane of the second, and count the number of inliers, which are defined as points with a SSD difference between (x', y') and (x, y) less than a certain threshold (in this case, 100). After all the iterations, we compute a homography out of our group of inliers and use those to warp the whole image.

Mouse over for Pre-RANSAC correspondance points. Notice how the outliers are eliminated.

Part 2f. Mosaicing

We then stitch together the two images using the methods from Part 1.

Mouse over to compare with results from manual stitching.







Part 3. Summary

I never questioned how the panoramic app in my phone worked, and I learned a lot about how panoramics were generated. It was fun to simulate the algorithms from scratch! I also learned to appreciate the power of RANSAC, and how efficient it was in discarding outliers. The nearest neighbors technique about comparing the first and second best in order to determine correspondance points was super neat as well.