CS194-26 Project 6 - Alex Zhang

(scroll down for part B)

Image Warping and Mosaicing (Part A)

Overview

In this project, we use homographies to manipulate the perspective of images in order to create panoramas from multiple images stitched together. These photos must be shot such that the lens is at the same point of view without any translation.

Images

My source images are all shot from views around my apartment rooftop and inside my apartment.


Recovering Homographies

In order to recover homographies, we set up an overdetermined system of equations and solve for the homography parameters using least squares. To set up this system of equations, we use ginput to get correspondences. We need at least 4 to generate the homography, but having more makes the result more resilient to noise. For each (x,y), (xp, yp) correspondence, we get two equations that we add to our least squares matrix:
    a*x+b*y+c-g*x*xp-h*y*xp = xp
    d*x+e*y+f-g*x*yp-h*y*yp = yp
    
Using least squares recovers the a,b,c,d,e,f,g,h parameters of the homography and allows us to warp the images.

Rectification

This part is a simple application of homography that only requires one input image. By using knowledge of the image (i.e. a certain surface must be planar or rectangular) we can manually define the geometry of the points we are warping to and select the source points appropriately. Thus, this allows us to warp an image into the perspective of a plane in the target image by using these correspondences to create a homography for the warp.

Image of tablet
Rectified to perspective of tablet
Image of Elevator
Rectified to perspective of square 'R'

Panoramas

We use the principles of rectification to also create panoramics/mosaics. We select an anchor image and warp the other image into the perspective of the anchor by using the correspondences manually defined by hand. We use linear blending when stitching the images together to ensure a smooth transition where the apperance of seams is minimized. Some care is taken for calculating the size of the canvas.

BAMFA

Left
Anchor
Result

Campanile

Left
Anchor
Result

Anchor
Result

Summary

This project lets me appreciate the power of linear algebra. I think the example of rectification is very striking, that just multiply an image by a matrix can let us have a completely different view of the same photo. Also, learning the numpy functions to do this project was very rewarding too.

Feature Matching for Autostitching (Part B)

Overview

We learned from the last part that picking correspondences in images in order to create mosaics is a painful and tedious process, so in this part of the project we utilize the Harris corner detector to have the computer try automatically finding the correspondences for us instead. Using those correspondences, we should be able to create mosaics that are just as good as (or even better than) the ones we made by hand-picking points.

Campanile (from previous part)

Left Image of Campanile
Right Image of Campanile

Harris Corner Detector

We use the provided harris corner detector code and find all the possible corners in each image.

ANMS

In order to reduce the number of points, we only select the most important points in a given radius using the ANMS technique described in the paper. This also has a nice effect of spacing out the points more.

Feature Matching

For each point in the ANMS filtered point set, we extract 8x8 patches by sampling from a 40x40 neighborhood around each point. Each patch is normalized to have a mean of 0 and a standard deviation of one. We then find the SSD between each patch in the first image and each patch in the second image. We take the ratio of the first and second nearest neighbors for each patch and check to make sure that this is below a threshold. In our case, we picked .4 as the threshold. This yields a set of correspondences between the two images.

RANSAC

To actually create the homography matrix for warping the images, we apply the RANSAC method described in lecture to the set of correspondences we have from the last step. We use the following steps (from the lecture slides):
  1. Select four feature pairs (at random)
  2. Compute homography H (exact)
  3. Compute inliers where SSD(pi’, H*pi) < threshold
  4. Keep largest set of inliers
  5. Re-compute least-squares H estimate on all of the inliers
We repeat steps 1 through 3 1000 times to find the set of four points that gives the most inliers.

Result

Autostitching Result
Manual Selection Result
We see that the autostiched version is much better. There is no 'ghosting' of the trees, and the roof top is much less warped. The sky has no hard edge in it.

Other Automatically Stitched Images

My Room

Left image of my room
Right image of my room
Result

BAMFA

Note that this is the same anchor image from part A, but this time stitched with an image angled to the right.
Left image of BAMFA
Right Image of BAMFA
Result

What have I learned

The accuracy of the correspondences from the autostitching is stunning. I didn't expect the feature matching to work so well; using the ratio of the best match to the second best match is especially clever and I think is what allows autostitching to work at all.