CS-194 Project 5A

By: Calvin Chen

Shoot the Pictures

For the beginning of this project, I had to initially take the pictures. It was important that the pictures I took overlapped enough to find correspondence, so I stood in one place and rotated my camera slightly to determine a ~50% difference between perspective between the two images.

Below are the images I took:

Recover Homographies

For this section, I found a matrix H that would be able to correspond points from my first image to my second image. The corresponding points for these images were labeled by hand, and in order to determine a system of equations that was overdetermined and could be calculated via least-squares, I needed at least 4 points, and I ended up using 8 points to conduct this transformation.

The system I set up looked similar to the following (taken from Google images):

Warp Images/Rectification

Once I extract the homography matrix for the transformation between these two sets of points, I transformed my initial image by warping it with the matrix to the second image. Afterwards, I determined different RGB pixel values through interpolation and constructed the rectified image.

Below are some examples of my images after they were rectified.

Pictures on the wall

Book on the table

Blend the images into a mosaic

For this section, I warped different surrounding images into a central image and stitched the images together. The resulting image had a similar panorama effect with the different images.

What I learned

What I thought was interesting/cool was how these different images could be easily translated to different perspectives via simple homogenous transformations. I think with the processes I've constructed above, I can conduct functionality very similar to creating panoramas (and could possibly be the process that occurs behind the scenes in camera phones as well).

Part B- Autostitching

For this part of the project, I focused on automatically constructing the correspondences between images.

Detecting Corner Features

For the first part of autostitching, we want to determine potential corresponding points in the images. This can be done using the Harris point detector. Some results of the point detector run on some images are shown below:

Adaptive Non-Maximal Suppression

From before, we can clearly see that there are a huge variety of different corner points that could be used. So, in order to reduce this number, we use a technique called Adaptive Non-Maximal Suppression. This technique lets us choose points that are spread out over an image, and not just take the top 500 points determined to be corners (they could all be clumped towards one part of the image). The algorithm took the following strucutre and took the top 500 points from this process:

Some of the resulting points can be seen below:

Feature Descriptor Extraction

For this section, we constructed feature descriptors for each of the resulting points from the ANMS process. This was done by applying a GaussianBlur with 5x5 kernel size across the entire image, constructing 40x40 pixel windows around each point, and downsizing each image to a 8x8 square. Two descriptors can be seen below:

Feature Matching

Once the features were determined, we conducted feature matching by trying to match each of these feature descriptors in the next image, but also ensure that this was a much better match than if it was compared to other images. This was done by using an error threshold of $e_{1-NN}$ / $e_{2-NN}$, which is the error of the first nearest neighbor divided by the error of the second. We also determined a threshold that images had to surpase to be qualified as good features (0.4), to ensure that these images were indeed much better for one image than any other.

The matched features for the two images from above are shown below:

RANSAC

To make sure that there are no outliers in our correspondences, we ran RANSAC to further filter out the different matched feature points to try and determine the "average" translation vector.

The algorithm was run as follows:

  1. Select four random pairs of points
  2. Compute the homography between these pairs of points
  3. Compute the inliers where the SSD of p' and H(p) is less than epsilon
  4. Keep track of the largest set of inliers while looping through 1-3
  5. Recompute the entire least-squares H estimate on all the inliers and return them.

The results of running RANSAC on the images from above are shown once again below:

Construct Mosaiscs

Now that we are able to automatically construct correspondences between the images, we can once again blend the images using the same processes as we did in Part A.

Disclaimer: For some reason, I could not get the autostitching of images to work completely correctly (perhaps due to how I constructed my homography matrix), so some of the results below will not look terribly well. I debugged for hours and couldn't come to a sound conclusion for why this was happening, but I tried my best to get them to work properly.

Here are a few of the mosaics shown below:

Manually stitched street

Automatically stitched street

Manually stitched study room

Automatically stitched study room

Manually stitched living room

Automatically stitched living room

What I learned

From the project, I learned a lot about different image processing techniques, and how corner detection can be an integral piece of computer vision + computational photography techniques. Using the automatic corner detection technique, it became much easier to reconstruct these images into mosaics, and I could clearly see how useful these techniques are in the real world (i.e. panoramas).