CS 194-26 Project 6

Phoebe So

6A: Image Warping and Mosaicing

Shoot the Pictures

The first step was to shoot the pictures for the mosaicing. When taking photos of a nonplanar scene, I tried to keep the center of projection the same for all the images to ensure the images will line up when projecting the photos onto a common plane.

Recover Homographies

Next, I had to recover the homographies, projective transformations, between the images. A homography has eight parameters and transforms one image into the image plane of another. Therefore, in order to solve for the homography, I had to solve the following lienary system of n equations:

Because the homography matrix has 8 unknowns, a minimum of 4 correspondence points are required to solve the lineary system. However, more correspondence points can be provided to produce an overdetermined system which can then be solved using least-squares.

Warp the Images and Image Rectification

Once I obtained the homography matrix, I was able to warp images from one perspective to another. Below are some examples of images being warped so that the plane is frontal-parallel. This was done by creating correspondence points between an object of a known shape (ex. a sqaure) in the original image to the exact points of that shape in the frontal-parallel plane (ex. [0 0; 0 1; 1 0; 1 1]). The original image is on the left and the warped image is on the right.

Image rectified to square muffin quote

Image rectified to square bookcase shelf

Blend the Images Into a Mosaic

Lastly, I was able to create an image mosaic by combining the images warped to the same plane. For my mosaics, I warped the left image towards to the plane of the right image.

Original images of my bedroom wall

Warped left image

Naive mosaic with no blending

However, with no blending, the mosaic contained harsh edges where the two images were naively stiched together. Therefore, linear blending was implemented to decrease the effects of the harsh edges. Below is my bedroom wall mosaic created with linear blending. Clearly, linear blending helped produce a more cohesive mosaic, for example, the bottom edge of the map in the blended image does not have the sharp edge that the naive mosaic contains.

Final mosaic with linear blending

Original images of my kitchen

Warped left image

Final mosaic with linear blending

Original images of an apartment hallway

Warped left image

Final mosaic with linear blending

For the apartment hallway mosaic, because the details of the warped left image and right image do not perfectly align (ex. the cement railing is coming towards the viewer at different angles in the two photos), the resulting blended mosaic is not as cohesive as the other two examples

What I Learned

Through this project, I learned how powerful and useful homographies and projective geometry is and am excited to learn more about them!

6B: Feature Matching for Autostitching

Harris Interest Point Detector

Instead of manually choosing corresponding points between photos like in Part 6A, Harris corners were used as feature detectors to increase the accuracy.

Below are the original photo and the photo with the Harris corners overlaid.

Adaptive Non-Maximal Suppression

Because the returned set of Harris points from the previous part was large and random, Adaptive Non-Maximal Suppression was implemented to select points that were evenly distributed over the photo. This was accomplished by calculating the distance between each Harris corner, xi, to every other corner, xj, where h(xi) < 0.9 * h(xj), and then selecting the top 500 corners with the shortest distance to another corner.

Below are photos with the Harris corners and the ANMS selected coordinates overlaid.

Feature Descriptor Extraction

In order to create a feature descriptor for each point, the following algorithm was applied to each point: 1. A 40x40 pixel region centered at that point was extracted from the image 2. The 40x40 region was then downsampled by a factor of 5 to create a 8x8 descriptor 3. The patch was bias/gain-normalized to generate the final feature descriptor. These feature descriptors allowed each point to be characterized and compared to points in other images.

Feature Matching

After creating feature descriptors for every point, I could then compare points across two images. This was done by performing the following algorithm: 1. Compute the SSD between every two pairs of points between the two images 2. Compute the 1-NN and 2-NN for every point in the first image 3. If the 1-NN/2-NN ratio is less than a certain threshold, then the point in the first image is matched to the 1-NN point in the second image.

Below are photos with the matched points overlaid.

RANSAC

Next, RANSAC was implemented to compute a Homography matrix to accurately project an image onto the plane of another. The following RANSAC algorithm was implemented: 1. Select 4 random corresponding feature points 2. Compute the Homography matrix from these 4 pairs of points 3. Warp all the features using the Homography matrix and compute the number of inliers 5. Repeat steps 1-4 10,000 times, keeping track of the maximum set of inliers. After exiting the RANSAC loop, I recomputed the Homography matrix using the maximum set of inliers. This Homography matrix was then used to warp an image and create a final, blended mosaic following the procedure from Part 6A.

Below are mosaics produced using the naive point selection process from Part 6A and the featuring matching process from Part 6B.

In the naive point selection process, the "HRC" sticker located at the bottom is not correctly aligned. However, the feature matching process produced a significantly more aligned and better mosaic.

I got lucky and my kitchen mosaic from Part 6A is already well-aligned, so the mosaic produced from Part 6B is not significantly different.

Below is a mosaic generated from photos not used in Part 6A.

This mosaic was less successful than the above two examples, resulting in the misalignment of the items on the table.

What I Learned

Through this project, I learned that feature detection is very powerful and surprisingly uncomplicated to implement. Additionally, I learned how important good photos are for producing a successful final outcome. Certain mosaics turned out better than others, potentially because the original images were taken from the same center of projection.