CS194 - Image Warping and Mosaicing

Nathan Yuchi(CS194-26-acl)

Shoot and Digitize Pictures

These are the photos I took and created for this project

6000x4000 Coaster Downscaled to 720x480

Coaster Target Shape

6000x4000 Gum Downscaled to 720x480

Gum Target Shape

Left Picture for Bar

Right Picture for Bar

Left Picture for Pier

Right Picture for Pier

Left Picture for Room

Right Picture for Room

Warp Images (Rectification)

After solving for the inverse homeography matrix, we can apply the matrix to points in our original image, warping the images to the target points we select.

Original Image

Target Image

Rectified Coaster

Original Image

Target Image

Rectified Gum

Blend Images Into a Mosaic

To blend images into a mosaic, I took two photos with overlapping points. I kept one image still while I warped the second image to points selected from the first. I padded the image with zeros or black pixels to allow extension of the image vertically and horizontally. I have results for no blending and for fixed-alpha blending.

First Pier Image

Second Pier Image

Pier Mosaic - No Blending

Pier Mosaic - Alpha Blending

Pier Mosaic - No Blending (Cropped)

Pier Mosaic - Alpha Blending (Cropped)

First Bar Image

Second Bar Image

Bar Mosaic - No Blending

Bar Mosaic - Alpha Blending

Bar Mosaic - No Blending (Cropped)

Bar Mosaic - Alpha Blending (Cropped)

First Room Image

Second Room Image

Room Mosaic - No Blending

Room Mosaic - Alpha Blending

Room Mosaic - No Blending (Cropped)

Room Mosaic - Alpha Blending (Cropped)

What I Learned

In this part of the project I learned how to use solve for a homeographic transformation matrix and how to use the matrix to change the perspective of an image. Using the matrix and blending techniques, I also learned how to create mosaics from images. I think the most interesting aspect of this project was having to use our own data, our own photos.

Part 2

Find Harris Corners

The first task of this project as to find the points that may be the feature points. I used get_harris_corners from the provided harris.py and changed min_distance = 8. Changing min_distance chooses less points as harris corners and therefore will speed up computation in future steps.

ANMS

We have too many points and later computations' runtime would be severely affected by so many points. We can lower the number of points by implementing adaptive non-maximal suppression (ANMS). We find the minimum distance of each point to a point that has .9 of the original point's Harris power. Then we select the 250 points with the greatest distances.

Feature Description Extractor

Next, we need to extract the feature descriptor for each ANMS point. We do this by taking the 40x40 pixels around each point and down-sample it to 8x8 We make sure to normalize the descriptors by subtracting by the mean and dividing by the standard deviation.

Feature Matching

We need to match the features that we just extracted to each other. We calculate the SSD between each point in the first image to each point in the second image. If (nearest SSD neighbor's SSD) / (second nearest SSD neighbor's SSD) < .4 Then we consider the point we are looking at and it's nearest SSD neighbor as feature matches. Using Ajay Ramesh's debug tool, we see the points that were matched with each other.

RANSAC

If we look at the debugging images from the previous sections, we can see that there are some points that are matched to multiple points and some points that are mapped to the wrong point. To fix these outliers and incorrect matches, we use random sample consensus (RANSAC). To implement RANSAC, we choose 4 random corresponding points from each image and compute the homeography matrix. Then we apply the matrix and calculate the SSD between each point and its respective transformed corresponding point. If the point SSD is below a threshold (I chose .5), then consider those points "inliers". We repeatedly choose 4 random points, calculate the homeography, apply the homeography, and find the number of inliers a large number of times (I chose 1000). Then we choose the homeography matrix that resulted in the largest set of inliers as our final homeography matrix.

Given our final homeography matrix, we can proceed as in part1 and warp the images. Below we can see the selected inlier points and see that the features do match up.

Manual vs Automatic

We can compare the results of manual alignment vs automatic alignment below.

Automatically-Aligned Bar

Manually-Aligned Bar

Automatically-Aligned Pier

Manually-Aligned Pier

Automaticaly-Aligned Room

Manually-Aligned Room

Most Interesting

The most interesting thing about this part of the project was how the automatic alignment performed with a mirrored image. The bar images included a mirror. I was worried that feature matching would select the wrong points because a mirrored feature would look the same for symmetric objects like bottles. I was impressed with how well automatic alignment and feature matching worked for even mirrored images.