CS 194-26 ~ Project #6A ~ Image Warping and Mosaicing

Matthew Sie : cs194-26-aep

Step 1: Shoot and Digitize Pictures

I obtained four different sets of pictures taken panorama style (one set from online, three taken with camera) and three images for the "rectified image" section (two from online, one taken with camera).

Step 2: Recover Homographies

Using the equation p'=Hp, and two input sets of corresponding points, we are able to find the homography matrix, H, using linear algebra. Since the homography matrix, H, has 8 unknowns, we need at least 4 pairs of corresponding points in order to solve the equation. Furthermore, if we have more than 4 pairs of corresponding points, we must use least squares in order to solve the equation.

Step 3: Warp the Images / Image Rectification

The warping function, which takes in an image and a homography matrix, first calculates the dimensions of the warped image by multiplying the homography matrix, H, by each of the corners of the original image and creates an empty image with the correct dimensions. Then, each pixel within this empyt image finds is proper warped pixel by applying the inverse of the homography matrix to find the proper point from the original image. Below are three examples of image rectification completed using this warping function.

Step 4: Blend images into a Mosaic

To produce a mosaic, we imported the two images to be stitched together, defined correspondence points, computed the homography, and warped one image to another. This gives us one warped image and another image that is still in its original state. The next step is blending. First, using one of the corresponding points we defined earlier, we calculate the necessary shift and shift the images to their appropriate positions so that they are overlayed with each other properly. Next, in order to minimize human error from manual selection of correspondence points, we run an SSD scoring function over a 200x200 pixel region around the correspondence point being used. And finally, using this optimal overlay positioning of both images, we perform a linear blending between the two images in their proper position to obtain the final mosaic. Below are four examples of mosaics constructed using this method, including a vertical panorama.

Better mosaics were produced when more correspondence points were used and when more time was spent to diligently and accurately select correspondence points. Additionally, I noticed that images with trees produced some blurry results probably due to movement of the plants while taking the photos.

CS 194-26 ~ Project #6B ~ Feature Matching for Autostitching

The second part of the project revolved around using the RANSAC methodology to perform autostitching. I will walk through the steps of the method using the following two images:

Step 1: Harris Interest Point Detector

The first step was to obtain all the Harris corner points, which we obtained using the provided source code.

Step 2: Adaptive Non-Maximal Suppression

To reduce the number of interest points to 500 (while maintaining distribution across the image), we used an approach called Adaptive Non-Maximal Suppression, as found in the provided paper. This approach selects points with the largest "minimally-suppressed radius". The equation, which can be found in the paper, essentially calculates the minimum radius from each interest point to another point such that the corner response of the given point is less than a constant times the response of the other point. We then select the 500 points with the greatest radii.

Step 3: Feature Descriptor Extraction

Using these 500 points, we produce feature descriptors. This is done by using a 40x40 window around each point, which is down-sampled to an 8x8 window using a low-pass filter, as in Project 1, and then bias/gain normalized. Once all the feature descriptors are extracted, we can proceed to "Feature Matching".

Step 4: Feature Matching

We, again, narrow down the amount of interest points we will use by selecting only the points with the most similar/consistent features/feature descriptors. This is done by computing the SSD from each feature in one image to every feature in the other image. Using the simpler Lowe ratio test, we only keep descriptors (and their corresponding points) that have a ratio between the distance to the first nearest neighbor and the distance to the second nearest neighbor that is less than some set threshold value. For different sets of images, I changed the threshold value to limit the amount of points kept.

Step 5: RANSAC

The RANSAC step of the project involved randomly selecting 4 of the interest points, computing a homography between the images, warping the first image points using the homography, and computing the distance between the warped points and the points in the second image. Those that had a distance less than 5 were deemed inliers for the randomly generated group of 4 points. We ran this cycle a large aount of times to find the group of 4 points that generated the most inliers. Using the homography found with this group, we continued to do the final warping and blending of the two images, as described above in Project 6A. For this part of the project, instead of the naive blending approach I used in part 6A, I implemented linear alpha blending to smoothen the transition between the two images. This was completed by finding the window of overlap between the images and iterating the alpha value that weighted the pixel values of one image over the other as we moved across this window.

Results

Here are my 3 final mosaics, with the mosaics produced manually from Part 6A on the left, and the mosaics produced automatically in this part of the project on the right.

The mosaics I produced automatically in this part of the project were significantly better than the ones I produced manually because of the greater precision in the selection of interest points and the use of linear alpha blending.

What I Learned

I think the coolest thing about this part of the project was seeing the power of automating the selection of interest points. The blends came out so much better using this method, and the gap in quality between the manual and automatic methods is quite significant.