Overview

In this project, I explored how to stitch images together that come from the same center of projection. By computing homography matrix for projective transformations, I can rectify images on to the different planes as well as stitch images together creating a panorama. Originally, specified correspondence points were selected manually to compute the homography matrix which is slow and tedious. Part 2 revolves around detecting feature points between the images using the MOPS process in order to selected points automatically.

Part 1: Image Warping and Mosaicing

Recover Homographies

In order to warp images into the same plane for alignment, we need to recover a homography matrix for transformations between each pair of images. The homography matrix is: p' = Hp where H is a 3x3 matrix with 8 degrees of freedom (lower right value is set to 1) and (p, p') is the set of correspondences points.

Even though 4 points is sufficient to recover a homography matrix, the matrix would be very prone to noise and therefore be unstable. So to accommodate this, I decided to use least squares to help solve for the values in the form Ax = b. The matrix and vector are defined as follows where h33 = 1 which helps us achieve the 8 degress of freedom.

Warp the Images

Once we found the homography matrix, we can now warp images using it. Here are the steps I took to warp an image to a different plane.

• Identify the corners of the image to be warped (called img from now on)
• Forward warp with the homography matrix to find the destination corner points
• On the case where the warped corners result in negative values, store those negatives as future translations
• Determine the new image's height and weight by taking the max of the warp corners defined as (new_H, new_W)
• Create a new image of side (shift_row + new_H, shift_col + new_W)
• Generate a polygon depicting the indices of the warped destination
• Inverse warp those polygon points with the homography matrix
• Interpolate the inverse points to populate the new image

Image Rectification

To rectify an image, I chose the front face plane to warp slanted images to by picking the 4 corners of a rectangle as the target positions to warp. Then, I took slanted images, picked the 4 correspondence points, and warped following the directions above to warp the picture into the front facing plane. Below are some of my results.

I tried to perform rectify on a sheet music paper; however, since the sheet has be used many times, the paper isn't crisp and resulted in the bend formation. This causes the homography to act weirdly and not produce a perfect rectangle.

Blend the images into a mosaic

To blend the images together, I warped all the images to a similar plane (I chose the middle image on the case if there is > 2). In order to blend the images together, I created a mask defining the intesection between the 2 images and from there was able to take the average between image. (This is essentially a broken down alpha channel). Below are my results.

Things I learned

In this project, I learned that it's a nontrivial task to stitch images together. When manually selecting points, I had to be extremely careful in clicking the right place otherwise the images would be very blurred. Alpha blending also helps mesh the images together with some successful; however, using a Laplacian stack to blend probably would achieve the best result.

Another nuisance I had to ensure is that the center of projection is constant in order to probably projectively transform one image to another which meant my camera had to stay stable when taking the pictures. I also needed to be conscious of available light sources when taking images. Looking at the images of unit 1 at berkeley, the stitching didn't appear well due to my camera being in the shade for the warped image thus resulting in an obvious stitch.

Part 2: Feature Matching for Autostitching

Harris Interest Point Detector

In order to automate the point selection process, we need to first identify potential point candidates. I used the Harris interest point detector single scale to identify potential corners. I calculated the corner strength for each pixel and selected the coordinates that were strong enough over a specific threshold (0.1). This will be the first subset of points which will be filtered out in the following steps.

Adaptive Non-Maximal Supression (ANMS)

I needed to restrict the number of interest points extracted per image due to the long computational cost (image above had over 335k points); however, I also needed to be conscious of where the points are distributed. A evenly distributed spatial image would benefit stitching more. Therefore, ANMS was implemented to help select the top 1000 points (I used 1000 due to how large my image was but the paper used 500) that had the highest minimum supression radius given by this equation. Now we filtered over 300k point down to the most important 1000 points.

Feature Descriptor Extraction

Now that we selected our interest points, we need to match features points between images to stitch together. To accomplish this, I decided to take a 40x40 window around each interest point and sub-sample an 8x8 axis-aligned patch to describe the features around the point. For the sub-sampling process, I used a box filter to extract the lower frequencies to downsample to a 8x8 patch. Below is an example of extracted feature path of an image.

Feature Matching

With a feature patch corresponding to each interest point, I now feature matched all points based on their corresponding patches. I used Lowe's thresholding in order to determine whether two points are likely to be good matches. For each interest point of image 1, I computed the sum of squares difference (SSD) for each of image 2's patch and found the first and second nearest neighbor. I only determined a feature is a match if the ratio between SSD(1-NN)/SSD(2-NN) was below a threshold (I used 0.5). This allows to confidently say that that pair of points on both images are in the same location. Below are the selected feature points

Here is a summary of all the points we located

• red - harris corner points
• blue - anms suppression points
• yellow - feature points

Random Sample Consensus (RANSAC)

Finally, I need to compute the homography matrix between the two images given the feature points. I decided to use RANSAC which is a consensus algorithm that relies on the fact that when selecting the correct points to compute the homogarphy matrix that will be the dominating scenario that has the most amounts of features points that are inline with it. Any other incorrect combination will result in less correct points.

• Select for feature pairs (at random w/o replacement)
• Compute the exact Homography matrix
• Compute all inliers where dist(p', Hp) < epsilon (used l2 norm)
• Repeat steps 1-3 T iterations, keeping the largest set of inliers
• Re-compute H using least squares on all inliers in the largest set

Auto Stitching Process

Once completed the automatically computed homography matrix, we now can fully automate the stitching process by combining parts 1 and 2 together. We will use the homography matrix outputted by RANSAC as the homography matrix to warp images. Below are the results comparing them to the manual selection.