Image Warping and Mosaicing

Overview

The goal of the project is to implement a cool application of image warping — image mosaicing. By rotating the camera about the center of lens without shifting and taking two or more photographs with slightly overlapping fields of view, we can create an image mosaic by registering, projective warping, resampling and compositing them. I learned about how to compute homographies and how to use them to warp images and create the panorama.

Part A

Recovering Homographies

The homography transformation has the following form where H is a 3x3 matrix with the last element i = 1 and 8 degrees of freedom.

To recover the homography, it can be shown from algebra that the homography can be solved be solving the following linear equation, where the vector h includes the 8 elements and can be reshaped to our homography matrix.

Image Rectification

One of the application of homography is image rectification. I chose four corners of the input image and forcing them to be parallel in the new image by selecting the corresponding position of the four corners in the new image. After that, I can compute the homography and warp the original image to the new rectified image. The following are examples of image rectification.

Mosaics

Finally for image mosaics, we can take two or more photographs of the same scene with slightly overlapping field of views by rotating the camera about the center of lens without shifting. To achieve the goal, I selected 15 corresponding points between each of the two images and created a blank background with the center image in the middle of the background.

Then, I compute the homography transformation of the left image so that its corresponding points can be matched with the points in the center image. Finally, I find the overlapping area between the two images and created a linear mask to blend the two images together to produce the mosaics.

The following are three examples using the techniques mentioned above to create image mosaics.

Example 1

Example 2

Example 3

Conclusion

The lesson I’ve Iearned from the project is that linear algebra is powerful in solving the transformation matrix and creating different effects of manipulating images. I’ve also noticed the difficulty when manually selecting the correspondence between images and I’m excited for the next part of the project for automatic alignment.

Part B

In the second part of the project, I created a system for automatically stitching images into a mosaic. The whole process includes detecting corner features in the image, extracting a feature descriptor for each feature point, matching these feature descriptors between two images, using a robust method (RANSAC) to compute a homography and finally proceed as in previous part of the project to produce the mosaic.

Harris Corners

I first start with Harris Corner Detector to detect corners in an image in order to extract features of the image. I applied Harris Corner Detector which finds corners in an image or where gradient in both x and y direction are both larger. Here are the results of Harris corners detected for the image.

Adaptive Non-Maximal Suppression

As we can see, Harris Corner Detector find too many points for each image . To restrict the number of interest points in each image, I implemented the adaptive non-maximal suppression algorithm to keep a subset of point that are evenly distributed on the canvas. For each Harris corner, I find the compute the minimum radius to the next corner such that the corner strength for the new point is larger than the corner strength of the original point multiplied by 0.9. I took 200 points with the largest minimum radius and here are the results of 200 points selected after adaptive non-maximal suppression for each image.

Feature Descriptor Extraction and Feature Matching

For each pass corner left after Adaptive Non-Maximal Suppression, I extract a 40x40 patch centered at the Harris corner. Then, I applied a Gaussian filter and downsample it to 8x8 patch to eliminate high frequency details. After that I computed the SSD as the distance between two patches, which is the squared norm of the difference of two patches. I use the ratio of distance to the first nearest neighbor and distance to the second nearest height, 1-nn/2-nn as matching score and consider two patch to match if the score is above threshold of 0.5.

Random Sample Consensus (RANSAC)

The feature matching still results in several errors. In order to improve this, I implemented the RANSAC algorithm. At each iteration, I selected four pairs of feature points at random and compute the homography. Then I applied the homograph to all points and compute the inliers where the SSD is less than ε(2 in my case). After that, I kept the largest set of inliers and recompute the SSD error between them and the real matching points. I ran 1000 iterations and keep the homography where there are maximum number of inliers. Here are some matching results after RANSAC.

Comparing the results

After getting the correspondence between images, we can use the same method in part A to warp images to mosaics. Here are the results of comparing automatic and manual image alignment.

What I Learned

The most interesting part I’ve learned from the project is the process of feature extraction and matching. I’m also surprised to see so many cool applications of homography and how simple ideas like Euclidean distance, randomness can be used together for our implementation of the cool mosaics application.