Image Warping and Mosaicing

In this project, I tried to achieve the panorama effect by using the idea of homography.

Shoot and digitize pictures

This is the first step to get a panorama. The key here is to keep the center of camera stable so that the pictures can actually align when they are transformed under a homography matrix.

I shoot as close together in time as possible, so my subjects don't move , and lighting doesn't change too much. I also use identical aperture & exposure settings. I also keep 40% to 70% of the photos overlapped.

Recovering Homogprahy

After taking photos from the same point of view, I want to transform the images into the right shape so that they could be stitched seamlessly together. What shape should the images be transformed into?
Here is the idea of homography. That we can linear transformation of the shape of the image and produce an image that will align with the other images.
Thus, given point (x,y) in image 1, its corresponding point in image 2 (x',y') can be calculated using the following formula:

After selecting corresponding points of each image, I can retrieve the transforamtion matrix by setting up linear equations and view it as a least square problem.

Warping and Rectification

With the idea of homography, I can now transform the image into any shape within the scope of perspective transformation. Say now we have an image that is taken slanted. I can rectify it finding the tannsformation matrix that takes the four corners of the a square in the image into a square coordinates.

Here is an example. I took the photo in the famous mirror room in Chateau de Versaille.

Welcome to Rome. I took a slanted picture in the Colosseum. Here is Recified Roman Glory.

The same logic goes with the warping process. Now that I have attained the transformation matrix. I can move on to transform all the image into a warped shape that corresponds with the overlapping area with other images.

Here are the two pictures I took of my kitchen. I select the corresponding points. Then, I fixed the first image and warp the second image so that the corresponding points fit one another. Finally, I used alpha mask blending and get the following result.

PANORAMA! Stitching from Both LEFT & RIGHT!

Here is My tini apartment

PANORAMA! My tini apartment.

I also played with some outdoor image. It is not as good as the indoor ones that because things tend to move and cannot perfectly align things together. I used Violet Fu's outdoor photo since i am stuck sheltering in place.

Detecting corner features in an image.

To automatically detect corners in the image, I use the Harris corner method. Here are two images covered with harris corners.Here, I select the top 500 points with the largest R value and I notice that a lot of the points are clustered together.

I select the top 500 points with the largest R value and I notice that a lot of the points are clustered together. This is not great for the developing corresponding points since we would like the corresponding points to scatter the image as much as possible. As a result, I adopted the ANM method. The basic idea is that for every point, find the smallest distance min_dis so that its R value is the smaller than c_robust(0.9 here)* R value of its neighbors. Then we rank min_dis and select the largest points.

Feature Descriptor

Then it's time to get the feature descriptor of each point. Find the 40*40 window in the oriented angle with the point being the center. Then subsample the descriptor down to 8*8. Then I match the points based off the SSD between the descriptors and set a threshold to get rid of the biased points. Using a threshold of 0.7. I get the following matching points. I select the threshold so that the points are scattered enough to produce a good homography. Note that there are still some poor matches.

Use RANSAC to compute a homography

This is the step to get rid of the poor matches. Using RANSAC, I ramdomly get 4 pairs of matching points and compute the homography. Since the good points often produce similar homorgraphy, we find the SSD of every points in img2 and the computed transformed points from points in img1 to img2. Setting a certain threshold, I get the number of inliers. Finally, I select the transformation with the largest number of inlier and compute the homography. There are the points selected after running RANSAC for 10000 time.

Produce a mosaic

Compare the following two stitched images: left being hand pointed and the right being the result of auto-stitching.

Some more results:

This project is really fun! I enjoy figuring out how ANM works by reading the paper. What I learn most is getting hands on reading research paper and implement cool techniques.