Project 5

By Samuel Sunghun Lee [CS194-26 - Fall 2020]

Part 1

Shoot the Pictures

I took two images of the view outside my window in my room. The images are taken at different angles of the same view of a buildings outside my room. Here are the two images:

dx dy

Recover Homographies

Using the image 1 and image 2 shown above, I will now calculate the Homography matrix H which is a 3x3 matrix with 8 degrees of freedom (lower right corner is a scaling factor). We first manually create corresponding points between the two images. I chose 9 points along the white building for each corner of the white building where it appears in both images. Then, using these points, we will calculate this from the following set of equations that was taken from this website: https://cseweb.ucsd.edu/classes/wi07/cse252a/homography_estimation/homography_estimation.pdf

dy
I used Singular Value Decomposition of the A matrix shown in the above image, and then took the eigenvector with the smallest singular value and reshaped it into a 3x3 matrix to get us the Homography matrix H.

Warp the Images

Now that we have two images, and a homography matrix H, we can now warp one of the images to the perspective of the other. In this case, since I calculated a H matrix toward image 2 (right image in the first section), I warped the first image with the H matrix using inverse warping.
Here is the before and after of the warping. (left = before, right = after)
dy dy
As you can see, the first image was warped into the perspective of the second image, which was slanted toward the angle that we see in the warped image.

Image Rectification

We are now able to rectify an image without needing two images. In this section, I took the following raw image
dy
and I took 4 points of the book and made 4 corresponding points in rectangular coordinates i.e made the points parallel to the x-y coordinates. Thus, we should expect this image to be in a front planar view. Here is the following image after 'rectifying' to the front planar view through calculating the Homography H Matrix from the points we described.
dy
Here is the complete before and after side by side. (left = before, right = after)
dy dy

We can see that the image on the left was 'straightened out' and the buildings now look front parallel to the viewer of the screen. This shows that the Rectification was successful as we chose corresponding points that accomplished what was intended. The book is now in the front planar view.

Stitched Mosaic

From the section 'Warp the Images', we can now take the warped Image 1 and the untouched Image 2 and combine the two images to create a 'Stiched Mosaic'. I did this by aligning both images by getting two corresponding points in both images, and rescaling, rotating, and matching the image size. After aligning the two images, I compute the average of the two images, with a heuristic of if one of the pixels of either image is black or white, then we will take the non black/white pixel instead of averaging the two. This turned out to produce a reasonable mosaic that shows a panoramic view of the white buildings outside.

Images to stitch

dy dy

Stitched Image

dy

What I have Learned

I have learned that to create a panoramic image, linear algebra comes in pretty handy to transform one image to another image's perspective. I also found it really interesting that after image alignment of two images that are in the same view, taking the maximum of the two images works quite well and the image that was produced as seen in the previous section looks really smooth and clean. Finally, I have learned that there is a lot of room for automation for stitching images to create a panoramic image. This is where the next part comes in! Aligning imaging manually to produce points and remembering the ordering is quite annoying so I learned that there is room for growth and will see how the next part will mitigate this issue.

Part 2

Image Stitching Example 1 - Gasworks Park


1. Here are the raw two images that we are going to stitch together to create a Mosaic.html

1. Raw Two Images

dy dy

2.Here are the raw corner points generated by the harris corners functions for both images. This function was provided in the starter code given by the staff that simply computes the corner points.

2. Corner Points

dy dy

3. There are too many corner points generated by the harris corners function. We want to prune some of those functions and were going to do that by using the adaptive non maximal suppression function that generates points that are more uniformly distributed across the image. We do this by using some radius r to create more uniformly distributed points.

3. Adaptive Non Maximal Suppression

dy dy
4. Now that we have some uniformly distributed corner points, we are going to do feature generation and feature matching and then run RANSAC to first generate corresponding points between the two image's corner points. We generate a 8x8 feature map using Gaussian Kernels and then subtracting by mean and then normalizing the features. Then, we are going to match the corner points by calculating 1st_NN / 2nd_NN and matching points that have that value < 0.4. I used 0.4 because 0.2 created too few matching points. After generating these matches, we will run 3000 iterations of RANSAC that will prune outliers of bad point matches. By the end of this, were going to have a list of corresponding pairs of points for image 1 and image 2.

4. Feature Matching + RANSAC

dy dy
5. Now that we have matching points, we will do the same thing we did in Part 1. We will generate a Homography Matrix H from the list of pairs of corresponding points from the above part. Then, we are going to warp the first image and then stitch it together with the second image. Here is the image generated.

5. Auto Image Stitching


auto stitched image
dy
manual stitched image
dy

Image Stitching Example 2 - Oil Tank


1. Here are the raw two images that we are going to stitch together to create a Mosaic.html

1. Raw Two Images

dy dy

2.Here are the raw corner points generated by the harris corners functions for both images. This function was provided in the starter code given by the staff that simply computes the corner points.

2. Corner Points

dy dy

3. There are too many corner points generated by the harris corners function. We want to prune some of those functions and were going to do that by using the adaptive non maximal suppression function that generates points that are more uniformly distributed across the image. We do this by using some radius r to create more uniformly distributed points.

3. Adaptive Non Maximal Suppression

dy dy
4. Now that we have some uniformly distributed corner points, we are going to do feature generation and feature matching and then run RANSAC to first generate corresponding points between the two image's corner points. We generate a 8x8 feature map using Gaussian Kernels and then subtracting by mean and then normalizing the features. Then, we are going to match the corner points by calculating 1st_NN / 2nd_NN and matching points that have that value < 0.4. I used 0.4 because 0.2 created too few matching points. After generating these matches, we will run 3000 iterations of RANSAC that will prune outliers of bad point matches. By the end of this, were going to have a list of corresponding pairs of points for image 1 and image 2.

4. Feature Matching + RANSAC

dy dy
5. Now that we have matching points, we will do the same thing we did in Part 1. We will generate a Homography Matrix H from the list of pairs of corresponding points from the above part. Then, we are going to warp the first image and then stitch it together with the second image. Here is the image generated.

5. Auto Image Stitching


auto stitched image
dy
manual stitched image
dy

Image Stitching Example 3 - Apartment

1. Raw Two Images


1. Here are the raw two images that we are going to stitch together to create a Mosaic.html
dy dy

2.Here are the raw corner points generated by the harris corners functions for both images. This function was provided in the starter code given by the staff that simply computes the corner points.

2. Corner Points

dy dy

3. There are too many corner points generated by the harris corners function. We want to prune some of those functions and were going to do that by using the adaptive non maximal suppression function that generates points that are more uniformly distributed across the image. We do this by using some radius r to create more uniformly distributed points.

3. Adaptive Non Maximal Suppression

dy dy
4. Now that we have some uniformly distributed corner points, we are going to do feature generation and feature matching and then run RANSAC to first generate corresponding points between the two image's corner points. We generate a 8x8 feature map using Gaussian Kernels and then subtracting by mean and then normalizing the features. Then, we are going to match the corner points by calculating 1st_NN / 2nd_NN and matching points that have that value < 0.4. I used 0.4 because 0.2 created too few matching points. After generating these matches, we will run 3000 iterations of RANSAC that will prune outliers of bad point matches. By the end of this, were going to have a list of corresponding pairs of points for image 1 and image 2.

4. Feature Matching + RANSAC

dy dy
5. Now that we have matching points, we will do the same thing we did in Part 1. We will generate a Homography Matrix H from the list of pairs of corresponding points from the above part. Then, we are going to warp the first image and then stitch it together with the second image. Here is the image generated.

5. Auto Image Stitching


auto stitched image
dy
manual stitched image
dy

What I learned

I learned that we can use various amounts of heuristics and pruning of things to produce very good stitchings. I thought it was very very interesting that using various thresholds and playing around with them can have drastic impacts on creation of images. Overall I am happy that I learned how to automatically stitch two images together without having to manually find corresponding points between two images. The feature extraction and matching was really cool and was surprising how well the Nearest Neighbor worked.