CS 194-26 Project 5B - Andrew Loeza

Overview:

In the first part of this project, we learned how to compute homographies, applied the perspective transformation to images, displayed image rectification, and then created mosaics of images. Following this, we then utilized a Harris Corner detector to detect the local maximum corners in 3 images. Then, we used Adaptive Non-Maximal Suppression to remove weak corners points while still keeping them spread out throughout the image. A Feature Descriptor Extractor was then used to generalize each point in a way that allows us to match the correct corresponding points between the images. We then use these feature descriptions to match the correct points by using Lowe's 2NN thresholding. Lastly, we then remove any incorrectly matched correspondences by using RANSAC. The correspondences that are returned by RANSAC are then used to compute the appropriate homography via least squares. Then the image is aligned and blended together using a Laplacian Pyramid.

Part A:

Shoot the Pictures:

Below are the images that I used for the entire project.

Perspective Transformation Images:

Image Rectification Images:

Mosaic Images:

Recover Homographies:

Here we calculated our homographies by solving the equation $p`=Hp$ by using least squares on the equation $Ah = b$, where $h$ contains the 8 variable coefficients of the homography matrix.

Warp the Images:

Here we apply the homography matrix to our Perspective Transformation Image for demonstration purposes.

Below you can see that the source image is warped to the target image. The prospective transformation not only translates and rotates the image, but it also skews it in the correct manner so that they can easily be overlaid.

Image Rectification:

Here we use our homography matrix to apply image rectification. This is done by warping an image to a series of correspondence points that force a certain perspective on any feature we desire. The first image we use is a hallway that make the tile face the camera, and the second image a painting where we do the same.

Image #1: Hallway

Here you can see that the tile is made perpendicular to the camera, distorting the sides of the image and the upper portions of it significantly. Despite this, though, the results are quite pleasing and if the top of the image were to be cropped, the illusion of looking down at the tile would be more convincing.

Image #2: Painting

As this is a painting, looking down at the tile in this image creates a very bizarre effect, as the very flat 2d bodies of the figures are disturbingly stretched towards the upper part of the image. The result of this is very interesting as it gives a feeling of being able to explore a created 2d world.

Blend the images into a mosaic:

In this part, we use our perspective transformations to allow us to easily align and combine images into large panorama images. After aligning them, a 2 layer Laplacian Pyramid is used to seamlessly blend them together.

Image #1: Picture Frame

Below is the image created by combining the two above images. There is some slight blurring in the top portion of the wave, but other than that the entire mosaic is properly aligned and there are no ghosting effects elsewhere.

Image #2: Cabinets

Below is the image created by combining the two above images. This image was much harder for me to choose the correspondence points accurately. This is visible by the slight ghosting effect that occurs on the bottom left portion of the middle cabinet. Despite this, though, the rest of the image looks very good.

Image #3: Dresser

For this image, a mosaic is created from three images. This is done by creating a intermediate mosaic from the center dresser image and the right dresser image. And then the final mosaic is created from the left dresser image and the intermediate mosaic.

Below is the intermediate mosaic created from the center and right cabinet images. The results of this look good and there are no noticeable blurring or ghosting issues.

Below is the final mosaic created from the left cabinet and intermediate mosaic images. The results of this also look good and there are no noticeable blurring or ghosting issues.

Tell us what you've learned:

This project was very challenging and I learned a lot. I learned how to apply least squares when calculating homographies, how to do image rectification, how to do deal with the side effects of perspective transformations since they can inadvertently crop images, and how to estimate bounding boxes for the transformation images to ensure that they capture the parts of the image we are concerned with.

Part B:

Harris Interest Point Detector:

Below you can see the Harris Interest Points for our picture frame images. As can be seen though, there are far too many corner points to be able to efficiently match the points together.

Adaptive Non-Maximal Suppression:

Below is the suppressed Harris Interest Points. As can be seen, the amount of points has been substantially reduced, removing the points that have a low corner strength while also keeping a nice even distribution of points throughout the images.

Feature Descriptor Extraction:

Below are a few examples of the point features for our images.

Feature Matching:

Below are the plots of our Suppressed Harris Interst Points. The red dots are points that have no corresponding match in the other image, and the blue dots are points that have a potential match in the other image.

RANSAC:

Below are the plots of our Suppressed Harris Interst Points. This points have the same meaning as the previous plot with the addition of the yellow points which are the points that we will use as correspondences for creating our mosaic.

Mosaics:

Picture Frame:

The below images are the mosaic image that was created with manual correspondence points and the mosaic image that was created with our automatic correspondences. Unfortunately, the manual mosaic appears to be of higher quality than our automatic mosaic. This is possibly because the images are quite plain which makes it hard for the edge values to be matched together with each other. Regardless, though, the automatic mosaic still looks quite good, and despite the slight warping on at the top of the image frame, the rest of the image looks as good if not better than the manual mosaic. One improvement is at the bottom image at the black border. In the manual mosaic, the black bleeds over slightly into the image, whereas this does not occur in the automatic mosaic.

Dresser:

For this image, we can see how sensitive our automatic correspondence point algorithm is to the input images. Below you can see that when the center and right dresser images are combined, the amount of correspondence points that we get is quite small. However, resulting automatic mosaic still looks at least as good as the manual mosaic. The issues, however, begin when the third image is added to create the final mosiac.

Below you can see that our algorithm isn't even able to get enough correspondence points to create a homography to create the mosaic. Moreover, the correspondence points it creates also tend to be incorrect. This is most likely due to the content of the image. It's possible that the camera inadvertently moved closer to the dresser in the left image or that the camera rotated slightly. Also, the dresser's uniform color and layout makes it really hard for our feature descriptors to really differentiate between certain correspondences. This all leads to a massive issue in calculating proper correspondence point pairs.

Cabinets:

Compared to the dresser images, the cabinet images work much better with the automatic algorithm. Several high quality correspondences are found, which gives us a very good quality homography to create our mosiac with.

As can be seen, this high quality homography gives us much better results that the manual mosaic. The ghosting effect at the bottom of the center cabinet is no longer there, and in my opinion, the black border is much better aligned and looks more visually appealing.

Bedroom:

In this mosaic, we once again try to combine three images to create the mosaic, but this time the image has a lot more distinctive features. The benefits of this become immediately obvious when we see just how many good correspondence points our algorithm finds.

And below we can see the benefit of having so many good correspondence points as our intermediate mosaic has been stitched together really well. The only noticeable issue is at the top of the image but that is most likely because of the fact that the right image was probably rotated slightly relative to the left image while I was taking the picture. And another issue is that there is some slight warping on the bed's backboard, most likely due to the same reason.

As before, we can see that our the images' distinctive features give us a plethora of correspondence points as seen in the below plots.

And these correspondence points give us a high quality homography that allows us to create a good quality final mosiac with no noticeable blurring or ghosting effects aside from some slight warping on the bed's backboard, which is most likely due to the images themselves and not the algorithm.

What have you learned?:

The coolest thing I learned from this project was the entire process of being able to automatically locate the correspondence points after filtering out all of the irrelevant Harris Points. Despite the fact that we simplified the process significantly, it is still suprisingly robust so long as the images inputted into it fall under the correct conditions.