Overview

In this project I learned how to blend images together of the same subject from different angles in order to make panoramic images.

Part 4A

Shooting and Digitizing Pictures (1)

For each panorama, I took 2 or more photos of my subject from different angles/positions. Then I chose a set number of corresponding points between my sets of images which I later used to form a homography matrix between them. I found the images were easier to align when I chose rectangular points.

Image Rectification (2)

Before blending images together, I first checked my warping function by warping some sample images so that they are frontal-parallel (based on a set of rectangular correspondences that I defined for my image). Enjoy these two frontal-parallel images of my dog, Link!

Blending the Images into a Mosaic (3)

Then I expanded this process to work for multiple images. For each set of images I defined an anchor image, and warped the other images to that one using homographies. Then I blended the images together using alpha masks and weighted averaging.

Part 4B

Detecting Corner Features (1)

I used Harris-Corner Detection to find points of interest in my images. Interest points should have significant changes in all directions. I used the corner_harris functiono from Python's skimage collection in order to find the Harris points for my images.

Adaptive Non-Maximal Suppression

There were way too many points generated using Harris Detection in the previous part, so I had to find a way to narrow them down. I could have just chosen the points with the highest h-values, but perhaps all of my points could be clustered in a small area. So, I needed a way to choose points that had high h-values and were well spread out throughout my image. I first included the point with the highest h-value with a suppression radius of infinity. I found the suppression radii for the remaining points by finding the minimum distance to a a neighbor with siginificantly higher strength (h-value). I then chose 500 points with the highest radii (sorted in descending order).

As you can see from the sample images above, the points chosen using Adaptive Non-Maximal Suppression are more evenly spread out throughout the image.

Feature Descriptor for each Feature Point (2)

In order to help match feature points across images, I generated a feature descriptor for every point. I gathered a 40x40 square centered around each point, and downsampled them to 8x8 patches. I then normalized each patch to have a mean of 0 and standard deviation of 1 to account for changes in intensity.

Matching Feature Descriptors Between Images (3)

Now that I chose my features and generated a descriptor for each one, I needed to find matches between my images. I first calculated the distance between every patch in my first image to every patch in my second image. To avoid outliers, I used Lowe's thresholding method in which for every patch in image 1 I found first and second nearest-neighbor patches in image 2 (1_NN and 2_NN). If 1_NN/2_NN was <= 0.5 (i.e. the closest match is a significantly closer match than the second-closest match), I declared the first nearest_neighbor patch to be a match.

Here is a visual showing matching points between Image 1 and Image 2.

Using RANSAC (4)

RANSAC stands for "Random Sample Consensus," and I used this method to estimate my final homography matrix. The method is carried out like this (taken fom lecture slides):

1. Select four feature pairs (randomly).
2. Compute homography H.
3. Compute inliers where dist(p_i, H*p_i) < ε. In my case, ε = 10.
4. Keep the largest set of inliers.
5. Re-compute least-squeares H estimate on all of the inliers.

I repeated steps 1-3 1000x in order to get a large estimate of inliers, and then used them to calculate my homography matrix H.

Forming Mosaics (5)

The final step was to use the estimated homography matrix and generate mosaics, just like in Part 4A! I used my functions from the previous part to automatically stitch images together without having to self-select points of interest.