Overview

The purpose of this project was to stitch together images taken from different angles to create one contiguous parnorama shot. I did so using the principle of homogenous matrices, image warping, and image combination / blending!

Shoot and digitize pictures

I first took some photos of my surroundings! These were taken with my camera staying still and with the iPhone exposure locking feature activated. As I did in the previous project, I stored my keypoints for each image as an array in text files, using the save_image_to_file function. To access these points, I used get_point_from_file -- this allowed me to only define and generate my points only once, which saved a lot of time when debugging an generating images! Here are the images I shot and used for this project:

Recover homographies

A homography is used to warp one set of keypoints to another. This useful to define a relationship between two images with given keypoints so we can warp images to align with one another. In order to compute the homography, I use the relationship that A*h = b, where A defines a a set of equations for each keypoint given, b is the transformed points, and h is a 1x8 vector corresponding to the flattened homography. Here is the math I used to solve for the system of equations to solve for A:

In this case, we let the homography matrix be defined as such (note that the bottom right corner is a scaling factor, established to be 1):

We want to use our H matrix to warp the points in x to points in y using the following vectors, given that H * x = y.

In order to solve for the values in the H matrix, we can use least squares! So, we can create a system of equations to solve for these values, and construct an A matrix and b vector to solve for them! In order to put H in the correct format, we must stack every element on top of each other as such:

After doing some equation manipulation, we result at equations of the form

in the A matrix. For every i, these matrices of A would be stacked on top of each other! For every y value, you can also stack the b's into a vector of the y points as such:

Taking all of these terms, we are able to solve for the H matrix in the following least squares equation!:

In order to solve for this, I simply used the np.linalg.lstsq function. Now we are ready to use the homography to warp our images!

I tried to select around 20 keypoints. With any less than that, I found that the image came out blurry. I also found that if the keypoints were localized to a specific part of the image, it was more in focus at the region, but blurry in the regions where the keypoints were more sparse!

Warp the images

To warp the keypoints, I first found the homography matrix. Then, I defined the bounds of my warped image (to make sure that once it was warped, the final output still contained all of the colored pixels necessary and the image wasn't cut off). Then, I created a set of x and y indices for my original image, flattened these arrays, and multiplied them by the inverse homogenous matrix. This essentially creates an array of indices that represent where the original indices map to in the final, warped image. These are the computed input vectors that represent the x and y "target" / warped values, as compared to the original source image. After transforming these indices, we must normalize them in order to make sure they are still on the same scale as the original indices! We are then able to use the openCV remap function to compute the inverse interpolation for us, in order to find the ultiamtely warped image!

I was running into an issue where I'd warp the image just fine, but the image itself would get cut off. The solution to this was simply just to increase the width and height of the target image before using the cv2.remap function used to warp the image using bilinear interpolation, and this fixed the problem!

Image Rectification

Blend images into a mosaic

Finally, after creating our images, we can finally blend them together! The bulk of my logic for blending is contained in the layer_images function. I first alpha "feathered" both images. Since both of these images didn't have alpha channels, I manually added one (a channel of 1s). Then, I was able to For the left image, I linearly decreased the alpha value from 1.0 to 0.0, corresponding to the left and right sides of the image, respectively. I did this same sequence, but flipped, for the right images!

Then, since the warped image was resized to account for parts of the image falling out of the original image size, I also had to resize the non-warped image to be able to add them together accurately! In order to resize them, I simply padded the image vertically and horizontally with arrays of 0s to fill in the gap of the difference.

I noticed that if I added the points together at this stage, it wouldn't quite come out correctly because I would effectually end up averaging / normalizing parts of the image that were simply layered over arrays of 0s. As such, a way of combating this was to simply double the pixel values of the parts of the images that were not overlapping.

To do this, I replaced the zeros that were being added to the images with the pixel values of that part of the image, so when I normalized the whole image it ended up being the original / correct value for every part of the image!

and here's the final stitching + some other examples!:

Image 1: "ants-eye-view" of my apartment aka a photo I took on my floor

Image 2: Photo from my roof!

Image 3: Photo in MLK!

Cooolest part of the project :)

The coolest part of the project is the fact that we had the ability to stitch together images using only a few points keypoints! I had always wondered how panorama stitching worked, so being able to both learn that in class and then implement it myself was super gratifying and interesting!