CS 194 Project 5: [Auto]Stitching and Photo Mosaics

Emily Hu


Part 1: Image Warping and Mosaicing

Image Rectification

In both the following examples, I had one input image and used the power of homography transformations to project the picture in a way that output what a camera would see if it was rotated to hover directly above the image plane.

Square Picture on iPad Screen

Left: Input, Right: Warped/Rectified Output

Rubik's Cube

Left: Input, Right: Warped/Rectified Output

Warping & Blending Images into Mosaics

In the following examples, I took the three displayed input images and warped each of the side pictures with a homography transformation to project that image so its pictured features would align with those in the center image. These resultant warps would then be stitched and blended together using an alpha channel and the Python variation of Matlab's bwdist function.

Playground

Neighborhood Street

Blacktop

Grass Field

Error Analysis: When I zoom into the image, some ghosting/blurring is observed around the center trees. After I ran this same mosaic constructing algorithm on the other image sets without the same issues, I concluded that this is the result of the slight movement of the trees in the wind. Though I did my best to avoid taking pictures of scenes with moving subjects, the result of this panorama stitch shows that even the fractions of a second between the time when I took the left image and the right image made a difference as the trees swayed every so slightly in the wind. This may be why the panorama of the still playground came out better.

Reflection: Part 1

The most important thing I learned in this part was the fact that the third row of the homography matrix are weights which the first and second rows must be divided by. I previously thought this was an arbitrary value to be cropped off.

Part 2: Feature Matching for Autostitching

Detecting Corner Features in an Image

Harris Corners

Shown below in red are the points returned by the basic Harris corner detection algorithm, overlayed on the leftmost and middle image of the playground mosaic.

Adaptive Non-Maximal Suppression

Shown below in red are the points remaining after ANMS is applied, overlayed on the leftmost and middle image of the playground mosaic. I used the recommended ANMS 500 here so only the 500 strongest Harris corner points are still remaining.

Feature Matched Points

Shown below now are points which have been determined to be matching features. The dots are color-coded so that the purple dot in the first image corresponds with the purple dot in the second image. Theoretically, this holds true for all the colored dot pairs in these pictures.

The same pictures and points as above are repeated below but with all dots displaying as red, for ease of noticing where the dots are.

Warping & Blending Images into Mosaics

Below are some results of the autostitching process, made possible by the feature matching demonstrated above.

Playground

Autostitching Result

Manually Matched Result

For this mosaic, the results are pretty similar between the automatically stitched and manually notated panoramas, aside from a bit of ghosting/blur visible in the top center portion of the automatically stitched version. However, I noticed the automatic feature matching algorithms had some issues with the content of this specific image set which prevented it from drawing enough matching correspondences between the right input image and this complementary panorama. Below the remaining panorama results, I will later on go into more detail regarding this input-dependent performance of the program.

Neighborhood Street

Autostitching Result

Manually Matched Result

Once again, for this mosaic, the results are pretty similar between the automatically stitched and manually notated panoramas, aside from a bit of ghosting/blur visible in the center portion of the automatically stitched version, in the branches of the tree.

Blacktop

Autostitching Result

Manually Matched Result

The autostitching algorithm really struggled with this image set, given that most of the overlap space between these two input photos was taken up by the negative space of the sky and blacktop—both of which are notably uniform and impossible for me to discern manually either. That left a much more limited pool of options for correspondence points, so the autostitching algorithm did not perform as well in this case in terms of determining which corner features match with which, but even I had a hard time matching points for these two images, despite my advantage of human context and knowledge of the subject. Similar to the playground panorama, the algorithm was also unable to stitch together the remaining right side of this panorama, for possible reasons that will be discussed below. Still, I definitely should have taken these pictures with a greater overlap space, given what I know now about the adverse effects of large expanses of uniform negative space.

Grass Field

Autostitching Result

Manually Matched Result

Once again, for this mosaic, the results are pretty similar between the automatically stitched and manually notated panoramas, aside from a bit of ghosting/blur visible in the center portion of the automatically stitched version, in the branches of the tree.

As mentioned above, I noticed a lot of issues with certain input images led to a failure of the feature matching algorithm. I break each case down in the comments above my code file, but the general takeaways I gleaned from processing the panoramas above was that actual content depicted in the image was just as relevant to the quality outcome as the image processing itself. Images with lots of negative space which more or less looked the same in terms of colors, textures, or even repeating patterns did not fare well in the automatic feature matching, which makes sense when we think about how the program is processing feature descriptors. Along these lines, the smaller and more repetitive/homogenous an overlap area appeared, the less likely the program would be able to match features across input images. Put simply, any images that a human would struggle a bit more annotating likely means the program will also have difficulty recognizing and discerning features. With all this in mind, I reshot a couple scenes and the resulting panoramas are shown below.

Street Cars

The blurring in the center portion of this image is likely due to a couple factors: the slight swaying of the trees, the aperture blur from the camera, and the more visually homogenous nature of the content in this overlap section.

Suburbia

Left Half

Right Half

I also tried to stitch these two panoramas together into a larger panorama spanning twice the width, but the program was not able to feature match enough points to compute a homography for warping the middle images to fit together.

Reflection: Part 2

The most interesting thing I learned from this project was that I can rectify any shots I take of food in restaurants or cafes if I don't want to stand up hovering over the food for a good shot. I often struggle with this especially when there is overhead lighting casting shadows of my phone down onto the food, so in the future, I may just try this method and project the picture from a regular perspective into the illusion of a top-down shot for the perfect food photo.

The most important thing I learned in this part was the importance of recognizing and understanding the limitations of whatever technology or system is being used. By having a stronger grasp on the limits of what these tools can do for us, we as the user can better adjust what we can control, which in this case was the content of input images. Whenever possible, we can use foresight to work around the potential shortcomings of a certain program to produce results now, instead of continuously tweaking the algorithms themselves, waiting for more advanced technology.