Overview
A single image inherently has limitations on how large a scene can be captured; however, when we mosaic, or stitch, many images together, we can create a larger field of view. The challenge is, how do we stitch the images together so they are nicely aligned? Then, how do we blend the images together into one cohesive image? We will attempt to create our own panoramas in this project.
Starting Images
Because we will compute homographies, which assume a projective transformation, our pictures must correspond to a projective transformation between each other. We do this by taking pictures with the same center of projection; standing in the same position, and simply rotating the camera around.
These are the photos I used (click to enlarge):
Lassen Volcanic National Park
Lassen Volcanic National Park
Crater Lake
Crater Lake
NYC
NYC
Homography, Warping, and Image Rectification
We compute the homography between 2 images using the equation p’=Hp, where H is a 3x3 matrix with 8 degrees of freedom (we set the lower right corner, the scaling factor, to 1). p and p' are vectors of corresponding points.
We first click on corresponding points in the two photos, to get these lists of correspondences (need at least 4 (x, y) correspondences to compute 8 degress of freedom). Then, we can compute the transformation between the two images: H.
Once we compute the transformation, we can warp all of image1 to align with image2 (or vice versa). Since H defines the transformation, we apply H (using inverse or forward warping), to transform our image into the desired shape.
To test this, I computed an image rectification. I took a picture of a rectangular object, at an angle, so that the object did not appear rectangular. I selected the 4 corners of the image, and manually set their "correspondences" to 4 corners of a rectified rectangle. Using the computed homography, I warped the original image into this new shape, and checked the object was indeed a rectified rectangle.
Original Image
Rectified Image
Creating Mosaics
For each pair of images above, I first expanded the images, to create space for the other one to "warp in". I clicked on correspondences, and computed the homography. Then, I warped one image to align nicely with the other.
Then, to blend the two images together, in order to avoid strong visible edges, I used the "spline" procedure (Laplacian stacks) from proj3. I used a sort of 3-tiered mask, to specify how much of each image I wanted in each region, then used this mask and Gaussian and Laplacian stacks to smoothly blend the two images.
Here are the resulting mosaics (click to enlarge):
Lassen Volcanic National Park
Crater Lake
NYC
The black borders around the edge are due to too much artificial enlarging at the beginning - I overestimated how much space was needed to add in the other photo.
You can also see some blurriness caused by imperfect aligning - this is probably due to the quality of the correspondences. Manually clicking on correspondences is not the most accurate thing in the world, and when there's a lot of little detail in the images to match up, little errors can be quite visible.
Summary
Blending the images was a struggle. Even though I used Laplacian stacks and a fancy spline, you can see places where the images aren't aligned very well. And this code takes a few minutes to run on my laptop! So the fact that I can create a way larger panorama, without any manual input, that's way nicer, and way faster, on my phone, is really impressive. I am extremely impressed by both phones, and the incredibly smart people who write panorama-creating software.
I thought this entire project was quite cool. Image rectification absolutely blew my mind. I was so impressed by the results; I felt so powerful, magical almost, in seemingly creating a different viewpoint out of nowhere. And though the code isn't exactly trivial, the theory behind the rectification and mosaic creating, isn't as complicated as it might seem. It's mind-boggling to me how just standard linear algebra, when applied in creative ways, can produce amazing effects.