Image Manipulation and Computational Photography

Project 6A: Image Warping and Mosaicing

Jianglai Zhang

Overview

This time let's explore how you can make a panorama, which is a wide-angle view of a space, by warping and stitching a set of pictures that share overlapping fields onto the same planar surface. The general idea is to first take these images with the same POV, or point of view, so that parallax is avoided, and then compute a homography matrix by defining pairs of corresponding points and use this matrix to warp the image(s) onto the plane we hope it is projected towards. At last, we use some blending technique such as multiresolution blending used last time to 'stitch' the images together.

Recovering Homographies and Warping

After taking some sets of nice pictures where each set shares some overlapping field and also shares the same POV, we can proceed to recover the homography matrix. Specifically, we hope to solve a linear system that is consisted of at least 4 pairs of in total 8 vector elements so that we can have a solution for all the 8 unknowns in the homography matrix H. Since we want high quality warping results, we would like to have more than 4 x 2 = 8 linear equations to solve and use least square to minimize the error in the system. Once we retrieve the 3 x 3 matrix H, we can use the inverse of it to inverse warp and get the corresponding pixel for each of the pixel in the result image we want, and after doing this for each RGB channel, we get the warped result image(s) that is mapped onto the plane we want.

Image Rectification

Now we have all we needed after computing the homography matrix and knowing how to warp the images, we can proceed and have some fun! The first application is to rectify images such that the result image is frontal-parallel to us. This is done by simply defining the second set of chosen points as the four corners of the image we want and play with the result matrix a bit. Here are some examples of the original and rectified images:

Mosaics

Another application is to create the panorama. The idea, as is introduced in the overview, is just to do the similar image warping procedure with an extra step of stitching the images together using some blending techniques. Here I used multiresolution blending with gaussian and laplacian stacks we did last time to make a smooth blend of the two images for each set. For the following results, I used two images for each set and mapped one of the two in each set onto the planar surface of the other; the number of corresponding point pairs I used is 8 for each set:

Note that the third set (the front of my house) of images have slightly different brightness and thus have more obvious stitching than the other two. Thus having a unchanging camera setting when taking the pictures is important:

Reflections

The idea of keeping the same POV and use homography and warping to create rectified images and panoramas is pretty cool. However, one thing to notice is to have the same camera settings when taking the same set of images (besides having the same POV, which is the most important thing).

Credits

Inspiration taken from friends, past projects, python libraries, papers and other online resources.