Project 4A: Image Warping and Mosaicing

William Ren

Shoot and Digitize Pictures

To get some data, I first took a few pairs of images with matching correspondences using my phone. The images shown below are the images I end up stitching together in this project:

Recover Homographies

Since the pairs of images I plan on stitching together are not yet aligned, it is necessary to recover homographies, or transformation parameters, between sets of correspondence points on the images. I first manually picked correspondence points on each image pair using Adobe Photoshop. Given a set of p' = [x', y'] and p = [x, y] points, we must estimate a 3 x 3 transformation matrix H such that p' = Hp. By using the manipulation shown below for each set of correspondence points, we can solve for the a flattened representation of H:

Since I pick more than 4 sets of points, the problem is overconstrained. I can this solve for H by minimizing the norm of AH'=b using least squares.

Warp the Images

By calculating H, I am able to warp the correspondence points on one image to those on another image as shown below. To calculate the size of the resulting image, I use the bounding box containing the four corners of the warped image. I then translate the warped coordinates to fit within this new image frame. Here is an example:

Image 1

Image 1 warped to match correspondence points on image 2

Image 2

Image Rectification

To rectify an image, I take images with planar artifacts and warp them such that the plane is front-parallel. To do so, I first define source points on the image. I then define destination points corresponding to a rectangular shape. Some results are shown below:

Floor tiles	Rectified floor tiles
Open door	Rectified open door

Mosaic Blending

With the previous parts, I can now stitch together multiple images by defining correspondence points. I warp one image such that the correspondence points match up with the other image, then overlay them both on the same canvas. I calculate the new canvas bounds in a similar manner to the way I calculate those of purely warped images— by using the bounding box containing the warped corner points of one image and the original corner points of the other. I include an alpha channel as well, setting the alpha values of the area outside of the image to 0. I then average the alpha values of both images where they overlap, to achieve a somewhat smooth blending effect at the seams. Here are some image mosaics I've been able to create:

What I've Learned

From this part of the project, I've learned about the importance of defining a good point correspondence. The results of my stitches vary a lot depending on how close the correspondences are. I also found that manually defining these correspondences is super painful, which makes me excited for the next part of the project where I will figure out how to automatically determine point correspondences.