Programming Project #6 (Part 1): Image Warping and Mosaicing

This project focuses on warping images in order to blend them together in order to form a mosaic. I've always been curious about how the iPhone constructs its panorama photos, so I found this project very interesting. I also found the math aspect of this project rather challenging, since this was one of the first time we were working in the z-plane in addition to the x and y.

Image Warping

In this case, the transformation for warping an image is a homography. You can see on the right that the homography matrix should transform a given point into its warped position. To solve for the variables in the homography matrix, I set up an equation Ax = b where b held the coordinates that I was trying to warp my image into. I had some difficulty setting up the A matrix for this equation, but eventually after some help from Piazza, managed to create a 3x3 homography matrix.

Image Rectification

After figuring out how to set up an equation to solve for a homography matrix, I inputted my corresponding points from an image. To warp the image to be frontal-parallel, I chose objects in the images that were rectangular or square and used the corners as points. I then clicked an approximation of the shape that the object should be if the image were to be frontal-parallel and entered these as the parameters for my calculateH function.

Once I got the homography matrix, I used inverse-warping to create the final image. This was similar to previous projects, with the main difference being that after I got the x and y coordinates from taking the dot product of the initial points and H, I divided x and y by the z value of the point in order to account for warping in the z-dimension. I additionally created a function to accomodate for the larger dimensions resulting from the warped corners for the house image.

Notebook: Unwarped
Notebook: Frontal-Parallel
House: Unwarped
House: Frontal-Parallel

Blending Images into a Mosaic

To blend images together, rather than using a square or rectangle approximation like in the image rectification section, I used corresponding points for overlapping features between the two images. Then I would warp one image into the perspective of the other and then naively blended the two. When I used enough points, the naive blending worked rather well, but as you can see, using 4 or 8 points had some misalignment. After realizing that 12 points provided the best results, I created three total mosaics as seen below.

4 input points
8 input points
12 input points
Image 1
Image 2
Street View: Mosaic
Image 1
Image 2
Vines: Mosaic
Image 1
Image 2
House: Mosaic

Overall from this part of the project, I learned a lot more about what homographies are and how to solve for them. It's different hearing about it conceptually in class and actually getting your hands dirty to code it yourself. I realize that it's important to make sure the equations are set up correctly and that having more data points can solve a lot of problems (as Professor Efros as said many times.) I honestly spent so much time trying to figure out what was wrong with either my mosaic or warping function, only to realize that the mosaics were unsatisfactory due to me using too little corresponding points as references. Despite the frustrations though, I genuinely enjoyed this part of the assignment, since it answered some questions I've had for a while about stitching photos together for panorama-style results.