Image Warping and Mosaicing

by Pauline Hidalgo

Overview

In this project, I compute and use homographies to do image rectification and create image mosaics.

Images and recovering homographies

Below are some images I am using to create an image mosaic, along with their correspondence points. As described in the spec, 4 correspondence points are enough to solve for the coefficients of the homography, but I use more points to improve the accuracy of the homography and solve for it using least squares. I used Photoshop to read off the pixel values and defined the correspondences as accurately as I could, but naturally there are errors. The below pictures show the actual points in green and the remapped points computed using the homography matrix in red.

Warping and rectification

Once H is calculated, I warp the images using inverse warping and interpolation for the pixel values. For rectification, I used my keyboard and a post-it note, which I know should be a rectangle (1:3 ratio) and square from the frontal-parallel perspective.

Image mosaic and blending

From the previous 2 parts, we can now make an image mosiac! One problem I faced was points that were not in the overlapping area of the images got cut off by the warp since they were mapped to negative locations. I resolved this issue by first multiplying the homography matrix by a translation matrix to shift any negative pixel mappings to positive so that they show up in the new image. I also padded the images by the amount of shift to ensure that there was enough space for both images once they are put together. I determined the amount of shift needed to fit both images by eye, which ended up being 4000 pixels for the below images.

left image

right image

mask

For blending, I took a weighted average of the pixels in the overlapping region. For the regions that did not overlap, I used the masks above to plug in the correct pixels directly into the final image. Unfortunately, the result below still has prominent seams, as well as blurring in the overlapping region! It's particularly noticeable from the many lines on the bridge, though the landscape in the background looks decent. These problems might be caused by error in my correspondence points, slight changes in my camera location (instead of only rotation) while taking pictures, and/or changes in lighting as I was using my phone. For reference, I also included the images just stacked on top of each other, without any blending. Now, there's no blurring, but the seams are stronger.

A couple more mosaics (of more nature scenes...not too much to see where I am in Illinois):

Overall, it was really interesting to see how much information is really encoded in images using image rectification! The most important thing I learned is to be careful when defining correspondence points, as fiddling with these changes the resulting warp a lot. I'm looking forward to automatically defining the points in the next part of this project.