Project 5 Part A

Image Warping and Mosaicing by Amy Hung

CS 194-26: Image Manipulation and Computational Photography, Fall 2020

Overview

In this project, I'll be experimenting with different aspects of image warping through image mosaicing, by taking two or more photographs and creating an image mosaic by registering, projective warping, resampling, and compositing them. I'll also be computing homographies, and using them to warp images.

Shoot the Pictures

First off, I took some pictures that I could work with. I struggled with fixing the aperature/exposure on my smartphone camera, and didn't pivot properly in some of the images, the effects of which we'll see in a later step.

Recover Homographies

In this step, I then created a function computeH(im1_pts, im2_pts, num_pts) which takes in two sets of num_pts points from images, and finds the 3x3 homographic transformation matrix to warp im1 to im2. The basic n=4 system leads to an unstable homography recovery that is prone to noise, so I used an overdetermined linear system as described in another class's notes that I found online: Robert Collins CSE486, Penn State -- Lecture 16 Slide 29. I was able then approximate this overdetermined linear system using np.lstsq.

Warp the Images / Image Rectification

Using the homography matrices, we can now warp images by applying the matrix H onto each point, and interpolating the value into the final warped image. I accomplished this by creating a grid of indices over the entire image so that I could take advantage of matrix multiplication. From there, I interpolated the values using cv2.remap.

With that in hand, I can now rectify images by selecting points that are well defined (e.g. corners of a textbook or window), and warp them to artificially set straight/box points.

Blend the Images into a Mosaic

In this step, I apply the same process, but now warping img1 to img2's shape, and blending the two images together using a weighted average over overlapped regions. Notice here that there is a noticeable difference in the brightness of the sky along the edge between the two images, and the trees are extremely blurry. I believe this is due to the exposure/aperture changing on my smartphone camera (I was unable to fix the parameters), and small errors in selecting corresponding points between the two images. (Differences of a few pixels can result in large errors when applying the homographic transformation). Notice here again that the trees are extremely blurry. It was very difficult to find easy-to-reclick corresponding points between the two images without enough square items, corners, straight lines in the image. Again, blurry from misaligned corresponding points, and some "ghost" leaves that are in one image but not in the other.

Conclusion

Overall, I learned a lot about how powerful matrices are in applying a homography transformation. It was super cool to see seemingly meaningless numbers in a matrix be applied onto an image and create actual results, even artificially warping images into shapes that I wanted through image rectification. I also learned that selecting the initial point correspondences is extremely important in creating blur-free blended images.