Project 5

Image Warping and Mosaicing by Amy Hung

CS 194-26: Image Manipulation and Computational Photography, Fall 2020

Part A

In part A, I'll be experimenting with different aspects of image warping through image mosaicing, by taking two or more photographs and creating an image mosaic by registering, projective warping, resampling, and compositing them. I'll also be computing homographies, and using them to warp images.

Shoot the Pictures

First off, I took some pictures that I could work with. I struggled with fixing the aperature/exposure on my smartphone camera, and didn't pivot properly in some of the images, the effects of which we'll see in a later step.

Recover Homographies

In this step, I then created a function computeH(im1_pts, im2_pts, num_pts) which takes in two sets of num_pts points from images, and finds the 3x3 homographic transformation matrix to warp im1 to im2. The basic n=4 system leads to an unstable homography recovery that is prone to noise, so I used an overdetermined linear system as described in another class's notes that I found online: Robert Collins CSE486, Penn State -- Lecture 16 Slide 29. I was able then approximate this overdetermined linear system using np.lstsq.

Warp the Images / Image Rectification

Using the homography matrices, we can now warp images by applying the matrix H onto each point, and interpolating the value into the final warped image. I accomplished this by creating a grid of indices over the entire image so that I could take advantage of matrix multiplication. From there, I interpolated the values using cv2.remap.

With that in hand, I can now rectify images by selecting points that are well defined (e.g. corners of a textbook or window), and warp them to artificially set straight/box points.

Blend the Images into a Mosaic

In this step, I apply the same process, but now warping img1 to img2's shape, and blending the two images together using a weighted average over overlapped regions. Notice here that there is a noticeable difference in the brightness of the sky along the edge between the two images, and the trees are extremely blurry. I believe this is due to the exposure/aperture changing on my smartphone camera (I was unable to fix the parameters), and small errors in selecting corresponding points between the two images. (Differences of a few pixels can result in large errors when applying the homographic transformation). Notice here again that the trees are extremely blurry. It was very difficult to find easy-to-reclick corresponding points between the two images without enough square items, corners, straight lines in the image. Again, blurry from misaligned corresponding points, and some "ghost" leaves that are in one image but not in the other.

Conclusion

Overall, I learned a lot about how powerful matrices are in applying a homography transformation. It was super cool to see seemingly meaningless numbers in a matrix be applied onto an image and create actual results, even artificially warping images into shapes that I wanted through image rectification. I also learned that selecting the initial point correspondences is extremely important in creating blur-free blended images.

Part B

In part A, I'll be implementing feature matching for autostitching images into a mosaic.

Harris Interest Point Detector

As the first step, I generated points of interest on the images by using the harris interest point detector.

Adaptive Non-Maximal Suppression

Notice that we have too many interest points generated to be of use to us, so I implemented Adaptive Non-Maximal Suppression, or ANMS. With this algorithm, we're only keeping the top 500 points with the largest radius to a neighboring stronger point (strength as determined by higher harris value). Now we have a much more useful and spread out set of points to work with.

Feature Descriptor Extraction

We now need to be able to identify the point and it's surroundings. So we now take a 40x40 patch of each point's surroundings, subsample it to 8x8, and normalize it to create a feature descriptor. Here are some examples:

Feature Matching

Now that every point has its own feature descriptor. we can compare points via their descriptors across the two images, to see how the points correspond/match between the images. Comparing each pair of points across the two images, I use Lowe's method to only keep pairs of points with a nn1/nn2 ratio below 0.3. This further reduces our point set to be useful points for aligning the two images.

RANSAC

With the points now matched up 1-to-1, I then implemented the RANSAC algorithm, which randomly samples 4 points at a time to generate a potential homography matrix, generates an inlier list of the warped points that are close enough to the intended destination point, and repeats, eventually choosing the largest list of inliers as the final point set of correspondences. The final set of points used to generate a final homography matrix:

The final warping of the image: Notice here that the final blended image still has some blurring on the Campanile -- I believe this is because I slightly rotated the camera when panning up to take the second image, and the current feature matching algorithm doesn't allow for rotation in matching the feature descriptors.

Other Autostitched Mosaics

Conclusion

Overall, I really enjoyed this project -- it was pretty cool to walk through the algorithms and the intentions behind them in understanding how to pick the best correspondence points. Through this, I learned that even automatic algorithms won't produce perfect results, but are a lot less painful than manually picking corresponding points :) I believe a portion of the blurriness in the resulting mosaics is partially due to incorrect photo taking on my part -- not pivoting correctly, being unable to fix aperture/exposure, not having enough overlap between the images, etc.