CS194-26 Project 6A: Image Stitching
Hersh Sanghvi | cs194-26-add
In this project, we learn how, based on correspondences, we can solve for transformations between images. Similar to a previous project in which we solved for affine transformations between triangles, we now generalize that to homographies between entire images. This comes in handy when stitching images together for panoramas, where you know that the camera will only be rotating around the optical center of the image. Thus, based on this knowledge, you can solve for a 3x3 homography matrix that relates the two images based on the correspondences, using SVD.
This matrix has 8 degrees of freedom, since we specify that the last element, h22, must be 1.
Part 1: Image Rectification
By manually specifying points in a planar image taken at an angle, we can rectify it by applying a homography. Examples are shown below (some artifacts may appear due to downsampling):
Notice how the rectification does not handle 3D aspects very well! For example, the speakers in the circuit become extremely stretched out, and on the keyboard, the keys appear to be much thicker than they actually are.
Part 2: Image Stitching
In this part, the task is to manually specify correspondences between two images that are taken from the same optical center, but with the camera rotated around the center, use those correspondences to calculate a homography that will then warp one of the images into the plane of the other image, and then stitch the two. This was quite challenging, as there are generally artifacts due to the fact that the images are not perfect rotations, and the stitching can be finicky due to lighting changes or translational motion. I found that using 8-10 correspondences worked well enough. Although you only need 4 at minimum, providing more helps to more robustly estimate the matrix. For the blending, I used a nonlinear blend in the overlap region. I basically took the 2-norm distance between the points in the overlap and the edges of the overlap region as the alpha factor for my mask. I found this worked well since it took into account both x and y into the masking, and thus produced natural looking results.
Here are some examples:
Left image warped into the plane of the right image:
There is a slight discrepancy at the border, due to a small vertical displacement between the frames (this shows up especially in the handrail).
Left image warped into right plane:
There is no stitch artifact here, but the plants are blurry because of both wind and because small translations produce disproportionate effects at close range. This is why having a stationary environment is important during taking a panorama. Also notice how certain people appear transparently in the walkway. This is because they moved between frames! Looks pretty cool.
I learned a lot during this project! It was interesting to see how theoretically simple homography matrices are, but the gory details required when inverse warping images.