Overview
In this first part of project 5, we demonstrate two applications of perspective warping. First, we show image rectification by taking a picture of a planar surface, off-axis, and warping it such that the plane is frontal-parallel. Second, we blend two images taken from different view directions with some overlapping fields of view, effectively creating a mosaic or panorama of a larger field of view. The pixel positions of the warped images are determined by using a homography matrix.
Recovering Homographies
For the perspective warp, the first image is warped to the perspective of the second image. This is a homography where the new pixel locations p' are determined by a transformation of the original pixel locations p using a homography matrix: p'=Hp. The homography matrix H is a 3x3 matrix where the last element is always 1, meaning the matrix has 8 degrees of freedom (DoF). H is computed by taking points of correspondences between the original image and the perspective of the image it is being warped to, such that n linear equations are set up where n is the number of points of correspondence. Given that H has 8 DoF, only 4 points of correspondence are needed; however, more points of correspondence help reduce error in the homographies recovered. This can be formulated as Ah=b where A is the matrix composed of points of correspondences, h is an 8x1 vector for the 8 DoFs of H, and b is the vector composed of all the correspondences in the second image. This is shown below:
Thus, using least squares, we can solve for h and reshape it to form H. For part A of this project, the points of correspondence were all selected manually using ginput().
Image Rectification
In this section, we warp an image of a planar surface such that it is frontal-parallel. The perspective transformation is performed using inverse image warping, and thus uses the inverse of the homography matrix computed. Since we are transforming a single image to a different perspective, we need to define the correspondence points for the new perspective. This is done by using a physically known element of the planar surface, such as whether it is square or rectangle. Then, correspondence points can simply be defined. As such, in the pictures shown here, only 4 correspondences are defined: the corners of the planar surface in the image.
Here, we demonstrate image rectification for three examples. Notice that the canvas size was increased to bound the new perspective of the image.
In the future, resultant images could be "cropped" such that only the planar surface was recovered, excluding the extraneous portions of the image.
Image Mosaics
In this section we demonstrate the ability to recover homographies to warp an image to the perspective of another image, and blend both images together to create a panorama, or wider field of view of the subject matter. As before, this procedure begins with selecting correspondences between the two images to compute the homography matrix H. Next, we determine a larger bounding box to be able to display the warped image, and the second image. This is done by computing the dot product between H and a 3x4 matrix C containing the coordinates of the corners of the original, unwarped image:
Using the dimensions resulting from the aforementioned dot product, a blank canvas is formed, in which the warped image is first placed, and then the second, unwarped image is placed on top, overwriting any overlapping pixels. In the next part of the project, we'll implementing a better method to blend the two images together. The result of this part is shown here:
Final Thoughts
The coolest thing I've learned from this part of the project is the formulation of the linear algebra behind computing the homography matrix. Coding an implementation to solve for the matrix helped me understand how the perspective warp takes place, moving the pixels from the source locations to destination locations in the new perspective. Also, this once again gave me an appreciation for the simplicity and capability of least squares.