Input images
The images I used are taken from the balcony at my home. The camera focus length is fixed when images are taken, and the two images are taken at a fixed position with a tripod.
Homography and Warping
From lecture, we know image projection from a fixed center of projection can be simulated by 2D image warping. The transformation that we will use is called homography, and it preserves straight lines but not parallelism (rectangle will be mapped to arvitray quadrilateral). To find the matrix for homography transformation, we select more than 3 pairs of corresponding coordinates and do Least-Square Regression to reduce distortion.
By transforming original coordinates on left image to corresponding new coordinates in the plane of right image, we can warp the left image to a new image in plane of right image.
Image Rectification
To test I have the correct function to find optimal transformation matrix for given corresponding coordinates, I try to rectify some images to front viewpoint. Below is an example by setting 4 corners of the book and 4 corners of the ipad to transform to 4 corners of a rectangle in front view respectively. The result image will be in top view.
Mosaic
To create the final mosaic, we blend the warpped left image (which already aligns with coordinates of right image) and right image together. The result is as below.
Another example with planar surface
The second example has images taken outside the window. Because the buildings are far away from the camera, we can regard them as on one planar surface. In this case, we can even use images which are not taken at a fixed camera center. The warping and blending result is as below.