Image Warping and Mosaicing

CS 194-26: Computational Photography, Fall 2018

Tianrui Chen, cs194-26-aaz



In this project, we performed image warping to stitch multiple images to create seamless image mosaics.

Recovering Homographies

Two images on the same planar surface are related by a homography in the form of a projection matrix.

$H= \begin{bmatrix} a & b & c\\ d & e & f\\ g & h & 1 \end{bmatrix}$

We can solve for the homography matrix with a minimum of 4 point correspondences. However, in practice it is better to solve for the values through least square optimization with many more correspondences to account for noise and inconsistencies.

With $n$ points correspondences, $x_{1, n}$, $y_{2,n}$ matching to $x_{2,n}$, $y_{2,n}$, we can express the system of equations in the form $Ax=b$ and solve for x to get the homography matrix.

\[ \begin{bmatrix} x_{1,1} & y_{1,1} & 1 & 0 & 0 & 0 & -x_{1,1}x_{2,1} & -y_{1,1}y_{2,1} \\ 0 & 0 & 0 & x_{1,1} & y_{1,1} & 1 & -x_{1,1}x_{2,1} & -y_{1,1}y_{2,1} \\ &&&&\vdots\\ x_{1,n} & y_{1,n} & 1 & 0 & 0 & 0 & -x_{1,n}x_{2,n} & -y_{1,n}y_{2,n} \\ 0 & 0 & 0 & x_{1,n} & y_{1,n} & 1 & -x_{1,n}x_{2,n} & -y_{1,n}y_{2,n} \\ \end{bmatrix} \begin{bmatrix} a\\b\\c\\d\\e\\f\\g \end{bmatrix} = \begin{bmatrix} x_{2,1} \\ y_{2,1} \\ \vdots \\ x_{2,n} \\ y_{2,n} \end{bmatrix} \]

Actual point correspondences were taken manually by hand using matplotlib.pyplot.ginput. This was not fun.

Image Warping

After obtaining the homography matrix, we performed image warping. We first mapped the corner coordinates through the homography matrix to get the bounds of the resulting warped image. With the transformed corners, we found all the pixel coordinates in the resulting image that need to sample from the original image using scikit's polygon function. We then found the inverse of the homography matrix to perform inverse warping to find where to sample from the original image for each result pixel. When sampling , we used interp2d to prevent aliasing.

Care was taken when performing transforms as resulting transformed points must be divided by the homogeneous depth value to all lie on the same plane.

With image warping completed, we could apply it to image rectification to test the correctness of our procedure.

woz rectify woz
desktop flatscreen

Creating Mosaics

To create a mosaic from two images, we find the homography matrix from corresponding points of the two images, and then warp one of the images to the perspective of the other.

To perform blending on the two images, we performed alpha blending by multiplying the transformed images with a gaussian blurred mask and stored the two images on a large empty image with the proper offsets. The mask of each individual image was also combined to create the alpha channel of the resulting image to allow transparency in the empty parts of the final result.

These did not work out perfectly with minor ghosting and slight mismatches. This is probably due to imprecision selecting correspondences by ginput and shakiness when I was taking the images. Low light photography also contributed to the blurriness.

To conclude, the most important thing we learned this project is how homography projections work and can be calculated. It made for an interesting, but slightly tedious project.

Note: I pray that I will never have to use ginput to select a bunch of point correspondences again.