Project 5 Stitching Photo Mosaics

spring 2020

Image Warping and Mosaicing

Shoot the Picture

I used my smartphone to shoot 2 images of the backyard of my house. Then, those images (HEIC type) were converted to PNG type using online converter https://convertio.co. The images are shown below.

Recover Homography

The transformation $H$ of a homography is caluclated by $p' = Hp$, where H is a $3\times 3$ matrix with 8 degrees of freedom, $p$ and $p'$ are the pairs of correspondence points from two images. The correspondence points are manually labelled by the user input using the bare-bone $ginput$ function. Since I have more than 4 pairs of correspondence points, least squares is used.

Assume $H = \begin{bmatrix}a &b &c \\ d &e &f\\ g &h &1\end{bmatrix}$, then $\begin{bmatrix}x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix}a &b &c \\ d &e &f\\ g &h &1\end{bmatrix} \begin{bmatrix}x \\ y \\ 1\end{bmatrix}$ can be rewritten as:

$\begin{bmatrix} x &y &1 &0 &0 &0 &-xx' &-yx' \\ 0 &0 &0 &x &y &1 &-xy' &-yy' \end{bmatrix}\begin{bmatrix}a\\ b\\ c\\ d\\ e\\ f\\ g\\ h \end{bmatrix} = \begin{bmatrix}x' \\ y' \end{bmatrix}$, which is in the form of $\textbf{A}\vec{x} = \vec{b}$. Use $\textit{np.linalg.lstsq(A, b)}$ to solve for $\vec{x}$.

Probably, the easiest way to approximate $H$ is to use $H = p'p\dagger$ where $p\dagger$ is the pseudoinverse of $p$.

Warp the Images

Inversely warp images using homography whose parameters are calculated in the previous part, using similar approach as in project 3. The example of warping image 0 to the projection plane same as image 1 is shown below.

Image Rectification

For image rectification, choose a geometry whose vertices can be calculated by hand, i.e., a rectangle. To solve for the homography, it requires that the chosen geometry has more than three vertices. Two examples of image rectification are shown below, which are a FedEx box and a painting respectively. By inspecting the painting rectification, it is noticeable that the result is not in perfect rectangle. There might be several reasons. First, if there are few pixels off when choosing the corners of the painting on the original image, the homography $H$ can vary a lot because the limited number of points used. In other words, there is no exact solution for the four chosen points. Second, the painting might not be in rectangular shape. Third, the original image might be distorted. All of them lead to no exact solution of $H$.

Blend the Images into a Mosaic

To blend two images into a mosaic, I used alpha blending on the overlaped portion of those two images. The resulting mosaic backyard image is shown below. There is a clear edge artifact due to the difference in light intensity. Comparing the original images, image 0 and image 1, image 1 is darker. I also tried to blend other two sets of images, and both failed. There are so many reasons to cause this failure. I don't have a tripod, so the view point might have changed. It's currently raining, so that the light intensity varies. The correspondence points found by $\textit{ginput}$ are not the real correspondence, i.e., they are off by few pixels.

What I have Learned

It's exicting when I got it working on the backyard image. It is tedious and error prone to select points by clicking the image or reading pixel values, leading to the failure cases. I am looking forward to finding correspondence automatically.