CS294-26 Project 4 Image Warping and Mosaicing

Jimmy Xu

Overview

The goal of this project is to explore the magic of homography via two of its applications: image rectification and image mosaicing.

Shoot and Digitize Pictures

The first step is to take some pictures with the same center of projection (i.e. rotating, but not translating, the camera).

Bathroom

Bedroom

Lewis Hall

Recover Homographies

$H$ . We need at least 4 points.

By definition, perspective projection can be expressed as follows.

\begin{matrix} [\begin{array}{l} x_{a} \\ y_{a} \\ z_{a} \end{array}] = H_{3 \times 3} [\begin{matrix} x_{1} \\ y_{1} \\ 1 \end{matrix}], [\begin{matrix} \hat{x_{1}} \\ \hat{y_{1}} \\ 1 \end{matrix}] = \frac{1}{z_{a}} [\begin{array}{l} x_{a} \\ y_{a} \\ z_{a} \end{array}] \Rightarrow [\begin{matrix} {\hat{x}}_{i} z_{a} \\ {\hat{y}}_{i} z_{a} \\ z_{a} \end{matrix}] = [\begin{array}{lll} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{array}] [\begin{matrix} x_{i} \\ y_{i} \\ 1 \end{matrix}] \end{matrix}

$H$ . Following this pattern, we can add more points to the giant matrix on the left.

\begin{matrix} [\begin{array}{cccccccc} x_{1} & y_{1} & 1 & 0 & 0 & 0 & - x_{1} {\hat{x}}_{1} & - y_{1} \hat{x_{1}} \\ x_{2} & y_{2} & 1 & 0 & 0 & 0 & - x_{2} \hat{x_{2}} & - y_{2} \hat{x_{2}} \\ x_{3} & y_{3} & 1 & 0 & 0 & 0 & - x_{3} \hat{x_{3}} & - y_{3} \hat{x_{3}} \\ x_{4} & y_{4} & 1 & 0 & 0 & 0 & - x_{4} \hat{x_{4}} & - y_{4} \hat{x_{4}} \\ 0 & 0 & 0 & x_{1} & y_{1} & 1 & - x_{1} \hat{y_{1}} & - y_{1} \hat{y_{1}} \\ 0 & 0 & 0 & x_{2} & y_{2} & 1 & - x_{2} \hat{y_{2}} & - y_{2} \hat{y_{2}} \\ 0 & 0 & 0 & x_{3} & y_{3} & 1 & - x_{3} \hat{y_{3}} & - y_{3} \hat{y_{3}} \\ 0 & 0 & 0 & x_{4} & y_{4} & 1 & - x_{4} \hat{y_{4}} & - y_{4} \hat{y_{4}} \end{array}] [\begin{array}{l} h_{11} \\ h_{12} \\ h_{13} \\ h_{21} \\ h_{22} \\ h_{23} \\ h_{31} \\ h_{32} \end{array}] = h_{33} [\begin{matrix} {\hat{x}}_{1} \\ \hat{x_{2}} \\ \hat{x_{3}} \\ \hat{x_{4}} \\ \hat{y_{1}} \\ \hat{y_{2}} \\ \hat{y_{3}} \\ \hat{y_{4}} \end{matrix}] where h_{33} can be set to be 1 \end{matrix}

Warp the Images and Image Rectification

Once we have the homography matrix, we can warp the image. If we set the destination to be a rectangle, we can make a plane frontal-parallel. I referred to [2] for some help with cv2.remap.

We can do the same thing for a more extreme example

Blend the Images into A Mosaic

I can also warp one image to another and blend them, creating an image mosaic. The overlapping region of the two images are the average of them.

To be continued...

References

[1] https://towardsdatascience.com/estimating-a-homography-matrix-522c70ec4b2c

[2] https://stackoverflow.com/questions/46520123/how-do-i-use-opencvs-remap-function