CS 194-26: Image Manipulation and Computational Photography
Kenny ChenProject 5: [Auto]Stitching Photo Mosaics
Overview
Part A: Image Warping and Mosaicing
We take two or more photographs and create an image mosaic by registering, projective warping,
resampling, and compositing them. Along the way, we compute homographies and warp images.
Shooting the Pictures
We shoot 2 or more images so that the transforms between them are projective. We do this by shooting
from the same point but in different view directions and overlapping fields of view. We can get
images of pretty good quality if we take pictures of planar surfaces or of very far scenes from different
views. The images used in this project were acquired using a Google Pixel 3XL.
Recovering Homographies
We want to solve for the homography matrix $H$, which is the transformation. We want to solve
$p^\prime=Hp$, where $H$ is 3x3 with 8 degrees of freedom. We have two $n\times2$ matrices that correspond
to $(x, y)$ locations of the $n$-point correspondences from the two input images. If we have $n=4$ points,
we can solve for $H$ without least squares. If $n>4$, we must use least squares because the system is
overdetermined. We opt to choose more correspondences because this decreases noise in the recovered homography.
We want
$$
\begin{bmatrix}
x^\prime \\
y^\prime \\
1
\end{bmatrix} =
\begin{bmatrix}
h_{11} & h_{12} & h_{13} \\
h_{21} & h_{22} & h_{23} \\
h_{31} & h_{32} & 1 \\
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
1
\end{bmatrix}
$$
With some algebra we end up with
$$
\begin{bmatrix}
x_1^\prime \\
y_1^\prime \\
x_2^\prime \\
y_2^\prime \\
x_3^\prime \\
y_3^\prime \\
x_4^\prime \\
y_4^\prime \\
\vdots
\end{bmatrix} =
\begin{bmatrix}
x_1 & y_1 & 1 & 0 & 0 & 0 & -x_1x_1^\prime & -y_1x_1^\prime \\
0 & 0 & 0 & x_1 & y_1 & 1 & -x_1y_1^\prime & -y_1y_1^\prime \\
x_2 & y_2 & 1 & 0 & 0 & 0 & -x_2x_2^\prime & -y_2x_2^\prime \\
0 & 0 & 0 & x_2 & y_2 & 1 & -x_2y_2^\prime & -y_2y_2^\prime \\
x_3 & y_3 & 1 & 0 & 0 & 0 & -x_3x_3^\prime & -y_3x_3^\prime \\
0 & 0 & 0 & x_3 & y_3 & 1 & -x_3y_3^\prime & -y_3y_3^\prime \\
x_4 & y_4 & 1 & 0 & 0 & 0 & -x_4x_4^\prime & -y_4x_4^\prime \\
0 & 0 & 0 & x_1 & y_4 & 1 & -x_4y_4^\prime & -y_4y_4^\prime \\
\vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots
\end{bmatrix}
\begin{bmatrix}
h_{11} \\
h_{12} \\
h_{13} \\
h_{21} \\
h_{22} \\
h_{23} \\
h_{31} \\
h_{32}
\end{bmatrix}
$$
Warping the Images
To warp the images, we use inverse warping, which gets rid of any noticeable artifacts.
We get all the coordinates in an image using the rectangle
function in the
skimage.draw
module. We transform all these coordinates using the homography
matrix $H$. We then set all the transformed points in a blank image to their corresponding
points in the old image.
Image Rectification
We take some images of planar surfaces and rectify them using the above process such that
the plane is frontal-parallel.
Blending the Images into a Mosaic
Part B: Feature Matching for Autostitching
Challenges
Conclusion
Overview
Part A: Image Warping and Mosaicing
We take two or more photographs and create an image mosaic by registering, projective warping, resampling, and compositing them. Along the way, we compute homographies and warp images.
Shooting the Pictures
We shoot 2 or more images so that the transforms between them are projective. We do this by shooting from the same point but in different view directions and overlapping fields of view. We can get images of pretty good quality if we take pictures of planar surfaces or of very far scenes from different views. The images used in this project were acquired using a Google Pixel 3XL.
Recovering Homographies
We want to solve for the homography matrix $H$, which is the transformation. We want to solve $p^\prime=Hp$, where $H$ is 3x3 with 8 degrees of freedom. We have two $n\times2$ matrices that correspond to $(x, y)$ locations of the $n$-point correspondences from the two input images. If we have $n=4$ points, we can solve for $H$ without least squares. If $n>4$, we must use least squares because the system is overdetermined. We opt to choose more correspondences because this decreases noise in the recovered homography.
We want $$ \begin{bmatrix} x^\prime \\ y^\prime \\ 1 \end{bmatrix} = \begin{bmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & 1 \\ \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$
With some algebra we end up with $$ \begin{bmatrix} x_1^\prime \\ y_1^\prime \\ x_2^\prime \\ y_2^\prime \\ x_3^\prime \\ y_3^\prime \\ x_4^\prime \\ y_4^\prime \\ \vdots \end{bmatrix} = \begin{bmatrix} x_1 & y_1 & 1 & 0 & 0 & 0 & -x_1x_1^\prime & -y_1x_1^\prime \\ 0 & 0 & 0 & x_1 & y_1 & 1 & -x_1y_1^\prime & -y_1y_1^\prime \\ x_2 & y_2 & 1 & 0 & 0 & 0 & -x_2x_2^\prime & -y_2x_2^\prime \\ 0 & 0 & 0 & x_2 & y_2 & 1 & -x_2y_2^\prime & -y_2y_2^\prime \\ x_3 & y_3 & 1 & 0 & 0 & 0 & -x_3x_3^\prime & -y_3x_3^\prime \\ 0 & 0 & 0 & x_3 & y_3 & 1 & -x_3y_3^\prime & -y_3y_3^\prime \\ x_4 & y_4 & 1 & 0 & 0 & 0 & -x_4x_4^\prime & -y_4x_4^\prime \\ 0 & 0 & 0 & x_1 & y_4 & 1 & -x_4y_4^\prime & -y_4y_4^\prime \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \end{bmatrix} \begin{bmatrix} h_{11} \\ h_{12} \\ h_{13} \\ h_{21} \\ h_{22} \\ h_{23} \\ h_{31} \\ h_{32} \end{bmatrix} $$
Warping the Images
To warp the images, we use inverse warping, which gets rid of any noticeable artifacts.
We get all the coordinates in an image using the rectangle
function in the
skimage.draw
module. We transform all these coordinates using the homography
matrix $H$. We then set all the transformed points in a blank image to their corresponding
points in the old image.
Image Rectification
We take some images of planar surfaces and rectify them using the above process such that the plane is frontal-parallel.
Blending the Images into a Mosaic
Part B: Feature Matching for Autostitching
Challenges
Conclusion