Assignment 3-2

CS 194-26: Image Manipulation and Computational Photography

Kenny Chen

Project 5: [Auto]Stitching Photo Mosaics

Overview

Part A: Image Warping and Mosaicing

We take two or more photographs and create an image mosaic by registering, projective warping, resampling, and compositing them. Along the way, we compute homographies and warp images.

Shooting the Pictures

We shoot 2 or more images so that the transforms between them are projective. We do this by shooting from the same point but in different view directions and overlapping fields of view. We can get images of pretty good quality if we take pictures of planar surfaces or of very far scenes from different views. The images used in this project were acquired using a Google Pixel 3XL.

Recovering Homographies

We want to solve for the homography matrix $H$, which is the transformation. We want to solve $p^\prime=Hp$, where $H$ is 3x3 with 8 degrees of freedom. We have two $n\times2$ matrices that correspond to $(x, y)$ locations of the $n$-point correspondences from the two input images. If we have $n=4$ points, we can solve for $H$ without least squares. If $n>4$, we must use least squares because the system is overdetermined. We opt to choose more correspondences because this decreases noise in the recovered homography.


We want $$ \begin{bmatrix} x^\prime \\ y^\prime \\ 1 \end{bmatrix} = \begin{bmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & 1 \\ \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$

With some algebra we end up with $$ \begin{bmatrix} x_1^\prime \\ y_1^\prime \\ x_2^\prime \\ y_2^\prime \\ x_3^\prime \\ y_3^\prime \\ x_4^\prime \\ y_4^\prime \\ \vdots \end{bmatrix} = \begin{bmatrix} x_1 & y_1 & 1 & 0 & 0 & 0 & -x_1x_1^\prime & -y_1x_1^\prime \\ 0 & 0 & 0 & x_1 & y_1 & 1 & -x_1y_1^\prime & -y_1y_1^\prime \\ x_2 & y_2 & 1 & 0 & 0 & 0 & -x_2x_2^\prime & -y_2x_2^\prime \\ 0 & 0 & 0 & x_2 & y_2 & 1 & -x_2y_2^\prime & -y_2y_2^\prime \\ x_3 & y_3 & 1 & 0 & 0 & 0 & -x_3x_3^\prime & -y_3x_3^\prime \\ 0 & 0 & 0 & x_3 & y_3 & 1 & -x_3y_3^\prime & -y_3y_3^\prime \\ x_4 & y_4 & 1 & 0 & 0 & 0 & -x_4x_4^\prime & -y_4x_4^\prime \\ 0 & 0 & 0 & x_1 & y_4 & 1 & -x_4y_4^\prime & -y_4y_4^\prime \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \end{bmatrix} \begin{bmatrix} h_{11} \\ h_{12} \\ h_{13} \\ h_{21} \\ h_{22} \\ h_{23} \\ h_{31} \\ h_{32} \end{bmatrix} $$

Warping the Images

To warp the images, we use inverse warping, which gets rid of any noticeable artifacts. We get all the coordinates in an image using the rectangle function in the skimage.draw module. We transform all these coordinates using the homography matrix $H$. We then set all the transformed points in a blank image to their corresponding points in the old image.

Image Rectification

We take some images of planar surfaces and rectify them using the above process such that the plane is frontal-parallel.



Blending the Images into a Mosaic

Part B: Feature Matching for Autostitching

Challenges

Conclusion