Part 1: Manual Rectification & Stitching
We seek to stitch together photomosaics to produce results similar to the "panorama" function on many smartphone camera applications.
Recovering Homographies
In order to rectify an image or stitch two images together we need to be able to recover a homography. This is a coordinate transform between pairs of keypoint correspondences \(\big\{(p_i, p_i')\big\}_{i=1}^{n}\) where \(p_i = [x_i ~~ y_i~~ 1]^T\) and \(p_i' = [x_i'~~ y_i'~~ 1]^T\), the two coordinates of the same keypoint of a scene in two different images.
A homography \(H\) is a map in \(\mathbb{R}^{3 \times 3}\) that given some image coordinates, will:
- Unproject them to rays into the optical center
- Apply a motion (rotation and maybe translation) on the optical center
- Reproject said rays into new image coordinates.
The homography \(H\) has eight degrees of freedom (as it can be solved up to a scaling factor): \[H = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1\end{bmatrix}\]
and applying \(H\) to a keypoint coordinate in the first image (using homogeneous coordinates) yields the corresponding keypoint coordinate in the second image:
\[w \begin{bmatrix}x_i' \\ y_i' \\ 1\end{bmatrix} = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1\end{bmatrix}\begin{bmatrix}x_i \\ y_i \\ 1\end{bmatrix} = H \begin{bmatrix}x_i \\ y_i \\ 1\end{bmatrix}\]
where \(w\) is a scaling value which is left unknown due to the fact that we do not know the true depth of the keypoint.
We want \(H\) to satisfy all \(n\) point correspondences:
\[ \begin{align} wp_1' &= Hp_1 \\ &\vdots \\ wp_n' &= Hp_n \\ \end{align} \]
and we can rewrite these relationships as the following matrix equation: \[ Ah = \begin{bmatrix} x_{1} & y_{1} & 1 & 0 & 0 & 0 & -x_{1} x_{1}' & -y_{1} x_{1}' \\ 0 & 0 & 0 & x_{1} & y_{1} & 1 & -x_{1} y_{1}' & -y_{1} y_{1}' \\ & & & & \vdots & & & \\ x_{n} & y_{n} & 1 & 0 & 0 & 0 & -x_{n} x_{n}' & -y_{n} x_{n}' \\ 0 & 0 & 0 & x_{n} & y_{n} & 1 & -x_{n} y_{n}' & -y_{n} y_{n}' \\ \end{bmatrix} \begin{bmatrix} a \\ b \\ c \\ d \\ e \\ f \\ g \\ h \end{bmatrix}= \begin{bmatrix} x_{1}' \\ y_{1}' \\ \vdots \\ x_{n}' \\ y_{n}' \end{bmatrix} = b \]
for \(A \in \mathbb{R}^{2n \times 8}, ~ h, b \in \mathbb{R}^8\). We note that for \(n=4\) correspondences there exists a unique solution homography but we will often use greater than four correspondences for stability, leaving an overdetermined system of equations. However, because we are students at Berkeley we are now salivating at the opportunity to deploy the most useful tool in an engineer's toolbox, ordinary least squares, to obtain an optimal homography \(h^*\):
\[h^* = \min_{h} \|Ah - b\|_2^2\]
Warping
To warp images using a homography \(H\), we use inverse warping and interpolaton instead of forward warping, which allows us to sample an interpolated pixel value in the source image for each pixel location in the warped image. This process is described in Project 3.
Rectification
We can rectify images by providing four vertices of a rectangular object in an image, even if it does not appear rectangular, and applying our method where the destination coordinates are the four corners of an image. This produces the following results:
