Part 1. Shoot Pictures
To get the data for this project, I took pairs of pictures on my iPhone throughout the day, whether it's of my Berkeley neighborhood, apartment building, or living room.
To get the data for this project, I took pairs of pictures on my iPhone throughout the day, whether it's of my Berkeley neighborhood, apartment building, or living room.
We have the following equations, which we can rewrite to set up a linear system of equations to solve for the homography matrix $H$ using least squares. $$\begin{bmatrix}x_a \\ y_a \\ z_a \end{bmatrix} = H \begin{bmatrix}x \\ y \\ 1 \end{bmatrix}, \begin{bmatrix}\hat{x_1} \\ \hat{y_1} \\ 1 \end{bmatrix} = \frac{1}{z_a} \begin{bmatrix}x_a \\ y_a \\ z_a \end{bmatrix}$$
We can rewrite these equations to get the following $$ \begin{bmatrix}\hat{x_1} z_a \\ \hat{y_1} z_a \\ z_a \end{bmatrix} = \begin{bmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \\\end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \\ 1 \end{bmatrix} $$
We can then substitute $z_a = h_{31} x_1 + h_{32} y_1 + h_{33}$ into the equations $\hat{x_1} z_a = h_{11} x_1 + h_{12} y_1 + h_{13}$ and $\hat{y_1} z_a = h_{21} x_1 + h_{22} y_1 + h_{23}$. We can repeat this process for all of the correspondence points we've defined $(x_i, y_i, \hat{x_i}, \hat{y_i})$. After setting $h_{33} = 1$, we can define the following system of equations: $$ \begin{bmatrix} x_1 & y_1 & 1 & 0 & 0 & 0 & -x_1 \hat{x_1} & -y_1 \hat{y_1} \\ 0 & 0 & 0 & x_1 & y_1 & 1 & -x_1 \hat{y_1} & -y_1 \hat{y_1} \\ x2 & y2 & 1 & 0 & 0 & 0 & -x2 \hat{x2} & -y2 \hat{y2} \\ 0 & 0 & 0 & x2 & y2 & 1 & -x2 \hat{y2} & -y2 \hat{y2} \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ x_n & y_n & 1 & 0 & 0 & 0 & -x_n \hat{x_n} & -y_n \hat{y_n} \\ \end{bmatrix} \begin{bmatrix} h_{11} \\ h_{12} \\ h_{13} \\ h_{21} \\ h_{22} \\ h_{23} \\ h_{31} \\ h_{32} \\ \end{bmatrix} = \begin{bmatrix} \hat{x_1} \\ \hat{y_1} \\ \hat{x_2} \\ \hat{y_2} \\ \vdots \\ \hat{x_n} \\ \hat{y_n} \end{bmatrix} $$
After defining the homography matrix, we can warp the images to be in the same projection space, letting us combine them down the road. Below are the resized im2's, the warped im1's, and the original im1's.
For image rectification, I define the correspondence points of a square-like object in the image and compute the homography matrix with the manually defined points $[[0, 0], [w, 0], [w, w], [0, w]]$. This flattens the square and "rectifies" the image.
For this part, I programmatically generated alpha masks to blend the resized im2 images with the warped im1 image. The masks were generated such that zones where only im2 pixels exist were set to $\alpha=1$, zones where only im1 pixels exist were set to $\alpha=0$. Zones where both pixels exist (contested regions) are set to $\alpha$ values that are negatively correlated with the distance to the center of the correspondence points.
$$\text{mask}[i, j] = \begin{cases} 0 & (i,j) \in (\text{im1} \setminus \text{im2}) \\ 1 & (i,j) \in (\text{im2} \setminus \text{im1}) \\ \gamma \sqrt{(i - c_r)^2 + (j - c_c)^2} & (i,j) \in (\text{im2} \cap \text{im1}) \end{cases}, \exists \gamma \in [0, 1]$$
This project was extremely interesting, but I learned how painstakingly difficult it is to define quality correspondence points. Furthermore, the importance of masking really stood out to me, because even if the correspondence points were good, stitching them together into a seamless photo is still a nontrivial task. While I only used simple alpha masking with a naive mask-creation algorithm, a better implementation would be to do multiresolution blending with a more sophisticated mask-creation strategy. Lastly, learning about how homography matrices can project images into the same plane was really cool!