Panoramic images, a popular feature on your phone camera, are created by taking photos that overlap each other and stiching them together using projection.
Here we use perspective projection (or homographies) to stitch images together. We recover the homography by using least squares to find the homography that best maps the correspondence points of each image together.
The math:
We are trying to find $H$ such that $$H\textbf{p} = \textbf{p'}$$ where $\textbf{p}$ and $\textbf{p'}$ are pairs of correspondence points.
We fix $i$ to be $1$. Let $\textbf{h} = [a, b, c, d, e, f, g, h]^\intercal$. Then: $$\begin{align*} H \textbf{p} = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ 1 \end{bmatrix} &= \begin{bmatrix} wx_1' \\ wx_2' \\ w \end{bmatrix} = w\textbf{p'} \\ \end{align*}$$
We get the equations: $$\begin{align*} ax_1 + bx_2 + c &= wx_1' & & \dfrac{ax_1 + bx_2 + c}{gx_1 + hx_2 + 1} = x_1'& &ax_1 + bx_2 + c - gx_1x_1' - hx_2 x_1' = x_1'\\ dx_1 + ex_2 + f &= wx_2' & \implies & & \implies &\\ gx_1 + hx_2 + 1 &= w & & \dfrac{dx_1 + ex_2 + f}{gx_1 + hx_2 + 1} = x_2' && dx_1 + ex_2 + f - gx_1x_2 - hx_2 x_2' = x_2' \end{align*}$$
Thus, in our least squares problem, we are trying to minimize $$|| Ah - b||^2 $$ where $$A = \begin{bmatrix} p_{1, x_1} & p_{1, x_2} & 1 & 0 & 0 & 0 & - p_{1, x_1} p_{1, x_1}' & - p_{1, x_2} p_{1, x_1}' \\ 0 & 0 & 0 & p_{1, x_1} & p_{1, x_2} & 1 & -p_{1, x_1}p_{1, x_2}' & -p_{1, x_2}p_{1, x_2}' \\ \vdots & & & & & & & \vdots \\ p_{n, x_1} & p_{n, x_2} & 1 & 0 & 0 & 0 & - p_{n, x_1} p_{n, x_1}' & - p_{n, x_2} p_{n, x_1}' \\ 0 & 0 & 0 & p_{n, x_1} & p_{1, x_2} & 1 & -p_{n, x_1}p_{n, x_2}' & -p_{n, x_2}p_{n, x_2}' \\ \end{bmatrix}$$ and $$b = [p_{1, x_1}', p_{2, x_2}', \dots p_{n, x_1}', p_{n, x_2}']^\intercal$$
Rectification (aligning plane to plane)
Aligned either side of building to frontal plane
Aligned laptop screen to frontal plane
Stitching
Tried it on the sample image
Results are good for windows, but because I didn't have correspondence points for the rails, their integrity was sacrificed. However, windows are aligned very well because I always selected correspondence points on those planes.
Tried it on my own image, ok results.
Cropped
Lots of distortion at the edges, which is to be expected as we are nearing $180^\circ$ FOV. Still a little blurry, probably needs more correspondence points. Also has artifacts because of unequal exposure.