Part 1

This project will eventually be about automatically stitching panoramas together but for now we are just trying to compute the homographies by hand. A homography is a 3x3 matrix that can sort of map equivalent points from one photo to another. It's calculated in the following way: First start with image correspondences, points that are the same in the real world but not the same pixel wise in the photo. You will need 8 points to compute a progressive transformation which is what we need. Stack them into a matrix like this: $$ p_i = \begin{bmatrix} -x_i \quad -y_i \quad -1 \quad 0 \quad 0 \quad 0 \quad x_ix_i' \quad y_ix_i' \quad x_i' \\ 0 \quad 0 \quad 0 \quad -x_i \quad -y_i \quad -1 \quad x_iy_i' \quad y_iy_i' \quad y_i' \end{bmatrix} $$ we then stack 4 of these to the the $$PH =0 $$ matrix where H is the homography we want and P is our matrix of points. $$ PH = \begin{bmatrix} -x_1 \quad -y_1 \quad -1 \quad 0 \quad 0 \quad 0 \quad x_1x_1' \quad y_1x_1' \quad x_1' \\ 0 \quad 0 \quad 0 \quad -x_1 \quad -y_1 \quad -1 \quad x_1y_1' \quad y_1y_1' \quad y_1' \\ -x_2 \quad -y_2 \quad -1 \quad 0 \quad 0 \quad 0 \quad x_2x_2' \quad y_2x_2' \quad x_2' \\ 0 \quad 0 \quad 0 \quad -x_2 \quad -y_2 \quad -1 \quad x_2y_2' \quad y_2y_2' \quad y_2' \\ -x_3 \quad -y_3 \quad -1 \quad 0 \quad 0 \quad 0 \quad x_3x_3' \quad y_3x_3' \quad x_3' \\ 0 \quad 0 \quad 0 \quad -x_3 \quad -y_3 \quad -1 \quad x_3y_3' \quad y_3y_3' \quad y_3' \\ -x_4 \quad -y_4 \quad -1 \quad 0 \quad 0 \quad 0 \quad x_4x_4' \quad y_4x_4' \quad x_4' \\ 0 \quad 0 \quad 0 \quad -x_4 \quad -y_4 \quad -1 \quad x_4y_4' \quad y_4y_4' \quad y_4' \\ \end{bmatrix} \begin{bmatrix}h1 \\ h2 \\ h3 \\ h4 \\ h5 \\ h6 \\ h7 \\ h8 \\h9 \end{bmatrix} = 0 $$ We add the constraint that the absolute value of H must be 1 and then use SVD so that $$P = USV^T$$. The last single value of V is the solution to H. From we can compute any point's corresponding point via $$ \begin{bmatrix} x' / \lambda \\ y' / \lambda \\ \lambda \end{bmatrix} = \begin{bmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{bmatrix}. \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$ Credit: This post for both explaining math and not making me write that god awful latex code.

This allows us to transform images like this:

As you can see this kind of looks like we are changing our perspective on the object. Which brings us to our next challenge. Image rectification.

Image rectification is a transformation process used to project images onto a common image plane. This process has several degrees of freedom and there are many strategies for transforming images to the common plane. (this was copied from the link provided because im getting lazy)

I'm going to use this amazing technology and math to make this picture of this weird serving tray thing look like I took it at a different angle.

Basically we apply the homography transformation from above to every pixel and move it to a new canvas. The homography is computed on the edges of the thing from the first photo and a set of coordinates I know to be square in the second image.If the canvas has holes in it we use bilinear interpolation to guess what should be there. If there's too many we blur and then down sample. Just like that we can magically make it look like I took this picture

Stringing Images Together

The next step in this whole process is attaching two images like you might when making a panorama. I'm currently quarantined so I apologize for these images being kinda lame. Here's two pictures of my backyard that I'm gonna try to string together.

Anyway here's that. I did a bad job defining corrospondances but I'm gonna find a work around for the next part.