If you have multiple images from the same XYZ location, but different rotations, you can combine the images using a projective transform.
For a projective transform, given initial coordinates \(x\) and \(y\), we want to calculate matrix \(H\) such that $$ \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} wx' \\ wy' \\ w \end{bmatrix}$$ That gives us the equations $$ax + by + c = wx'$$ $$dx + ey + f = wy'$$ $$hx + gy + 1 = w$$ \(w\) is not a variable we want to solve for (since it is not an element of the homography), so we convert the above 3 equations into 2 using \(w = hx + gy + 1\) $$ax + by + c - hxx' - gyx' - x' = 0$$ $$dx + ey + f - hxy' - gyy' - y' = 0$$ We want the bottom right parameter of the matrix to equal 1, so we add a \(1000i = 1000\) constraint (1000 so that it is sufficiently weighted vs the other pixel-valued constraints), and then use least squares on a sufficient number of points (at least 4, I used 6 points each time).
Once we have a homography \(H\) converting points from image 1 to image 2, we can perform the warp. The pseudocode goes like this:
sk.RectBivariateSpline
)Once you have a warped image, you can combine the warped image and the image it was warped to to create a mosaic! Here are some examples
There are some issues with the borders here: I used a simple take-the-pixel-by-pixel-max blend, which worked pretty well except for the chairs in the Soda 6th floor image and teh portrait in the pool table image.
I learned a lot about homographies and transforms: esp. how to solve for a homography using least squares.