We can form an image mosaic or panorama by taking multiple overlapping photos from the same point of view and combining them. To align these images properly, we need to warp them using perspective projection onto the same plane, requiring us to first recover homographies between the images that allow us to perform the perspective projection. Warping images allow us to "rectify" them, transforming surfaces in the image that are not frontal-parallel to be so. Once we've successfully warped the images onto the same projective plane, we can blend them together using alpha blending.

A homography is a perspective projection between two projective planes with the same center of projection. Finding a homography in this scenario requires finding a 3x3 perspective projection, i.e. finding 8 unknown variables in matrix H such that p' = Hp. p and p' in this case are

correspondences between the two images/projective planes.

Given 4 or more correspondences (4 p and p') between the two

images, we can use least squares to solve for the 8 unknowns

a, b, c, d, e, f, g, h (we set i = 1). We solve the equation on the

right to be equivalent to 0 to get the A and b in Ah = b, and run

numpy.linalg.lstsqto get h, giving us the 8 unknowns.

Once we have H, we can use it to warp from coordinates in one

image into coordinates for another image. We normalize these output coordinates so that w = 1 as in the input coordinates.

To warp from one image A to another image B, we first recover the homography/perspective projection from A to B. We then use inverse warping from A to B by first calculating the inverse of H, and then for every coordinate in B applying this inverse H onto it to get the corresponding coordinate in A. Note that these output coordinates in A will be float instead of integer, so we use linear interpolation to get the pixel value to reduce aliasing in the output image. In this case, I used scipy.interpolate.RegularGridInterpolator.

We can see image warping in action by performing rectification on some images. This transforms parts of images that are not frontal-parallel to appear to be on a frontal-parallel plane. Here we simply need 4 correspondences (where the points in the output form a rectangle) to get a good looking rectification. Below are a few examples of rectification of floor tiles, wall art, and a laptop screen.

For my image mosaic, I took three pictures (A, B, C) of my room. To blend them together, I first put the middle picture (B) into a large enough canvas to contain all three images (call this B'). I make sure to offset the points selected for B so that they are accurate for B'. Then I computed homographies from the left picture (A) to B' and from the right picture (C) to B'. From there I simply inverse warp from A and C to B'. To avoid edges in the blending, I applied alpha masks to each image, using simple linear fall off of values from the center. Below shows pictures I took in my room and the mosaics I created out of them.

This part helped me a lot with understanding how projective projection works and how it can be calculated easily using a set of correspondences and least squares. I was also surprised by how good the rectifications and mosaic looked. Given a high enough resolution starting image and linear interpolation, the rectifications look like they're completely separate photos taken from a different angle rather than a transformation of the original image, and I loved how that effect could be accomplished with just a simple perspective projection.