Image Warping and Mosaicing

Introduction

In this project we explore how to create panorama images! Specifically, we will take photos with the same point of view but with rotated viewing directions and with overlapping fields of view. Using point correspondences to recover homographies from the one image to the other, we can then perform a projective warp and blend the results together to create a panorama.

Pictures!

In order for this to work, we need some well taken pictures. Unfortunately, I don't happen to have a tripod at my disposal and we're under quarantine. That being said, I did my best to take some pictures using my smartphone.

I took some pictures of my room in my apartment and the view from out on my balcony from the same point of view but with different viewing directions, making sure that each pair of pictures had a substantial amount of overlap.

Before beginning to think of how to combine the two images into a panorama, we must define pairs of corresponding points on the two images by hand (yes, this can unfortunately get quite tedious). After defining pairs of corresponding points between the two images, we move on to the next part...

Recovering Homographies

Given the set of corresponding points between two images, we can relate the two according to the transformation $Hp = p'$ where $$H = \begin{bmatrix} a & b & c\\ d & e & f\\ g & h & 1 \end{bmatrix},$$ $$p = \begin{bmatrix} x_1 & & x_{n}\\ y_1 & \cdots & y_{n}\\ 1 & & 1 \end{bmatrix},\ \ p' = \begin{bmatrix} wx'_1 & & wx'_{n}\\ wy'_1 & \cdots & wy'_{n}\\ w & & w \end{bmatrix} $$ Thus, we have $8$, unknowns which we are trying to solve for with $n > 8$ equations. Since we have an overdetermined system, we will need to use a leas squares approximation to solve for the values of $H$. With some algebra, it can be shown this is equivalent to solving $Ax = b$ which can be written as the following: $$ \begin{bmatrix} x_1 & y_1 & 1 & 0 & 0 & 0 & -x_1 x'_1 & -y_1 x'_1 \\ 0 & 0 & 0 & x_1 & y_1 & 1 & -x_1 y'_1 & -y_1 y'_1 \\ & & & & \vdots & & & \\ x_n & y_n & 1 & 0 & 0 & 0 & -x_n x'_n & -y_n x'_n \\ 0 & 0 & 0 & x_n & y_n & 1 & -x_n y'_n & -y_n y'_n \\ \end{bmatrix} \begin{bmatrix} a \\ b \\ c \\ d \\ e \\ f \\ g \\ h \end{bmatrix} = \begin{bmatrix} x'_1 \\ y'_1 \\ \vdots \\ x'_n \\ y'_n \end{bmatrix} $$ For the purposes of this project, I ended up using correspondences anywhere between $n\in[20,30]$.

Image Rectification

Now that we have solved for the homographies, we should be able to perform an inverse warp to fill in every pixel value in the desired output image based on the corresponding coordinates in the input as defined by the homography transformation.

A cool application of this is image rectification! Here we take a few sample images with some planar surfaces, and warp them so that the plane is frontal-parallel. This is acheived by selecting 4 points representing the corners of the plane in the original image, and choose the correponding points in the output to be the corners of a rectangle. Here are some examples:

Mosaics

Now comes the challenge of building a mosaic (panorama) image! All we do is use the recovered homography between the two images and then warp one onto the image plane of the other. Then all that is left is to just blend the results together.

We just use a simple alpha blending on the resulting warps on the overlapping sections of the images. The results are shown below.