Project 4, Part A: Image Warping and Mosaicing

by: Gavin Fure

Overview

We'll be using projective warping to create image mosaics. This is pretty similar to what we've done in previous projects, but here we get to use 8 degrees of freedom in our warps!

Part A1: Shooting the Pictures

I took a few pictures of the campenile:


Upper Image

Upper Image with points

Lower Image

Lower Image with points

Part A2: Recovering Homographies


In this section we will write a function to compute a homography, a transformation matrix from one set of correspondance points to another. This homography matrix will have 8 unknowns, allowing us to perform perspective warps. The original problem formulation looks like this:


p' = H*p

$$ \begin{bmatrix} wx'\\ wy'\\ w \end{bmatrix} = \begin{bmatrix} a & b & c\\ d & e & f\\ g & h & i \end{bmatrix} \begin{bmatrix} x\\ y\\ 1 \end{bmatrix} $$


Where p is the first set of correspondances (an nx2 matrix) and p' is the second (also nx2). H is a 3x3 matrix. We can reformulate this as a system of equations, specifying for each point:

$$a*x_1 + b*y_1 + c = w_1*x_1'$$ $$c*x_1 + d*y_1 + e = w_1*y_1'$$ $$g*x_1 + h*y_1 = w_1$$ $$...$$ $$a*x_n + b*y_n + c = w_n*x_n'$$ $$c*x_n + d*y_n + e = w_n*y_n'$$ $$g*x_n + h*y_n = w_n$$


For each point, we can substiture for wi to remove it:

$$a*x_1 + b*y_1 + c = (g*x_1 + h*y_1)*x_1'$$ $$c*x_1 + d*y_1 + e = (g*x_1 + h*y_1)*y_1'$$


We can formulate this system of equations as a least-squares problem:

$$A*h = b$$ $$ \begin{bmatrix} x_1, y_1, 1, 0, 0, 0, -x_1'*x_1, -x_1'*y_1\\ 0, 0, 0, x_1, y_1, 1, -y_1'*x_1, -y_1'*y_1\\ ....\\ x_n, y_n, 1, 0, 0, 0, -x_n'*x_n, -x_n'*y_n\\ 0, 0, 0, x_n, y_n, 1, -y_n'*x_n, -y_n'*y_n \end{bmatrix} \begin{bmatrix} a\\ b\\ c\\ d\\ e\\ f\\ g\\ h\\ i \end{bmatrix} = \begin{bmatrix} x_1'\\ y_1'\\ ...\\ x_n'\\ y_n'\\ \end{bmatrix} $$

As long as we have more than 4 points, we can solve this overdetermined least squares problem!

Part A3: Warp the pictures

Now that we have a function that can produce a homography matrix, we can use it to transform some images. Here is the result of warping the 'upper' campenile picture to the 'lower' one's points:


Warped Upper Image

Warped Upper Image with pts

Lower Image

Lower Image with points

Part A4: Rectify some images!

Now that our warp is working, we can warp some flat surfaces to directly face us! This is one my favorite features in phone scanner apps (like camscanner). It's really helpful, and can produce some really cool effects when applied in the right way. Here are my results:

Original Rectified

I used the corners of each object as correspondance points, and warped them to the corners of the image. This method was pretty quick and dirty, and doesn't preserve the original aspect ratios. That's why Link looks squished in that second warp. However, I think that this is proof our warp is working as intended!

Part A5: Create a Mosaic (Panorama)

At this point, we have everything we need to make some panoramams! We can calculate the homography matrix between the points on two images, then warp them onto the same canvase. Of course, we'll have to create a canvas that can fit all the pictures. To do this, we can send the corners of each image through its homography matrix, then use the max x and y values to create a blank canvas using np.zeros. If we have negative minimum values (which we often do!), then we can subtract the max values by the negative min values. Then, when we are warping, we can translate the results by the negative min values. This allows us to line everything up. For blending, I used a basic average mask. Each pixel value will be divided by the number of images that use that pixel. For example, if three images all include some arbitrary pixel, then the mosaic will add all of their corresponding pixel values to the output canvas. Since there are 3 of them, we divide the resulting value by 3. This is done by creating a mask (adding 1 to an empty array in the location of used pixels for each image). Then, we divide the raw mosaic by this mask.
Here are some of my results!


Upper Campenile

Lower Campenile

Campenile Blending Mask

Campenile Mosaic

This one turned out pretty good! The trees on the right are unfortunately a bit blurry, but it was windy that day so they probably moved around between pictures. There is a fairyly visible seam in the sky, because of the way I blended images. There wasn't nearly enough time for me to create a better blending option, but I think it turned out okay anyways. Let's look at another mosaic.


Left Gate

Right Gate

Left Gate Points

Right Gate Points


Gate Mosaic


The trees on this one are very blurry, probably also because of wind. Also a strange eyeball monster was walking through the gate while I was taking pictures, so they ended up moving. It's kind of a fun effect, they look like a ghost. The gate itself isn't as clear as it could be. This is probably due to point selection. I was only able to just click on them with my mouse (not do any sort of cross-correlation procedure around clicked points, which would definitely improve the results). This one is less successful than the others due to those factors. Let's look at one final mosaic:


Left Flag

Left Flag

Right Flag

Left Flag Points

Center Flag Points

Right Flag Points


Flag Mosaic


This one turned out pretty well! It has a decent amount of edge artifacts (again due to the simplistic blending technique I am using), but besides that it looks pretty good, and it uses 3 images!!

Part A Conclusion

We wrote functions to compute homographies, perspective-warp images according to those homographies, and compose warped images together to create a mosaic/panorama type image.