Overview

The goal of this assignment is to recify images and combine several images into a single mosaic. Rectifying images involve finding the homographic transformation between the image and a predefined geometry. Combining images requires both alignment and cross-dissolve. For alignment, we need to find the homography between the points in the source image and destination image, before blending them together.

Shoot and digitize pictures

First, we need to have images to work on. I took photos of both indoors and outdoors, from the same point of view but with different view directions. I also made sure there are overlapping fields of view so that we can establish correspondence between images.

The below images shows the photos I took:

Recover Homographies

In order to do alignment between images or rectification, we need to recover the Homographies. The first step is to establish correspondence between corresponding points in source and destination images.

The homography matrix H is a 3x3 matrix connecting each pair of corresponding points. As shown in the formula below:

To solve matrix H, we need to vectorize this system of equations and include more data points. As seen above, it has eight degree of freedom, which means at least 4 points' corrdinates are required. Howvever, to reduce noise, we often select more correspondance than 4 to make the system overdetermined. Thus, the matrix H can be approximated using least square.

The vectorized equation is:

Warp the Images

After defining the correspondance, we can recover the homographies with the equation above. I used inverse warpping to avoid holes in the resulting image.

I first use the recovered homography to identify the size of the warpped image by warpping four corners of the original image. Then I used inverse warpping to find the coordinates in the original image and then interpolate these pixels. Lastly, we just put the interpolated values back to the resulting image. It's important to note that we need to scale the resulting coordinates by the third value (w) after warpping with homographies.

Image Rectification

Image rectification is a good way to test our warpping method. In this task, I warpped some photos to make the object in the photo frontal-parallel. In all the following images, I first picked some planar points (usually the corners of the object). Then I define some corresponding points based on my knowledge of the object's frontal-parallel aspect ratio. I used my method above to recover the homography, and then use the recovered homography to rectify the entrie photo to be frontal-parallel.

Below are some examples of the origianl images and the rectified face.

Blend the images into a mosaic

Now we are ready to blend some images!

To blend two images, we need to identify correspondance between the imageas. Then We can recover the homography. I used inverse warpping to ensure consistent result. With the homography, I first warpped the corners to find out the shape of the resulting image. Then for all pixels in the warpped image, I found the coordinates of their counterparts in the source image. I used these coordinates for interpolation. Lastly, I put the interpolated value back to the warpped image. This will make the two image aligned.

With the aligned images, we need to blend the colors together. We can not simply overlap them, because this will have edge effects. I implemented a two level Laplacian Pyramid to ensure smooth blending.

Below are some exmaple mosaics I blended:

For blending together more than two images, we can blend them one by one, using previous output as the input for the next blend, slowly growing the mosaic.

Bells and Whistles: Cat Replication!

Now I have two Nilas!

Final Thoughts

It is indeed very fun to work on this project. I used to use Photoshop to stitch photos together, but it was very high level. Now I really know the math behind this cool technique. One thing to note is that please remember to scale the warpped x and y by the third value of the warpped coordinates. Also, larger size and sigma gaussian kernel in Laplacian pyramid gives smoothier transition between images. However, the down side is that it takes really long to run!