CS 194-26 Project 4

Kyle Hua, CS194-26-AGX

IMAGE WARPING and MOSAICING

Manuel Feature Detection

Shoot Pictures

I shot various pictures using a Nikon camera from various angles. Some pictures are of my room and roomates along with some objects. In order to compute the homographies of the images, I selected similar points of interest between a couple of pictures and saved the points into a JSON file.

Selected Points

Recover Homographies

The 3x3 homography matrix H is the transformation matrix that transforms points from one image to another set of points as seen this equation: p′=Hp.

Expanding on this we get:

h11 x+h12 y+ h13=zx′

h21 x+h22 y+ h23= zy′

h31 x+h32 y+h33 = z

h33 is 1

We can then substitute z into the first two equations.

h11 x+h12 y+ h13= (h31 x+h32 y + 1) x′ → h11 x+h12 y+ h1 - h31 xx′+ h32 yx′ +x′ = x′

h21 x+h22 y+ h23= (h31 x+h32 y)y′ → h21 x+h22 y+ h23 - h31 xy′ -h32 yy′ = y′

Expanding on these equations we can build a system of equations up to N points. Below is an example of 4 points from each image. (Image sources: https://towardsdatascience.com/estimating-a-homography-matrix-522c70ec4b2c )

We can then use the least squares algorithm to solve for the H vectors.

Warp the Images

Once we have the homography, we can calculate the the new coordinates of one image in another images plane. Once we have the coordinates we can interpolate the pixel colors of our source image to the target plane. Here are some examples of warping angled pictures to a top down view (rectification)

Mosaic

With the projected images, we can create stich them together to make a mosaic. Here are some mosaics without blending.

Source Images:

Mosaics:

Source Images:

Mosaic:

Automatic Feature Detection

Feature Detection

Instead of manually selecting the points between the two images, we can have the computer do it for us instead! Using the Harris interest points, where the image has a wide spatial gradiant in all directions.

The resulting harris points (top 500 points):

Putting an image through the Harris point dector image, yields tens of thousands of points. We can use adaptive non-maximal suppression (ANMS) to select points that are spaced out and the maximum in their local area. This is done by sorting the list of points by their minimum supression radius. The equation is below. For each point in Xi we take the minimum distance of points in Xj where the harris value of xi is smaller than the harris value of xj times a constant. The equation is below.

The resulting amns points (500 points):

Extracting and Matching Features

Now with the processed amns points of the images, we need to find the matching points between the two. This can be done by taking features of each point in the image and find which feature in the other image matches best.

To get the features of the the anms points, we take a 40x40 pixel window of the a blurred version of the image and then downsize the window to around 8x8 pixels to compare it with other windows.

After we get the features of each image. We can find matching features by computing the SSD of one feature with every feature in the other image. We only want points that we have the most confidence in. As such we can follow Lowe's rule and take only features where the ratio most similiar, or nearest neighbor, and the 2nd most similiar is greater than a constant (.5 in my case). This ensures that we only have points where we know is a strong match in the other image.

The resulting matching features:

Ransac

While we got a set of matching points. There may still be some outliers that can throw off the homography calculation. Using randsac (random sample consensus) we can find the best points to keep. Ransac runs as follows:

Resulting points from Ransac

Mosaic

We can stich the points together the same way as before.

Other examples: