CS 194-26 Project 4
Kyle Hua, CS194-26-AGX
IMAGE WARPING and MOSAICING
Manuel Feature Detection
Shoot Pictures
I shot various pictures using a Nikon camera from various angles. Some pictures are of my room and roomates along with some objects. In order to compute the homographies of the images, I selected similar points of interest between a couple of pictures and saved the points into a JSON file.
Recover Homographies
The 3x3 homography matrix H is the transformation matrix that transforms points from one image to another set of points as seen this equation: p′=Hp.
Expanding on this we get:
h11 x+h12 y+ h13=zx′
h21 x+h22 y+ h23= zy′
h31 x+h32 y+h33 = z
h33 is 1
We can then substitute z into the first two equations.
h11 x+h12 y+ h13= (h31 x+h32 y + 1) x′ → h11 x+h12 y+ h1 - h31 xx′+ h32 yx′ +x′ = x′
h21 x+h22 y+ h23= (h31 x+h32 y)y′ → h21 x+h22 y+ h23 - h31 xy′ -h32 yy′ = y′
Expanding on these equations we can build a system of equations up to N points. Below is an example of 4 points from each image. (Image sources: https://towardsdatascience.com/estimating-a-homography-matrix-522c70ec4b2c )
We can then use the least squares algorithm to solve for the H vectors.
Warp the Images
Once we have the homography, we can calculate the the new coordinates of one image in another images plane. Once we have the coordinates we can interpolate the pixel colors of our source image to the target plane. Here are some examples of warping angled pictures to a top down view (rectification)
Mosaic
With the projected images, we can create stich them together to make a mosaic. Here are some mosaics without blending.
Source Images:
Mosaics:
Source Images:
Mosaic:
Automatic Feature Detection
Feature Detection
Instead of manually selecting the points between the two images, we can have the computer do it for us instead! Using the Harris interest points, where the image has a wide spatial gradiant in all directions.
The resulting harris points (top 500 points):
Putting an image through the Harris point dector image, yields tens of thousands of points. We can use adaptive non-maximal suppression (ANMS) to select points that are spaced out and the maximum in their local area. This is done by sorting the list of points by their minimum supression radius. The equation is below. For each point in Xi we take the minimum distance of points in Xj where the harris value of xi is smaller than the harris value of xj times a constant. The equation is below.
The resulting amns points (500 points):
Extracting and Matching Features
Now with the processed amns points of the images, we need to find the matching points between the two. This can be done by taking features of each point in the image and find which feature in the other image matches best.
To get the features of the the anms points, we take a 40x40 pixel window of the a blurred version of the image and then downsize the window to around 8x8 pixels to compare it with other windows.
After we get the features of each image. We can find matching features by computing the SSD of one feature with every feature in the other image. We only want points that we have the most confidence in. As such we can follow Lowe's rule and take only features where the ratio most similiar, or nearest neighbor, and the 2nd most similiar is greater than a constant (.5 in my case). This ensures that we only have points where we know is a strong match in the other image.
The resulting matching features:
Ransac
While we got a set of matching points. There may still be some outliers that can throw off the homography calculation. Using randsac (random sample consensus) we can find the best points to keep. Ransac runs as follows:
- Repeat for 10000 iterations
- select four random points to use in each image
- calculate homography matrix for the four points
- apply homography to points in first image
- count the number of points where ssd(pj, Hpi) < epsilon
- pj are points in the second image, pi are points in first, epsilon is the threshold value.
- keep track of the largest set of points that are valid and return those at the end.
Resulting points from Ransac
Mosaic
We can stich the points together the same way as before.
Other examples: