CS 194 Project 6: Image Warping and Mosaicing

Michelle Ling

cs194-26-abk

In this project, we examine how to create a mosaic - or stitch multiple photos together into one picture. This project will involve taking two photographs and create a mosaic by registering, projective warping, resampling and compositing the pictures. In order to do this, we will compute homographies and use them to warp the images to allow for the mosaic to be created.

Manual Mosaics

Part 1: Image Rectification

We begin the project by first taking images and rectifying them into a different plane. By doing so we manipulate the images such that it looks as though we are viewing them from a different perspective. We will pick at least four points on the image and match these points with those of a shape that we want these points to look like in the new perspective. After we have these pairs, we want to calculate the parameters of the homography which we will use to transform the image. where p' = H * p. This homography has 8 degrees of freedom which we simpliy into 7 by setting the scaling factor, w, to be 1. We then solve for these 7 unknowns using the 4 pairs of points we defined. We want to solve the H matrix as seen below:

H is the homography, (x, y) are the points on the image that will be warped and the (x', y') are points that we want to warp (x, y) to. w in this case is the scaling factor. We want to solve for a, b, c, d, e, f, g, h in the below matrix. We want to use multiple points to increase the accuracy of the warp and transformation. We plug in these values into our H matrix and use this matrix with inverse warping to map the coordinates p of the image to its p' in the warped image.

Below are rectified images that so that they are front parallel.

Laptop
Original laptop position
Rectified laptop position
Rectified laptop position
Art
Original art
Rectified art
Exhibit
Original exhibit
Rectified exhibit

Part 2: Mosaic Blending

This portion of the project involved using the warp homography transformation that was found in part one in order to warp images with overlap together to create mosaics. By warping these images with the projective transformation so that they are all on the same plane and then stitching these photos all together we create a photo that is able to show much more than a single picture alone. I picked one of my images to be a photo that remained unwarped and then rectified the other photo to match this first one. The points chosen were that of the same object / features in the overlap between the two pictures. These points were used to create the homography and then inverse warping was applied to create the warped image. I ultimately used two different methods to blend, neither gave perfect results. One resulted in faint overlaps/ misalignments while the other with Laplacian blending gave a noticeable seam.

Doorway
Door picture 1
Door picture 2
Door mosaic v1
Door mosaic v2
Library
Library picture 1
Library picture 2
Library mosaic v1
Library mosaic v2
Swiss mountains
Swiss picture 1
Swiss picture 2
Swiss mosaic v1
Swiss mosaic v2
New York City
NYC picture 1
NYC picture 2
NYC mosaic

Automatic Stiching and Feature Matching

Because manually selecting correspondences is not only very tedious but also very error-prone and sensitive, we implement feature matching in order to automatically find these correspondences to create our mosaics.

Part 1: Harris Corners

We first want to generate points of interests on the image. We use a Harris Corner Detector in order to do so. This will detect corners in an image - points where there is a change in intensity in very direction. I ended up substituting the peak_local_max function in the given harris.py file and using corner_harris instead because this resulted in a faster run time and far less points. Harris corners identified in images below.

NYC
NYC picture 1
NYC picture 2

Part 2: Adaptive Non-Maximal Suppression

We use ANMS in order to narrow down the identified interet points as well as making sure the points that are selected will be evenly distributed across the entire image. For each coordinate (xi, yi) we will calculate a radius for that point where that point is a local maximum with an intensity that is greater than all other points. The ri found is the minimum suppression radius at point xi.

f represents the corner response we get from a particular point which is given to us from the matrix that we obtain from the previous harris function. We set crobust = .9 and we keep only the top 500 points with the largest radii. The results of this can be seen below, where the points are much less and more spread apart.

NYC
NYC picture 1
NYC picture 2

Part 3: Feature Descriptors and Feature Matching

Now that we have our interest points, we want to use them to create feature descriptors. These feature descriptors will describe the point that it's centered around and the immediate neighborhood around it. We first select a 40x40 patch of pixels around the coordinate. Then we downsample the patch to 8x8 by convolving with a Gaussian filter as well as resizing/ subsampling. This will create a patch that has preserved only the low frequencies. We will also normalize the patches so that the features become invariant to affine changes such as bias or gain in intensity.

Feature
Feature
Feature
Feature

Once we've generated all of these feature descriptors, we want to match them across two different images. We will compute the squared distance from every feature in the first image to the every one in the second image. We apply the Lowe test afterwards such that if 1-NN / 2-NN is less than a threshold of .3, then the pair will be a match. A feature match will only have one very good match, and if there are multiple close good matches, that it is likely not a match at all.

NYC
NYC picture 1
NYC picture 2

Part 4: Random Sample Consensus

While the above matching is able to generate many good matches, there are some outliers that should not be matched together that have been as can be seen in the NYC pictures. In order to fix this, otherwise our homography will be wrong, we must ensure that all outliers are rejected. We use RANSAC for this purpose. We select four random corresponding coordinates from the matched points and then compute a homography based on these four points. After, we will compute a (x'', y'') = H(x, y) on all other points (x, y) where the corresponding point is (x', y'). If the (x'', y'') is close to (x', y') then the homography is good at mapping these corresponding points as long as it is below some error threshold. We will repeat this process many times, keeping track of the largest set of inliers, or points that can be accurately mapped by the calculated homography. These inliers are the true feature matches. We will find the best H matrix with the most inliers and that homography will be used to warp the images. I set an error threshold of .5 and did 500 iterations of this algorithm. You can see in the below images that the outliers are eliminated via this method.

NYC
NYC picture 1
NYC picture 2

Mosaic Blending

NYC
Manual correspondences
Automatic correspondences
Door
Manual correspondences
Automatic correspondences
Swiss mountains
Manual correspondences
Automatic correspondences

For me, the automatic stitching was much better as I was not particularly good at rigorously selecting points when using manual stitching and thus the alignments / warps of the images were not very precise as well. Overall automatic stiching results were much better.

Summary

It was interesting to see how everything worked in order to arrive at the automatic mosaicing. There were a number of different steps, but it was cool to see how the pieces all fit together to eventually results in the final mosaic. All of the algorithms were pretty complex, but it was satisfying to finally understand them all at the end of the project.