CS 194-26 Project 6 - Image Warping and Mosaicing

Austin Luong

Overview

The goal of this project is to be able to compute homographies between images. Using this, we can project the points of an image to match those those of another image. This lends itself to interesting applications such as image rectification and mosaic / panorama building. In the first part of this project, we manually select correspondences between images to compute the homographies. In the second part of this project, I implement a method to compute the correspondences automatically.

Part 1: Computing and Applying Homographies

Computing Homographies

Let p' represent points in a desired output image and p represent points from an input image (where p and p` are homogenous coordinates). We wish to find a transformation (a homography in this case) H such that p' = Hp. We can set up a linear system of equations to find such an H exactly with just 4 point correspondences between the images (which are chosen manually). However, doing so with just 4 points will make our results very prone to noise and error in the choosing the point correspondences. In actuality, we use many more than 4 point correspondences and use least squares to find an H that minimizes the error between the predicted and desired output. After some math, we find that we need to solve the least squares problem below:

Where h_ij represents the elements of H, (x_i, y_i) represents the ith point in the input and (x'_i, y'_i) represents the correspondences ith point in the output.

Rectifying Images

Calculating homographies allows us to perform image rectification. First we pick points in an input image. Then we set the output points to those that we know those points would correspond to if we were looking at those points head on. For example, if we had an image of a room which had square tiles on the floor, we could rectify the image to the perspective of looking at the tiles top-down. We do this by selecting corners of the one of the tiles as our input points and then picking their correspondences to be a four points that make a square. We calculate the homography from these points and apply it to our input image to get a rectified view of our original image.

Scanning a Document

Sistine Chapel

Creating Mosaics / Panoramas

Homographies also allow us to create image moasics from a set of input images with corresponding points. To do this, we pick point correspondences between pairs of overlapping images. We then apply the homographies to each pair of images to project one image onto another. In my examples, I only ever use three images, so in all cases I mapped a left and right image onto a center image and blending the resulting warped images.

I used two types of blending on the warped images: average blending and linear blending. For average blending, I simple averaged the portions of the warped images that overlapped and kept the rest of the warped images that did not overlap. Linear blending, is essentially the same but with one small modification: instead of simply averaging the overlapping regions, I used mask that gradually increased from 0 to 1 in the overlapping region (essentially a gradient from black to white) to blend the images. Average blending, usually created noticeable artifacts in the panorama while linear blending did not. Lienar blending did have problems with ghosting, however, which could have been solved using a blending technique such Laplacian pyramid blending.

For the results that follow, all images have been cropped to remove any black borders from the image (as seen in the picture at the top of the page).

Outside My Apartment

Average Blending

Linear Blending

Sproul Plaza

Average Blending

Linear Blending

Campanile

Conclusion

This project taught me the power of homographies in be able to achieve a variety of useful and interesting effects. Image rectification, for example, has recently been popular in phone scanning apps.

Part 2: Automatic Correspondence Selection

Automatic Corner / Feature Detection

The first step in automatically selecting images correspoendences is automatic feature detection. The algorithm used for this is the Harris corner detector. The idea behind this algorithm is to detect x and y derivatives for each pixel in the image. These derivatives are then used to compute a corner reponse for each pixel in the image. The corner reponses are then thresholded to get the most corner-like points in the image. Regions of nearby corners are then further collapsed into a single corner point by taking the local maxima of the region.

As can be seen from these results, this generates far too many points so we must further subset the points to use for future calculations. One way to select fewer points naively is to simple take the points with the highest corner responses.

However, this naive approach results in points being clumped around the same region higher corner reponse points tend to congregate around the same area. To prevent this, we can use Adapative Non-Maximal Suppresion. This technique works by calculating a surpression radius for each corner point. The surpression radius is the minimum distance from the current point and every other point that has a sufficiently higher corner response than the current point (how much higher is controlled by a chosen constant factor). The final points are then chosen from the points with the highest surpression radii.

Extracting and Matching Feature Descriptors

Now that the feature points have been obtained for each image, we can generate features descriptors to use to match feature points between the images. To do this, we extact image patches centered around each of the points. To protect against errors to due to excessive detail in the image patches, we select large image patches (40x40 in this case) and scale them down to a smaller resolution (8x8 in this case). To have some invariance to intensity shifts between the images, we normalize each patch by subtracting its mean and dividing by its standard deviation. We then detect the best matches by computing the sum of squared difference (SSD) between each of the patches in one image to each of the patches in the other image. The ratio between the best match and second best match is calculated for each patch and used to threshold which featrues we choose.

Outlier Removal with RANSAC

Even with feature matching, there are still plenty of correspondences between the image that are incorrect. We can eliminate these outliers by using random sample consensus (RANSAC). The first step in this algorithm is to sample 4 correspoendences we have found and use it to compute a homography. This homography is used to transform the feature points from one image to generate predicted output points that correspond to the other image. The distances between the predicted points and observed points are calculated and used to determine the number of inliers for this sample of correspondences. The inliers are determined as points that have prediction errors below some tolerance level. This process is then repeated for a number iterations. We then use all the inliers of the sample with the highest number of inliers to compute final homography.

Automatic Panorama Creation

Using this final homography we can now create panoramas as before but wihout the need to manually select correspondences.

Outside My Apartment