IMAGE WARPING
and
MOSAICING

Image Rectification

Image rectification is the tranformation of an image into a new perspective. To derive such a transformation solely from one image, you need to assume something about the geometry of the world. The easiest assumption to make is to locate features of the image that are supposed to be squares or rectangles when viewed from a different perspective.
 
Original Rectified
Original Rectified
Original Rectified

Panorama

Next we will attempt to create a panorama created from multiple photos of the same scene. In order to do this, we need to select features that are common across images. The best features to select are corners of shapes in the scene, so that is what was used.
 
Using a transform with 9 degrees of freedom, we can transform one of the images in scale, translation, and perspective in order to align them. I solve a least squares problem given points from the source image and their final positions in the target image. 
 
Though we can align the features, the colors of the images could be markedly off even when taken at the same time and place. The need for some kind of smoothing of color across the panorama is needed to achieve a seamless stitch. I tried two ways of achieving this:
 
1) Linear Weighting: For every mapped pixel of image 1 that overlapped with a pixel from image 2, I calculated the euclidean distance from the pixel to the center of each image. Then, I weighted the color value of the corresponding pixel from either image and added the two together. The weight was determined by dividing the distance from the corresponding image center by the addition of both distances from the center. This effectively adds more of an image's pixel to the final pixel value if the index is closer to that image, making for a more convincing transition. 
 
2) 'Quadratic' Weighting (Bells and Whistles): I wasn't satisfied with the Linear Weighting results, so I tried a differet weighting scheme. This method is the same as the previous one, except I used the squares of absolute euclidean norms. This would weight pixel values near a certain image even more toward that image. I think the results were on average better than the Linear method.
 
Below are some examples of my panorama program:
 

 Living Room (Berkeley)

Original 1 Original 2
Mosaic (No Blending)
Mosaic (Linear Blending)
Mosaic (Quad Blending)

Family's Kitchen (San Jose)

Original 1 Original 2
Mosaic (No Blending)
div class="approach animate-box">
Mosaic (Linear Blending)
div class="approach animate-box">
Mosaic (Quad Blending)

Roof of The Met (Berkeley)

Original 1 Original 2
Mosaic (No Blending)

div class="approach animate-box">

Mosaic (Linear Blending)

div class="approach animate-box">

Mosaic (Quad Blending)


Part 2: Automatic Panorama

For this part, I attempted to create a panorama automatically with help from a provided harris feature detector function. This function returns a huge number of potential points - we need to choose a smaller subset of these. As you can see there are way too many points to compute over from the harris function:
 
 
 
At first I tried a random subset of points:
 
 
 
As you can see, the random subset picks points that arn't necessarily distributed evenly since the the behavior of the harris detector usually concentrates points around certain features. The picked points may also not be prominent features of the image. We need some way of picking a subset of points optimized over even distribution and corner strength. Enter Automatic Non-maximal Suppression.
 
ANMS works in the following way: Find distances between every point and all the other points, then for each point, find the minimum distance from another point that has a corner strength similar to its corner strength. Pick a subset of the points with the top distances. This algorithm helps to keep the point distribution somewhat even while keeping strong points:
 
 
 
Now that we have candidate features from each image, we need to match similar points in the scene. We do this by creating a feature descriptor, notably a downsampled patch around each point. To compare the features and screen out points that don't have a match in the other image, we use the Lowe thresholding technique, which computes a ratio of the one-nearest neighbor over two-nearest neighbor. We then remove points which have a ratio larger than some threshold. The resulting features are displayed below:
 
 
 
 
Examining the images, there are still some points in one image that don't have a corresponding point in the other image. This will mess with the homography calculation, so we must screen for outliers. Using RANSAC, we can pick a random subset of 4 points, calculate a crude homography, then check for inliers. We can repeat this for many iterations and find the largest set of inliers, and then use only those points to calculate the homography. The points after ransac are shown below:
 
 

 

After RANSAC, all that is left to do is apply the homography from part 1 using the inlier points. Here are some comparisons of the manual warp on the left vs automatic warp on the right:

 

 


 

Summary:

The coolest thing I've learned is the theory behind framing the panorama problem as a linear transformation.