Image Warping and Mosaicing: CS194-26 Fall 2018 Project 6A - First part of a larger project

Nikhil Uday Shinde (cs194-26-aea)



Project Outline

The purpose of this project was to explore the applications of applying different transforms to images. We focused on the specific application of stitching images together in a panorama/mosaic fashion. To do this we took images and found homographies to project them onto some plane. A process that can also be used for image rectification, this allowed us to project images onto the same space and use blending tactics like linear and laplacian blending to blend images together to create a lovely panorama or mosaic.


Shoot the Pictures

All the pictures displayed in this project were shot using a Google Pixel XL smartphone by me: Nikhil Uday Shinde other than the image labeled paper with scissors which was taken from a google search. For mosaics the camera was held still in hand and rotated about a point with significant overlap between the scenes captured. This allowed for the images to appear as though they were taken from approximately the same point with the difference being accounted from only the angle that the camera was facing. By photographing the images like this we see better results for the homographies that are used for the projections to generate the panorama.
Fun fact: All mosaic images were shot on a weekend trip to climb Mount Whitney the tallest mountain in the contiguous United States/Lower 48!

Recover Homographies

To start the process of creating the panorama we find transform that can warp the image. This allows us to rectify or warp one image to another making it easier for us to blend images together into a panorama as they are transformed onto the same plane. In order to accomplish this we solve for the following transformation or homography matrix (image credited to cs194-26 slides:

Homography matrix


In order to do this we have to select the initial points on the image we want to transform and either select or specify corresponding points for them to transform to. Once this is done we use these points to construct the following matrix:

Homography Least squares matrix


We use this matrix along with the added constraint that the last coefficient in the homography matrix is one to do least squares to find our actual homography. Here (x,y) are the points of the image we are transforming from and (X,Y) are the points of the image we are transforming to.

Warp Images

Now that we have found the homography matrix we can warp our image. To do this we adopt the inverse warp method we utilized in project 4. We first use the forward homography on the corners of the image to establish bounds of the target image. Then we use the inverse transform on each pixel in the target image to find which pixel value of the original image it should correspond to. We can then use an interpolation function to fill out these pixel values and construct the warped image. In fact all these computations can be done using a matrix multiplication and some vectorization!

Rectified Images

We can use the warp defined above to transform images such as they appear to be viewed from a different perspective. The following images were taken looking at objects on/near pieces of paper from a side angle view. By choosing the appropriate points in the initial image and appropriate corresponding points for them to map to we can compute a transform that generates a different perspective on the object, or a rectified view.

Rectified Images
Original image
Rectified image with a top down view
Original image
Rectified image with a top down view

Blend the images into a mosaic

Now that we know how to warp the images we can create a mosaic! First we start with 2 images: 1 of which we warp to the plane of the other. To do this we select correspondence points between the two images and use these to generate a homography that rectifies one image to the plane of the other. We then compute the size of the target image for the panorama/mosaic and place the warped and unwarped image on their own copies of a blank background of the target image size. Since we have warped one image to the plane of the other allignment is simply a translation that we can compute using information that we found while computing our transform. We then do the correct translations to allign the images properly and use blending to properly blend the overlap.
For the blending I tried two techniques: linear or alpha blending and laplacian blending. For linear blending in the overlapping region I weighted pixels from the two images with an alpha and 1-alpha respectively. These were determined by the distance of the pixel from the central point of the two images and the amount that a pixel from an image contributed was inversely proportional to its distance from it's image's center. Though this worked in a lot of images there were some visual artifacts that were apparent as seen below.

Mosaic/Panorama blended with linear or alpha blending
Original
Original
Mosaic
Original
Original
Mosaic

Blend the images into a mosaic

Since linearing blending was giving a lot of artifacting I switched to laplacian blending from project 3 in order to blend the two images together which gave a significantly less noticeable seam. Here is an example of laplacian blending on the same images shown above with linear blending:

Mosaic/Panorama of images shown above with laplacian blending
Original
Original

More Panoramas

To compute panoramas with multiple images we do it in steps. We first stitch two images together and then use the result and stitch another image on that using the procedure described above. Here are some more examples of panoramas made:

View of 99 Switchbacks from Trail Crest up to Mount Whitney
Original
Original
Original

Two images blended

Three images blended

Guitar Lake
Original
Original
Original

Two images blended

Three images blended

View of Whitney from permit pickup point
Original
Original
Original

Two images blended

Three images blended

View of Trail Camp from 99 switchbacks at sunrise
Original
Original

Two images blended

PART B: Outline

In the first part of the project all the warping was done using points chosen manually by the user, however this can get tedious and the user input may not always be the most reliable. As a result in this section of the project we try to automate the process of choosing correspondences between images to get better mosaicing results.


1: Harris Corner Detection

In our journey to automate correspondence selection we start by using Harris corner detection to choose the relevant points in the images. This detector selects points that correspond to corners in the image. Such sharp points can be considered as good objects to lock on and use as correspondences between two images. However as can be seen in the image below there may be many results that correspond to corners of different strengths.

Unfiltered result of running Harris corner detection on image with min_distance in harris corner function set to 10
Trailcrest mt. whitney

2: ANMS

When we filter the Harris corner results we do not want to simply take the strongest corner points as they may be localized in parts of the image. Rather we want correspondences that are distributed through the image to get good results. In order to accomplish this we use an algorithm called ANMS: Adaptive non-maximal supression. In this algorithm we rank points by the minimum distance of the closest corner point to them that has a harris corner greater than them (off by a scaling factor c_robust which I chose to be 0.9). This ensures that the stronger points are selected as they have a big radius but also spreads out the points giving us a good set of filtered correspondences.

Image shown above with ANMS
Trailcrest mt. whitney

3.1: Feature Extraction

For feature extraction we assumed that there was no rotation. We chose a 40 by 40 window around each feature returned by ANMS. These were then resized to 8 by 8 blocks. Finally the features were demeaned and their standard deviation was normalized to one.


3.2: Feature Matching

For feature matching we took the SSD of each feature we found in image 1 against every feature we found in image 2. In order to prevent a feature in one image from matching to multiple features in the other image we take nn1, nn2: the 2 most closely matching features (first 2 nearest neighbors) found in image 2. If (ssd of nn1)/(ssd of nn2) is not below a certain threshold this means that the 2 nearest neighbors are actually quite similar in terms of the metric meaning that there is a chance of the aforementioned problem occuring. Thus if (ssd of nn1)/(ssd of nn2) < threshold we add the feature from image 1 and the corresponding nn1 in image 2 to a list of matched features otherwise we throw away the feature in image 1 and continue.

Above image plus another shown with the output results of feature matching
Trailcrest mt. whitney

4:RANSAC

If you look closely at the corresopndences in the image above there are still some correspondences that don't quite make sense. Though feature matching helps narrow down and pair features up some of these matchings can be incorrect. To prevent these 'outliers' from interfering with our results we use RANSAC. In this algorithm we start by randomly taking 4 of the matched points returned from feature matching. We then use these sets of points to compute a homography. After this we transform all the matched points in image 1 with the calculated homography. If the homography were correct then these should match/be fairly close to the corresponding matched points in image 2. If the transformed point from image 1 is within some tolerance to the corresponding point in image 2 we add it to a list of 'inliers', otherwise we call it an 'outlier'. We do this over several iterations, I chose 2000, and keep track of the largest set of inliers we found over all the homographies that we computed. Finally at the end of all the iterations we use the largest set of inliers to compute another homography which we will then use to warp the adequate images for mosaicing.

Result of mosaicing the above images after RANSAC
Trailcrest mt. whitney

Comparison to manual correspondence selection

In some cases like for the image above there can be a drastic difference if the points are not chosen properly. Here is an example of the above images mosaiced using manual correspondence points:

Above image with manual correspondence points
Trailcrest mt. whitney

More comparisons to manual correspondence selection

With many of the images we see slight differences in the warping if the manual correspondence selection was done well.


Manual
Auto

Manual
Auto

Manual
Auto

Manual
Auto

Manual
Auto

Manual
Auto

Manual
Auto

Summary of what I learned

I learned how to create panoramas! I also learned fundamentals about using linear algebra to transform images to project them onto desired spaces for both visual changes and to help solve certain problems. I also learned more about the nuances that go into selecting good points for a homography and how improperly choosing points or choosing a homography that warps the image too much can lead to some visually undesirable results. I also learned interesting techniques such as corner detection. I learned how to parse and replicate parts of a paper. I also learned an interesting approach to filtering outliers with random trials.


Website template inspired by: https://inst.eecs.berkeley.edu/~cs194-26/fa17/upload/files/proj1/cs194-26-aab/website/