Student Information

Ian Albuquerque Raymundo da Silva
ian.albuquerque [at] berkeley.edu

International student participating in the Brazilian Scientific Mobility Program (BSMP) at University of California, Berkeley, for the 2015-2016 academic year. Enrolled as a Computer Science Extension student. Born and raised in Rio de Janeiro, Brazil. Undergrad student at Pontifical Catholic University of Rio de Janeiro, studying Computer Engineering and Mathematics. Traditional and Digital art lover and currently interested in Computer Graphics and Artificial Intelligence - but who knows what new field of study I might fall in love with! Trying to learn the most I can during this one year here in Berkeley.

This is the second part of a larger Project!

Click here for Programming Project #7 - Part A - Image Warping and Mosaicing - Homographies

The Assignment: (as extracted from the assignment's website)

The goal of this project is to create a system for automatically stitching images into a mosaic. A secondary goal is to learn how to read and implement a research paper (Multi-Image Matching using Multi-Scale Oriented Patches). The project will consist of the following steps:

A paper about this process can be found in http://inst.eecs.berkeley.edu/~cs194-26/fa15/Papers/MOPS.pdf

Detecting the Corners Response:

The first part of the assignment is to detect the corner features in an image. Luckly, skimage.feature has a method for creating the corner response map of a image. The higher value on this corner response map, the more corner-like it is. We want to find the local maximums on this map. Below, you can see some images, its corner response maps and a few of its local maximums. As you can see, it matches corners.

Below, we can see that many points that correspond to the same feature are local maximums. However, it seems that we need more points of correspondence. Not all local maximums are being chosen.

Adjusting the Corners Local Maximum Detection:

Here we will adjust the minimum intensity of the peaks as relative to the global maximum. By decreasing the minimum intensity, more points are found as local peaks. This increases the number of interest points found. Below, we can see some examples of this parameter being adjusted.

Lena Original Image and Corners Response Map:

Relative Threshold = 0.1, 0.05 and 0.01: (348, 655 and 1755 points found)

Dispersing Crowds: (Adaptive Non-Maximal Suppression)

Okay. Now our interest points are concentrated in the same place. If we were to take the highest N points in the last image, probabily they would all be centered near the hair of Lena (because it is full of corners). We need a way to distribute those points along the image. We are going to solve this problem with the Adaptative Non-Maximal Suppression. It defines a minimum radius that each point need to be from each other. The idea is to suppress the interest points that do not match this criteria. Of course, the larger the minimum radius, the less points we can possibly find. The way we do this is by iterating from "infinity" to the minimum radius. Whenever a point is not in any of the regions defined by the points already selected, we add this points to the points selected. Confusing? Yes, I know.

Lena Original Image and Corners Response Map:

Minimum Radius = 10, 20 and 30: (460, 182 and 100 points found)

Extracting Descriptors:

Now we need to extract descriptors for each of the points selected. This is actually pretty simple. Once we have the points, all we need to do is extract a black and white patch around it. Here, we will choose 8x8 patches, as described in the paper. A mosaic with the descriptors for the image chosen is displayed below:

Original Image and Corners Response Map:

Interest Points Found and Descriptors: (Minimum Radius = 2, Number of Points = 50, Descriptor Size = 8x8)

The Dating Game (Feature Matching): The Russian Grandma Criteria

Once we have our descriptors of one image, we may start matching them with the descriptors of the other image. We do that by comparing each descriptor. For the matching criteria, we use the Russian Grandma wise advices for choosing a husband: "If you are not sure which of the 2 husbands you want to pick, maybe you should not pick neither." The way we measure this is: we calculate the best and the second best SSD for each descriptor. If the ratio between them is lower than a threshold, then it means that the best is much better than the second best. If that happens, it is a match.

The Russian Grandma and The Descriptors Dating Game:

Original Images:

Interest Points Found and Descriptors: (Minimum Radius = 10, Number of Points = 300, Descriptor Size = 8x8)

Matching Points: (Grandma Threshold = 0.1)

Matching Points Zoom: (Grandma Threshold = 0.1)

As you can see, we still get a lot of incorrect matches. However, the good matches are still there. We need a way to separate those false alarms from the real points we want. We will do that with RANSAC.

D.E.M.O.C.R.A.C.Y. : (Discarding Existing Match incOnsistenCies with RAnsaC - Yay!)

Random Sample Conseunsus (RANSAC) is the way wer eare choosing to get rid of the false matches found. The idea is: we sample points from our pool. We assume they are right. Then, we calculate the error from every other point to the points chosen. If they meet a certain threshold, they agree. If not, they disagree. By doing this several times, we can choose the largest set of points that agree with each other. Those are our correct points. We can see the result below:

Matching Points:

After RANSAC: (1000 iterations, RANSAC Threshold = 0.1)

After RANSAC Zoom: (Yep, we got the right points!)

Results:

After getting rid of tons of points, it works! The results compared to the manual point selection ones are displayed below.

Automatic Panorama:

Manual Panorama: (Almost no difference!)

More Examples: I-House:

Originals and Interest Points:

Matches:

After RANSAC:

Automatic Panorama:

Manual Panorama:

With Cylindrical Projection:

Originals and Interest Points:

Descriptors:

Matches:

After RANSAC:

Automatic Result:

Manual Result:

One More Image for the Panorama:

Descriptors:

Matches:

Automatic Result:

Manual Result:

Failures: (Stadium)

Sometimes the algorithm fails. Below, we can a example where even a bad manual panorama as better than the automatic one. Improvements on the algorithms needs to be made so it does not get interest points near the black areas of the deformed images. Also, some tweaking on the parameters are necessary.

Automatic:

Manual (Still OK Here):

Automatic:

BAMMM! WRONG!:

Manual: (Still bad, but better...)

Better with the Automatic Planar Projections:

What I Learned:

I learned that there are some pretty simple algorithms for matching features that works really well. I loved this first step on how to detect objects on a scene. Also, no more clicking for panoramas - Yay!

Last, But Not Least:

Special thanks to Alexei Alyosha Efros, Rachel Albert and Weilun Sun for help during lectures, office hours and questions on Piazza.

Website built using bootstrap.