International student participating in the Brazilian Scientific Mobility Program (BSMP) at University of California, Berkeley, for the 2015-2016 academic year. Enrolled as a Computer Science Extension student. Born and raised in Rio de Janeiro, Brazil. Undergrad student at Pontifical Catholic University of Rio de Janeiro, studying Computer Engineering and Mathematics. Traditional and Digital art lover and currently interested in Computer Graphics and Artificial Intelligence - but who knows what new field of study I might fall in love with! Trying to learn the most I can during this one year here in Berkeley.
The goal of this project is to create a system for automatically stitching images into a mosaic. A secondary goal is to learn how to read and implement a research paper (Multi-Image Matching using Multi-Scale Oriented Patches). The project will consist of the following steps:
The first part of the assignment is to detect the corner features in an image. Luckly, skimage.feature has a method for creating the corner response map of a image. The higher value on this corner response map, the more corner-like it is. We want to find the local maximums on this map. Below, you can see some images, its corner response maps and a few of its local maximums. As you can see, it matches corners.
Below, we can see that many points that correspond to the same feature are local maximums. However, it seems that we need more points of correspondence. Not all local maximums are being chosen.
Here we will adjust the minimum intensity of the peaks as relative to the global maximum. By decreasing the minimum intensity, more points are found as local peaks. This increases the number of interest points found. Below, we can see some examples of this parameter being adjusted.
Okay. Now our interest points are concentrated in the same place. If we were to take the highest N points in the last image, probabily they would all be centered near the hair of Lena (because it is full of corners). We need a way to distribute those points along the image. We are going to solve this problem with the Adaptative Non-Maximal Suppression. It defines a minimum radius that each point need to be from each other. The idea is to suppress the interest points that do not match this criteria. Of course, the larger the minimum radius, the less points we can possibly find. The way we do this is by iterating from "infinity" to the minimum radius. Whenever a point is not in any of the regions defined by the points already selected, we add this points to the points selected. Confusing? Yes, I know.
Now we need to extract descriptors for each of the points selected. This is actually pretty simple. Once we have the points, all we need to do is extract a black and white patch around it. Here, we will choose 8x8 patches, as described in the paper. A mosaic with the descriptors for the image chosen is displayed below:
Once we have our descriptors of one image, we may start matching them with the descriptors of the other image. We do that by comparing each descriptor. For the matching criteria, we use the Russian Grandma wise advices for choosing a husband: "If you are not sure which of the 2 husbands you want to pick, maybe you should not pick neither." The way we measure this is: we calculate the best and the second best SSD for each descriptor. If the ratio between them is lower than a threshold, then it means that the best is much better than the second best. If that happens, it is a match.
As you can see, we still get a lot of incorrect matches. However, the good matches are still there. We need a way to separate those false alarms from the real points we want. We will do that with RANSAC.
Random Sample Conseunsus (RANSAC) is the way wer eare choosing to get rid of the false matches found. The idea is: we sample points from our pool. We assume they are right. Then, we calculate the error from every other point to the points chosen. If they meet a certain threshold, they agree. If not, they disagree. By doing this several times, we can choose the largest set of points that agree with each other. Those are our correct points. We can see the result below:
After getting rid of tons of points, it works! The results compared to the manual point selection ones are displayed below.
Sometimes the algorithm fails. Below, we can a example where even a bad manual panorama as better than the automatic one. Improvements on the algorithms needs to be made so it does not get interest points near the black areas of the deformed images. Also, some tweaking on the parameters are necessary.
I learned that there are some pretty simple algorithms for matching features that works really well. I loved this first step on how to detect objects on a scene. Also, no more clicking for panoramas - Yay!
Website built using bootstrap.