Continue with the glorious streak of making mosaic panorama image. We are attempting to recreate the effect without manually aligning the pictures. The process of alignment can be found in the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al. The image sources we are going to use will be the images from the previous project down below.
Desk left
Desk right
Shelf left
Shelf right
Room left
Room right
Part 0: Find Harris Interest Points
Luckily for us in this part, we are provided harris.py. After running the algorithm through the image the result is the following.
Desk Left Harris Points
Desk Right Harris Points
Part 1: Implement Adaptive Non-Maximal Suppression
Following the paper, we implemented the algorithm in section 3. What this algorithm do is finding adaptive points. This is important because harris points actually gives us so much information, which is too much for us (in my case more than 8,000 points). With this algorithm, we will only extracted out 500 points which is the strongest reponses to adaptive non-maximal suppression.
Adaptive Non-Maximal Desk Left Points
Adaptive Non-Maximal Desk Right Points
Adaptive Non-Maximal Shelf Left Points
Adaptive Non-Maximal Shelf Right Points
Adaptive Non-Maximal Room Left Points
Adaptive Non-Maximal Room Left Points
Part 2: Implement Feature Descriptor extraction
Following the paper, the next step we need to do is to find the cordination (adaptive points that are meaningfully related). In this part we follow section 4 of the paper. A bias/gain normalized sampling of a local patch (8x8) from a higher pyramid level (40x40) is used to create a vector feature descriptor for each interest point.
A Desk Extracted Feature
Computer Edge
All Features in Desk
A Shelf Extracted Feature
Shelf Edge
All Features in Shelf
A Room Extracted Feature
Shelf Corner
All Features in Room
Part 3: Matching
Following the paper, the next step we need to do is to find the cordination (adaptive points that are meaningfully related). In this part we follow section 5 of the paper. The similarity between two interest points is computed by taking the SSD between their two descriptor vectors.
Desk left matching cordinates
Desk right matching cordinates
Shelf left matching cordinates
Shelf right matching cordinates
Room left matching cordinates
Room right matching cordinates
Part 4: RANSAC
There are many way to find homography matrix in the last project we have seen how it is done with eight points. RANSAC described in the lecture only need four points, which we random sampled from the adaptive points we have.
Part 5: Warp and Do Mosaic
In this part I just use the code from the last project to warp the image and fit it into one another to create mosaics.
Desk Mosaic
Shelf Mosaic
Room Mosaic
Part 6: What I Have Learned
I have learned a lot in this project actually. In the last project while making the moaic panorama is awesome and all, it is extremly impractical due to the fact that we need to manually select points that corespond to each other to create the image. I learn in this project how to extract features (edge) in the image without using CNN. I learn how to determine which is meaningful and which is not. I also learn how to compare them and match them up to one another. Lastly, I learn how to use RANSAC which is a very cool algorithm to find homography.
