overview:
part 0: final results, and the images i started with
part 1: manually selecting the points
part 2: automagically selecting the points:
2.1: harris
2.2: matching
2.3: ransac
2.4: auto vs manual points, + comments
part 3: stitching the images together
First, I took some pictures of a lobby of my apartment.
Then I went to the roof and took some more pictures.
Then, I used the ginput thing to select the points.
To avoid confusion, all the points will be stored in dictionaries.
So the lobby points will be in the lobby_points dictionary. The keys will be like:
{'1to2': points on image one that are meant to match with image 2,
'2to1': points on image two that are meant to match with image 1, etc.}
Are going to be shown later, side by side with the automatically chosen points.
I used Harris with a min distance of 5 pixels. Then I used the ANMS algorithm from the paper, limiting it to 500 points.
Afterwards, I cut it down again, only keeping 200 points.
These are the points:
I did feature extraction by extracting 40x40 pixel squares then downscaling them to 8x8. No rotation. Then I did the matching, and used a best vs second best ratio of 0.5 as a threshold. These are the points I was left with:
As you can see, the majority of these points are valid. But there are a few bad ones.
I ran RANSAC with 2000 iterations for each pair of images (4 pairs total). After running ransac, these are the largest groups of inliers:
If you look closely, you can see that all the point pairs (which are color-coded) match to the correct locations.
Here are the manually selected points
The auto and manual selected poitns in the roof pictures are similar. However, in the lobby picture, it has a lot of depth relative to how far away i was from the scene.
With that knowledge, when I manually picked points I took care to only pick points that are on on roughly the same plane.
The computer does not know to do this. So as you can see later, the stitching of lobby pictures from the automatic points suffered.
However, the stitching of roof pictures from the automatic points was actually a little better than the manual one.
To illustrate this point, here are the images in question:
The one on the bottom is the automatic one. It is worse.
In this section, I will only include images and explanations from when I did it for the manually selected points, since I used the same code for both the manual and automatic and the images are pretty similar too.
To phrase this into equations:
$$x'w = ax + by + c$$$$y'w = dx + ey + f$$$$w = gx + hy + 1$$However, we need to remove $w$ from this set of equations, since we do not know the value of $w$.
So:
$$w - (gx + hy) = 1$$$$w - 1 = (gx + hy)$$$$x' = x'w - x'(w-1)$$$$x' = x'w - x'(gx + hy)$$$$x' = ax + by + c - gx'x - hx'y$$By the same logic, $$y' = dx + ey + f - gxy'x - hy'y$$
This allows us to use the following system of equations:
$$ \begin{bmatrix} x'_0\\ y'_0\\ x'_1\\ y'_1\\ \vdots\\ x'_n\\ y'_n \end{bmatrix} = \begin{bmatrix} x_0 & y_0 & 1 & 0 & 0 & 0 & -x'_0x_0 & -x'_0y_0\\ 0 & 0 & 0 & x_0 & y_0 & 1 & -y'_0x_0 & -y'_0y_0\\ x_1 & y_1 & 1 & 0 & 0 & 0 & -x'_1x_1 & -x'_1y_1\\ 0 & 0 & 0 & x_1 & y_1 & 1 & -y'_1x_1 & -y'_1y_1\\ &&&&\vdots&&& \\ x_n & y_n & 1 & 0 & 0 & 0 & -x'_nx_n & -x'_ny_n\\ 0 & 0 & 0 & x_n & y_n & 1 & -y'_nx_n & -y'_ny_n \end{bmatrix} \begin{bmatrix} a\\ b\\ c\\ d\\ e\\ f\\ g\\ h \end{bmatrix} $$Note the orange and blue dots on each picture. They are the centroids of their respective sets of points. They represent a point where each pair of images are supposed to align to.
The orange and blue dots serve as anchors for when we position the images against each other.
Using the positions of those dots with respect to the size of each image, I calculated the total size of the mosaic. Then, using those positions again, I pasted the warped image onto the mosaic in the correctly aligned positions.
I used a gaussian filter (sigma = 20) to get the blurry components of each image then subtracted it from the original to get the sharp components.
For the blurry components, I used a linear alpha along the overlaps between each pair of pictures.
For the sharp components, I just picked an x value. Then, I just put one image on the left, and one image on the right, with no averaging.
