I shot the pictures in my friend's room, because he has a lot of cool posters and a nice setup:
For recovering homographies, I went through the entire Piazza thread and found this article: https://towardsdatascience.com/estimating-a-homography-matrix-522c70ec4b2c. I needed to find H from p'=Hp. I found this description of the matrix to be particularly helpful:
I got corresponding points of each image using ginput inside my get_points1 function that I ran in terminal. I selected the points on the image and then copy and pasted the point values into jupyter. I wrote my computeH function and then used it to find the relation between the each set of points in my 2 images.
I used inverse warping with an inverse H matrix. I initialized all the pixels in the resulting image to 0, so that the non-relevant areas of the image would be black, essentially using a black alpha mask. Then I iterated through each pixel and used the inverse H matrix to set the respective pixel on the resulting image to the warped result.
To rectify these following examples, I first selected 4 points on each of the original images, and then chose 4 new points on each image that would make each of them a perfect rectangle. I used my get_points1 function to define the corresponding points on the original image (called get_points1 in terminal to select points there, then copy and pasted the values into jupyter), and then created an array for the reference points by hand. I basically eyeballed the image axes and chose which coordinate values would make a good-sized rectangle, sort of surrounding/over the original image. I then passed in the image points and the reference points to my computeH function, and used the resulting H and the original image as parameters for warpImage. For each of the following images, you can easily tell that it has shifted from a side view to a front view.
For the mosaic blending, I had to slightly modify my warpImage function to suit the size of the new canvas. Instead of doing a for loop through just the dimensions of the image, I had to use corners to calculate min and max values for x and y and use that to place both images on the newly defined canvas with a new, larger size. Then, I created an alpha mask for the new canvas using np.zeros, and iterated through the pixels, mapping the pixels from warped image 1 and regular image 2 to the new canvas. This created the stitched effect, and added my photos together nicely. Before creating the mosaic, I also had to use my get_points function here to get corresponding points on both images. I ran that in terminal and copy and pasted the points into jupyter. Here are my few examples:
The biggest thing I learned from this project was how matrix multiplication and what I learned in Math 54 can actually be used to find the relationship between various sets of points between two images, and how useful this can actually be. The concept of warping was fascinating to me and it was really cool to see how I could actually warp an image so that the objects of the image were in the same orientation as the reference image. I never knew that matrices could be so useful, even though they appear everywhere——I had never had as hands-on of an experience with the usefulness of linear algebra until now.
I ran the harris.py code to get Harris corners on my two original images of the room. I set the threshold_rel to 0.07 and kept min_distance=1, with padding of 20 pixels on the edges, as this seemed like a good number for preventing way too many points, but still having more than enough. It was okay to have too many points for this step because in the next step I would implement ANMS.
I then implemented Adaptive Non-Maximal Suppression after re-reading the article and Piazza 50,000 times. I started out by ordering the corners' coordinates in descending order of Harris corner strength. I added the point with the largest Harris corner strength as an interest point. Then, I iterated through these sorted corners and only added a point as an interest point if it was outside (distance-wise) the chosen radius. I decremented the radius by a tiny bit each time I iterated through the outside for loop. At the end of the function, if my interest points array length was the same as my num_points parameter, I returned the interest points array. Here they are, overlaid on my images.
It's kind of hard to see the points being more evenly distributed on these two pictures, so I did the whole process on another photo that is less high-contrast (the room has a lot of posters against white walls). This photo is the one I took from the balcony.
Then, I extracted feature descriptors by starting with a 40x40 patch around each Harris Interest Point, blurring that using a Gaussian filter, and then downsampling that to an 8x8 patch. I ended up with 250 8x8 patches, one for each one of my Harris Interest Points. Here are a few from different points. You can clearly see the difference between them.
I matched features using Lowe's trick covered in lecture. I found the smallest SDD match for each point, and then the second smallest SDD match for each point, and found the ratio of 1-nn/2-nn (for each point). I then compiled these ratios in an array. I also kept track of the indices using np.argsort, so I could go back and extract the actual points belonging to the ratios. I ended up with 23 points using a threshold of ratio < 0.37.
I followed the Piazza thread to write the RANSAC function and iterated through, selecting a random set of 4 points/correspondences each time, and calculating the error from the H computed from those points. I also kept track of the maximum number of inliers and what set of points and H this corresponded to, so at the end of the iterations, I returned the H with the best inliers. I ended up choosing 43,000 iterations.
I used my mosaic and warpImage code from part 4A to warp my images and stitch them into a mosaic, based on the H that was auto-generated through Harris corners, feature descriptor extraction, feature matching, and then RANSAC. It worked well! Below, I have (in order from left to right): original images, manually made mosaic from part A, and then the autostitched mosaic from part B.
As we can see, the autostitched mosaic looks much better! Success :)
My favourite thing that I learned from this project was really understanding how things like this can be done automatically, by minimizing error/distance between points and potential matches. It's really awesome how we can create something as cool as this just using math and selection of best points, and really shows how important error functions are!