CS 194-26: IMAGE WARPING and MOSIACING

Amol Pant

Shoot the Pictures

To begin with, I take a couple of photos of some landscape to make a mosiac out of and some closeups of square items at an angle to rectify.

Left Image	Right Image



	This image will be rectified
	This image will be rectified. Note: This image has been taken from: https://www.diynetwork.com/how-to/rooms-and-spaces/floors/laying-a-new-tile-floor

Defining Correspondences and Recover Homographies

I personally chose 8 feature points and by hand defined them. After that I estimated how much padding would be required for the warped images to fit.
Then, using least squares regression, formatted to the equation below, found the Homography matrix for transformation.
Bruh

(Image taken from: https://towardsdatascience.com/estimating-a-homography-matrix-522c70ec4b2c)
Here are the corresponding results.

Image and Points

Warp the Images and Image Rectification

We can now take these sets of points and warp images using our homography matrix. First, estimate the padding needed for the resulting image after the warp and then warp the padded image. Here are some of the results of rectifying some of the images of square objects taken at an angle.

Original Image	Rectified Image

It is a bit hard to estimate the sizing of the warped image since we have to do an estimate warp before we pad the image and do an actual warp, which leads to a circular block.
So, some fo the resulting images are a little cut off at the end.

Blend the images into a mosaic

We now can take our mosiac setup images and then do a similar process. We pad them, find the homography matrix, sequentially warp one image to the next, stich them, and blend them.
For blending, I first created a mask of one image and then used that mask to stitch both images together.
I used some laplacian blending, however, since the depth of the stack is low, we can still see some faint seams at the end. So, sometimes it was just better to not use any blending at all.
This is also a result of the fact that I used my phone camera to take photos and controlling lighting and contrast between photos is very hard to do.

Left Image	Right Image	Resulting Mosiac

Tell us what you've learned

The one thing this project taught me was how important it is to keep track of data of images and transformations.
I had to step back and reorganize my pipeline from the start before even attempting to fix issues.
Also the simplicity of debugging because we can just directly see and interpret intermediate results never gets old and makes any computer vision project really fun.

Part B: FEATURE MATCHING for AUTOSTITCHING

Walking through all the steps for a sample image then showing results on 2 other images as well

Harris Points

We begin by finding Harris corners of an image, in which we find a large set of points and their corresponding Harris "strength". Cutting off some of the edges, here are the harris corners of our sample images.

Room	Room Corners Strength

We see that there is a lot of corner strength around the keyboard, the back drape, and the table corners

Adaptive Non-Maximal Suppression

Next, we perform Adaptive Non-Maximal Suppression (anms).
We can do this by setting a radius minimum, finding the global maximum, and sequentially adding points so that they are the next strongest given that they are the farthest apart from the already existing ones. Here is the result of performing this on our sample image.
Bruh

Now we have a decent set of the best 400 points according to our contraints. They are decently far apart and are also on the edge or corner.

Features and Feature Matching

Features

For each of the suppressed set of points we found above, we create a feature patch set by taking a 40x40 window of the image around the points and then downsampling that to an 8x8 feature for each points.
Here is a feature patch high res and downsampled:

High Res	Downsampled Feature

Feature Matching

Now for each point, we use the corrosponding feature to then find the best set of corrosponding points for 2 images. This is done using nearest neighbors with the sorting function being the SSD (Sum of Squared Differences).
We then apply a threshold for if the set of points should be included in the best set using the ratio between 1st NN and 2nd NN.
This is the result of doing feature matching on the 2 images of our sample set:

Room Left	Room Right

We now have a relatively accurate set of corrosponding points for the 2 images.

RANSAC Homography

We can make the above set of points even more robust by performing RANSAC on them. We repeatedly take a set of 4 points to compute a basis homography. Then we ask all the other points whether they agree with such a homography matrix.
This is done by transforming the points using this homography and seeing if they line up with the corrosponding feature matched point in the other image within a certain threshold (between 1 to 3 pixels off).
We add all the points that agree with the homography to a list.
We then take the list with the largest number of mutually agreeing points to then be our robust set of RANSAC points for the 2 images.
Here is the result of performing RANSAC on the 2 set of points we calculated above:

Room Left	Room Right

We have now sequentially reduced the set of corrosponding points to be used, making our setup fairly robust. Here is the Homography matrix that this set came up with:
[[ 1.74391527e+00 4.78344503e-02 -3.71019962e+02]
[ 1.67720936e-01 1.41634957e+00 -8.92568223e+01]
[ 7.44742930e-04 -3.27679730e-05 1.00000000e+00]]

Auto-Mosiac vs Manual Mosiac

Here is the result of automatic mosiacs when compared to manually setting points:

Auto-Mosiac	Manual Mosiac

Even though there are some errors here and there because I took photos with a phone camera, the results look pretty decent and close.
The room has the most glaring errors since taking a close up photo warps the photo even more and the fact that ransac chose points away from the seam, it means the warp wasnt accounted for.

Learned Lessons

Here are some of the things I learned while doing this project:
1. Computing anms is no joke. Let alone basic conversion of the paper to code, tuning parameters and setting bounds took a lot of time to tune so that points were either not too close or had decent great corner strength. The tradeoff took a while to balance.
2. Keeping track of sets of points takes a lot of thinking before coding.
3. Even though RANSAC is a decent process for making a set of points robust, I feel like I could have made more ways to verify/ error correct by taking the overall result.
4. Lastly, even though this pipeline of reducing and perfecting points is decent, it exposes how much we are just approximating a good solution, and that actually having a good solution would probably require a more meta process of managing this pipeline.
Thanks for reading!