Overview

In this project, I took pictures of a scene with two different perspectives, and I stitched them together to create a mosaic. To do this, I picked key points in each of the pictures and found a homography between each of those pictures. After creating this homography. I found the shape that one of the pictures was warpping to, and I inverse warped each one of those pixels to get the pixel value of the warped image with bilinear interpolation. Finally, I stitched the images together by finding where they overlap and taking a weighted average of the pixel values.

Shoot the Pictures

I took pictures of a couple of scenes: one in front of my apartment, one in my friend's house, and one on campus. The mosaic with the image on campus will have three pictures stitched together. I took these pictures by rotating around my camera, so the origin point of the camera will be the same.

Recoving Homographies

To get the homography, we needed to create a least squares problem that solves for the eight unknowns in the homography matrix. I will define a homography matrix as so:

We have points from one picture (x_0, y_0) and points from another picture (x_1, y_1), and we can generate this equation with the homography matrix.

We see that these equations get us:

To solve this for x_1 and y_1, I can plug in what we got for z.

Finally, we can generalize this in matrix form for all of the points that we chose. The vectors look something like this:

We can create a matrix multiplication Ax = b equation, where b is filled with the feature points of one of the image, and the corresponding vector for each image shown above are stacked together to create the A matrix. The x vector is the vector of unknown elements, and then we use least squares to find the best values for our unknowns.

Image Rectification

For image rectification, I chose the four corners of the part of the image I want to make front-facing. After that, I manually chose points to be where the corners should be. I calculated the homography between the four corners I picked on the image and the four points I manually created. I then warped my image to line up with the manually chosen corner points so that the image faces forward.

For the apple juice tag, because this image was not a completely planar surface, the warp for the bottle looks strange, but my goal was to rectify the slanted apple juice tag, so I cropped that part out by itself.

Because the Bunz sign is on a flat surface, the output became better. The black section are parts that we cannot fill in because we don't have that part of the building in our picture.

Image Mosaicing

To do image mosaicing, I found the homography matrix and found the shape of the warped image by first warping only the corners. Some of the warped indices may have negative index values because they go past the original image size. From there, I found all of the points within that warped image shape using the polygon method, but they only use positive indices, so I first need to shift the warped corner points so that they are all positive. After getting all of the pixels indices that are in the warped image shape, I shifted them back to do inverse warping to find the corresponding pixel values. Finally, I shifted both images down a specified amount so they would all have positive index values. To stitch the images together, I found where the images intersected and took the weighted average the pixel values in that area.

For stitching three images, I stitched together one image at a time. I made sure to shift/transform the original image points to where they are at after the previous stiches to make it easier for the next picture to align with it.

The first pictures are taken outside of my apartment of the other side of the street. The pictures line up pretty well for the most part and there is a small difference in color near where the pictures align.

This mosaic is of my friend's apartment. The darker parts along where the edges align are more prominent. This is because each picture had only one of the two light sources, so the other side of the picture was darker, but in the mosaic, both light sources can be seen.

This mosaic is the view from the Campanile with three images stitched

Coolest Thing?

I think the coolest thing was leveraging least squares to find the homography matrix to change perspectives. I think being able to approximate that matrix with just picking pairs of corresponding points between two pictures was really cool to implement and see work well with my mosaics.

Overview

In this project, I automatically found matching points by first looking for Harris corners in both pictures that needed to be stitched together. I then filtered these corner points by setting a threshold for the corner strength, then using adaptive non-maximal suppression, and finally we matched features by using the Lowe's trick and seeing if points had matching nearest neighbors and found the best homography through RANSAC.

Images

We will be using the same pictures as the previous section. I took pictures of a couple of scenes: one in front of my apartment, one in my friend's house, and one on campus. The mosaic with the image on campus will have three pictures stitched together.

Harris Corner Points

I used the given code to solve for the Harris corner points. To decrease the number of interest points, I filtered out the points with a corner strength threshold, like the paper did. The threshold I used was 0.5.

Adaptive Non-Maximal Supression

To implement ANMS, I found the radius value for each point (x) by checking all other points (y) and filtered by the corner strength where f(x) < 0.9 f(y), and found the minimum squared distance for each of them. Finally, I sorted by the minimum squared distances, and took the top 500 values.

Feature Descriptor

To match features, I took a 40x40 patch around each point returned by ANMS, and averaged together every 5x5 chunk to end up with an 8x8 feature patch. I then normalized it to be invariant to intensity. Some of the images shown below are clearly corners and others look a little more like edges.

Feature Matching

To do feature matching, we first find each point's nearest neighbor and second nearest neighbor from one picture to the other. The first thing I checked was whether or not two points had each other as the nearest neighbor (symmetry). I then used the Lowe's trick to find the by dividing the error of the first nearest neighbor with the second nearest neighbor. I set the threshold for this quotient to be 0.6.

RANSAC

After getting the pairings from Lowe's and symmetry, we use RANSAC to find the best homography for these given points, and this will allow us to remove any outliers we may have left. I randomly chose 4 pairs from the pool of pairs we had left and make those the four points to create the homography matrix from. I then tried transforming points from one to the other, and they are considered inliers if their distance error is less than 2. I did this for 1000 iterations and found the inliers for all of the homography matrices, and I kept the one with the most inliers. This is now my final group of feature point pairs and the final homography matrix will be made with these points.

Mosaics

Finally, I used the final group of feature point pairs to create a homography matrix, and I stitched the images together the same way I did in project 4A.

The automatic stitching is definitely a little different than the manual stitching, as we can see by the different angle of stitching and different overall size of the mosaic. However, the quality of the stitching is about the same as manually choosing points, so I think this process did very well. I found that for my campus picture, there were many points on the roof of the building but not as many along the grass and sidewalk, so the alignment may not be as good for the lower half.

Coolest Thing

I think the coolest thing was cutting down the number of feature points through symmetry and the Lowe's trick. It is very simple to think of the nearest neighbors, but finding the different between second nearest neighbors makes the points chosen more certain to be correct pairs. I think it's really interesting how simple the concept is but how much it does when cutting down outliers.