CS194-26 Project 6

Shooting the Pictures

Below are some photographs I took that are from the same point of view with overlapping fields of view, but differnt view directions, thus ensuring that the transforms between the two images were projective. I took pictures with the same camera settings between the same images to ensure the images could be stitched properly. The pictures I took were street view pictures from the front of my apartment, as well the front of my apartment.

Recover Homographies

Recovering the homography matrix is similar to finding the affine transformation matrix but there is one more degree of freedom (8 total). As a result, I used a procedure similar to the equation listed beow where (X, Y) represent points in the original image while (X', Y') represent points in the new point. I also included a last row to the matrix of [0, 0, 0, 0, 0, 0, 0, 0, 1] and included a one at the very bottom of the column vector on the right side of the equation.

Using the procedure shown above, but using multiple points rather than just the one shown in the image above, I am able to retrieve the 3x3 matrix. Finally, to use the homography matrix, one should right multiply the homography matrix with a column vector consisting of (X, Y, 1). The product will be of form (W * X', W * Y', W) which you can just divide out the W factor to retrieve the transformed points.

Warp the images

To warp the images, I simply computed the homography matrix between the two images by selecting points of correspondences on both the images that correspond to the same object or the same shape of the object I'm attempting to warp into. Afterwards, I just iterated over all the pixels in the new image and then found the corresponding pixel in the original image using the homography matrix, and then copied over the original image's pixel value over to the new image.

Rectified Image

Below are some examples of rectified images that I created using the methods I spoke above. I chose the correspondence points to be 4 points on my original image to be the four corners of a square within my original image but isn't an actual square in the image, and the other 4 points to be [(0, 0), (1, 0), (0, 1), (1, 1)] which represent a square. I then found the homography matrix and then warped the image. Unfortunately, I didn't have enough time to sort out the finer details that come with coding rectifying an image.

Blend the images into a mosaic

Creating a mosaic is very similar to rectified images, except you rectify the images to each other rather than an imaginary square. I chose correspondence points that correspond to the same objects within each of the images (but that obviously have differnt points because they're from different perspectives). After rectifying several images to a single base image, you would just do a simple average blending to stitch the images together to create a mosaic.

Tell us what you've learned

I learned again the power of data! Through having taken pictures from the same point of view with overlapping fields of view but different view directions, I was able to create interesting images such as mosaics and rectified images. I also learned that there is a lot of power in having a sort of point of reference (points of correspondence in our case) to use to manipulate images.

Detecting Corner Features in an Image

Below are the base images that I used for this part of the project that were resized down here so they could fit into my website folder (mostly just pictures of inside my house because clean air is a valuable resource).

I used the Harris corner detector which finds corners and features within an image by looking at regions that are effected the most when small changes are applied to it.

Adaptive Non-Maximal Suppression

In order to reduce the number of points of interests that we look at while keeping them well spread out throughout the image, we use a method called Adaptive Non-Maximal Suppression which allows us to acheive this good spread of interest points.

Extracting Feature Descriptors

In order to extract feature descriptors for each of our remaining points, I sampled a 40x40 window around each of the points and then downsampled them to a 8x8 box to make computation a bit faster.

Matching Feature Descriptors

Now that we have feature descriptors for each point, I now found the first and second nearest neighbor (ranked by distance between feature descriptors) and found the ratio between the two. If the ratio was under a certain threshhold, I would consider the original point a match and discard it otherwise.

Using RANSAC to compute a Homography

Although I found a good set of features to be used for transformation, not all of them match up correctly and in order to find the points that do not match, or the outliers, I used a process called RANSAC. RANSAC basically picks a random subset of 4 features and computes the homography matrix. Then I calculate the the errors given the features and the computed homography matrix and keep the features whose errors fall below a threshhold, or the inliers. I keep sampling and keep the largest set of inliers to be used for my actually homography.

Once you've found the inliers, you can use these points to compute the homography matrix and then transform your points accordingly.

Creating a Mosaic!

To be honest, I didn't complete project 6A and project 3 completely so I couldn't really make a mosaic but enjoy the mosaics that I would've created if I actually coded this section!

Tell us what you've learned

I learned the power of homographies and automatic corner detection. A big problem for a lot of the projects that I faced in this class was choosing good, robust correspondence points and now having a process that does it for me is very helpful. I also learned that, through RANSAc, random sampling, with enough iterations, will give you a really good result.