CS194-26 Project 4a: Mosaic

Overview

In this project, I used perspective warp to warp images onto the a different plane, thereby rectifying them or stitching images from different angles together. This is done by finding the homography matrix using point correspondances, and warping images onto different spaces.

Images that I took

Here are the images that I took that are going to be used in this assignment.

Left of the bench

Right of the bench

My shelf

My hangers

Left side of courtyard

Right side of courtyard

Left side of shelf

Right side of shelf

Recovering homographies

I first wrote the function for recovering the homography matrix from point correspondences. We have this equation for the matrix:

I selected at least 4 points for each image and then solved H using this setup with the help of the np.linalg least squares function.

Warping Image

I then wrote a function that warps an image according a homography matrix. Specifically, it first uses the four corners to find the warped boundary; and then creates polygon within the boundary; then interpolates the pixel values inside the polygon to create the warped image by multiplying by the inverse of the homography matrix. An example is here where I warped the right side of the bench according to 10 point correspondences to match the space that left side of the bench is in.

Left of the bench

Right of the bench

Warped right side of the bench

Rectifying images

I then tried to rectify some images to frontal-parallel view. They're rectangles or parallel lines, whose are point correspondences are basically chosen based on common sense rather than from another image. Only 4 keypoints were chosen for each image with satisfying results.
The first one is the shelf that's viewed from an angle, and then rectified into front view, an actualt rectangle in the image.

My shelf

My shelf rectified

The second one is my hanger which has supposedly parallel lines but view from the side. In the rectified image, the lines are parallel in the image, like viewed from the front.

Parallel lines on my hangers

Rectified parallel lines

Mosaics

Lastly, I combined what I did above, and stitched images together. I used an weighted average at the points where the images overlap. Each mosaic has 10 keypoints chosen.

Left of the bench

Right of the bench

Warped right of the bench

Stitched bench

Left of the courtyard

Right of the courtyard

Warped right of the courtyard

Stitched courtyard

Left of the shelf

Right of the shelf

Warped right of the shelf

Stitched shelf

What did I learn

The first thing I noticed is that the warping isn't exact --- its edges are pretty clear, and not all pixels are lined up well. This is probably because hand-picked points are optimal because of their low precision. An automated keypoint detection would be key to the success of a mosaic. The warping and stitching is really cool though, because through a simple transformation matrix, we can get view of an item from a different angle using the same image.