Project 4A: Image Warping and Mosaicing

Timothy Kha

1.1: Shoot the Pictures

For this project, I took pictures of objects that could be used for rectification. I took a picture of a text book at a slanted angle so that we could rectify it to see the top image. I took a picture of a walgreen reward card that I would also rectify to view it from a better perspective. I also used an image of an art gallery from the internet.

Images for Rectification:

Chinese Textbook	Walgreens Reward Card
Art Gallery

I also took pictures for my mosaics. I took a lot pictures of outdoor scenes such as our campus. For these images, I took 2+ images of each scene.

Images for Mosaics:

Forrest Left

Forrest Right

Campus Left

Campus Middle

Campus Right

Backyard Left

Backyard Middle 1

Backyard Middle 2

Backyard Right

1.2: Recover Homographies

I first manually selected points by hand for each image (usually 8-10).

Tree Left w/ Labeled Points

Tree Right w/ Labeled Points

Following the least squares approach as described in lecture, we have the equation p' = Hp.

We then can rewrite the coefficients of H as a vector h where h = [a,b,c,d,e,f,g,h].T. We only have 8 unkowns since we set the scale factor = 1.

We can now set up a system of equations in the form Ah = b.

We know that:

wx' = ax + by + c*1
wy' = dx + ey + f*1
w = gx + hy + 1

Substituting w into the first and second equation:

(gx + hy + 1)*x' = ax + by + c
gxx' + hyx' + x' = ax + by + c
x' = ax + by + c - gxx' + hyx'

(gx + hy + 1)*y' = dx + ey + f
gxy' + hyy' + y' = dx + ey + f
y' = dx + ey + f - gxy' - hyy'

Thus we can construct a matrix as such and solve the overdetermined system using least squares (np.linalg.lstsq) and achieve our H matrix:

Source: https://towardsdatascience.com/estimating-a-homography-matrix-522c70ec4b2c

1.3: Warp the Images

Now that we have a Homography matrix, we need to do warping.

For my approach, I chose to warp the keypoints of the left image into the plane of the keypoints of the right image.

First, I created a bounding box by finding the coordinates of the 4 corners of the left image after forward warping (H * corners). Once I had these values, I used meshgrid to make a canvas that would fit the warped image and proceeded to inverse warp. Following the approach in proj 2, I performed inverse warping for the left image. With my meshgrid, I filled in the output image with interpolated values using cv.remap.

The right image is placed correctly by finding the offset between the correspondent keypoints and shifting the right image accordingly to the right location on the canvas.

Left Tree Warped

Right Tree Warped/Shifted

1.4: Image Rectification

With our Homography matrix that we computed from the previous section, all we need to do is warp our image to a new plane of our choosing. After chosing 4 points in the source image (typically the 4 corners of the object I'm trying to rectify like the textbook), I chose 4 points representing the shape I want the source image to be warped to (in my cases, I just chose the shape of a rectangle).

Here are my results:

Textbook Source Img	Textbook Rectified
Walgreens Card Source Img	Walgreens Card Rectified
Painting Source Img	Painting Rectified

1.5: Blend the images into a mosaic

2-Image Mosaic

First I tried to make masks to blend the middles 50/50. I started off by creating masks for my two warped images (right and left).

Left Tree Warped	Right Tree Warped
Tree Left Img Mask	Tree Right Img Mask

Next I split my masks into 3 regions: mask of pixels only from the left image, mask of pixels only from the right image, mask of pixels that intersect both images.

Tree Left Mask	Tree Intersection Mask	Tree Right Mask
Tree Left Mask	Tree Intersection Mask	Tree Right Mask

Here is what the three combined together looks like together with our averaged middle section:

Tree Averaged Intersection

Doesn't look good so instead, take the maximum of each pixel in the intersecting region. In the regions that don't overlap, simply take the points from the original images.

Result:

Tree Maximum of Intersection

That looks better! There's less blurring and noticeable edges with this appraoch, so I chose to use the maximum pixel blending approach for the rest of the images because it seems to perform better than the average blending.

Tree Cropped

N-Image Mosaic

Rather than having this only work for 2 images, I needed to extend this to allow for an arbitrary number of images.

I achieved this by using the one at a time approach as described in the spec. I leave the right image unwarped, and then warp the left image into its projection. I repeat this from left to right iteratively until all images are warped and blended together.

Left side of campus

Middle of campus

Right side of campus

First I merged the left picture with the middle:

Left joined with middle

And then finally the left-middle merged picture with the right:

Left and middle joined with right

3 images Cropped result:

For my final example, I merged 4 images that I took of my backyard.

Left

Middle 1

Middle 2

Right

Here is the merging process:

Left merged with Middle 1

Left, Middle 1 merged with Middle 2

Combined merge

Cropped result of the 4 images:

4 Backyard's Cropped

1.6: What I've learned

The coolest thing I learned is how the making of panoramas on our phones works under the hood. Working through this project, it was definitely amazing to learn about this process because of all the parts that had to be accounted for (shifting after warping, seamlessly blending the two images at the intersection, aligning the images correctly). After making these mosaics myself, I learned that it's important to be steady and consistent when taking pictures for the best quality.