Part I

Part II

Image Warping and Mosaicing (Part I)

Ron Wang

CS 180, Fall 2023, UC Berkeley

1. Recover Homographies

The homography, also known as a perspective transform, operates on homogenous coordinates. We have

\[ \begin{bmatrix} x' \\ y' \\ w' \end{bmatrix} = H \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} \]

where H is an arbitrary 3 x 3 matrix,

\[ H = \begin{bmatrix} h_{00} & h_{01} & h_{02} \\ h_{10} & h_{11} & h_{12} \\ h_{20} & h_{21} & 1 \end{bmatrix} \]

To solve for the 8 unknowns, we set up a linear system of n equations of the form Ah = b. These can be constructed using our coordinates:

\[ (h_{20}x + h_{21}y + h_{22})x' = h_{00}x + h_{01}y + h_{02} \] \[ (h_{20}x + h_{21}y + h_{22})y' = h_{10}x + h_{11}y + h_{12} \]

Thus, given im1_pts and corresponding im2_pts, the computeH function gives a homography matrix that recovers the transformation. In this first part, the corresponding points are found using a mouse-clicking interface. I labeled the following images, which were taken on my phone.

Image 1

Image 2

Image 1 (labeled)

Image 2 (labeled)

2. Warp the Images

For this part, I wrote a function, warpImage, that takes in an image im and a homography matrix H, and computes the warped image.

Image 1

Image 1 Warped

3. Image Rectification

The image warping function also allows us to "rectify" an image, which means transforming images with planar surfaces so that the plane is frontal-parallel. Here are a few results:

The Economist

The Economist (frontal-parallel)

The most notable difference is perhaps the line containing date information. It was barely readable from a distance, but now, after the transformation, it is much more legible.


Costco (frontal-parallel)

4. Blend the images into a mosaic

Finally, we can blend the images to create a single image mosaic. We leave one image (image 2, the "target" image) unwarped, and warp the other into its projection. Here are the two input images again:

Image 1

Image 2

Here is the mosaic result:


Example 2

Fire Trail 1

Fire Trail 2

Here is the mosaic result:


Manually selecting correspondences is both exhausting and inaccurate, which causes the blurry central region in the image. In the second part of the project, I will work on automatically selecting correspondences.

Example 3

Learn more about the Tyndall effect here.

Tyndall Effect 1

Tyndall Effect 2

Here is the mosaic result:


Since the two images are essentially screenshots from a video, they are significantly affected by changes in lighting conditions. Smoothing out these lighting variations to make the two parts of the mosaic appear more natural is another aspect I will address in the upcoming phase of the project.

Feature Matching for Auto-Stitching (Part II)

In this second part of the project, I re-implemented the algorithms for multi-image matching described by Brown et al but with several simplifications. Specifically, the five steps are:

1. Interesting Point Detection

Using the harris package provided by staff, I was able to find the Harris corners for each image.

Indoor 1 (Harris Corners)

Indoor 2 (Harris Corners)

One immediate thing we can notice here is that there are way too many Harris corners. Ultimately, we are looking for invariant and distinctive point descriptors, so we need to get rid of the noisy corners and only keep the best results.


To this end, we use the adaptive non-maximal suppression (ANMS) method discussed in the original paper. Here are the results of both images after we've applied ANMS to suppress the less optimal corners.

Indoor 1 (ANMS)

Indoor 2 (ANMS)

3. Feature Descriptor Extraction

Next, we extract the descriptions of local image structures to support the reliable and efficient matching of images. As decscribed in the MOPS paper, we start with a 40 x 40 window, and downsample it to 8 x 8 (this is performed at a higher pyramid level, i.e. with Gaussian filter applied). The descriptor vector is then normalized to mean 0 and standard deviation 1.

4. Feature Matching

With the descriptors extracted, we can move to the feature matching stage. We are essentially finding nearest-neighbors for the descriptors. It works as follows: Here are the matched features:

Indoor 1 (ANMS)

Most matches are good, although there are a few outliers, which we will address using the RANSAC algorithm.


The RANSAC loop works as follows: Here is the mosaic generated using homography H computed this way:

Indoor (Mosaic)

Let's compare it to the previous version, whose homography was computed using hand-labeled points:

Indoor (Auto-Stitching)

Indoor (Manual)

Auto-stitching provides us with a more stable and reliable way to create photo mosaics.

More Examples

Example 2: Trail

Trail 1 (Harris Corners)

Trail 2 (Harris Corners)

Trail 1 (ANMS)

Trail 2 (ANMS)

Trail (Matches)

Trail (Mosaic)

We can compare and see the improvements:

Trail (Auto-Stitching)

Trail (Manual)

Example 3: Sky

sky 1 (Harris Corners)

sky 2 (Harris Corners)

sky 1 (ANMS)

sky 2 (ANMS)

sky (Matches)

sky (Mosaic)