CS194-26 Project 6: Image Warping and Mosaicing

By: Shaan Appel (-abu)


Background and Approach

In this project we warp and stich together sets of images to create a panorama or mosaic. In part A we select correspondence points by hand. In part B we find these correspondence points automatically. We use the following steps:

Part A:
  1. Shoot and digitize pictures. (For mosaic photos we also carefully select about 10 correspondence points using Photoshop to get exact pixels)
  2. Recover Homographies: Solve for homography using system of equations and correspondence points.
  3. Warp Images: Apply projective transformation from homography using inverse warp.
  4. Blend Images into Mosaic: Use laplacian blending to blend mosaic images together.
Part B:
  1. Shoot and digitize pictures.
  2. Detect corner features in image using Harris corner detection and ANMS.
  3. Extract feature descriptors for each feature point.
  4. Match feature descriptors between two images.
  5. Compute homography using robust RANSAC method.
  6. Complete mosaic as in part A starting with step 3.

Part A

Image Rectification:

Before creating the full mosaic, we test our homography recovery and image warping by rectifying images. For this we pick 4 points in our image (need 4 correspondences for a projective transformation) and map them to a rectangular shape such that we get a frontal-parallel view.

Hut Art

Hut Art Rectified

Mural

Mural Rectified

Turtle Art

Turtle Art Rectified

City Painting

City Painting Rectified



Mosaic Images (Panorama):

Now we stich together image sets. Each picture in the set is taken from the same point of view, but with different viewing directions and overalapping fields of view. Because they have the same center of projection, we can recover the homography that encodes a projective transformation. This allows us to map all images to the same plane, so we can then align and blend to get a final image. I used laplacian blending to blend the images together. Using this technique we get the following results:

Living Room:

Original Images:

Left

Middle

Right

Warped Images: (Left and right warped)

Left

Middle

Right

Pano:

Blended Mosaic

Pano Cropped:

Cropped (By Hand)


Backyard:

Original Images:

Left

Middle

Right

Warped Images: (Left and right warped)

Left

Middle

Right

Pano:

Blended Mosaic

Pano Cropped:

Cropped (By Hand)


Church:

Original Images:

Left

Middle

Right

Warped Images: (Left and right warped)

Left

Middle

Right

Pano:

Blended Mosaic

Pano Cropped:

Cropped (By Hand)



Part B: Automatic Panoramas

In this section we create mosaics automatically as described above. We go through one example photo to demonstrate the steps:

Auto Panorama Steps:

Original Photos

Left Garden

Middle Garden

Harris Corner Detection (Min Dist = 10)

Left Garden Harris Corners

Middle Garden Harris Corners


ANMS (500 points)

Left Garden ANMS points

Middle Garden ANMS points


Feature Matching

Left Garden points with matching features

Middle Garden points with matching features


RANSAC

Left Garden RANSAC points

Middle Garden RANSAC points

Full Panoramas:

Hand Selected Points

Hand Selected Garden Pano

Auto Selected Points

Fully Automatic Garden Pano

Hand Selected Points

Hand Selected Church Pano

Auto Selected Points

Fully Automatic Church Pano

Hand Selected Points

Hand Selected Living Room Pano

Auto Selected Points

Fully Automatic Living Room Pano

Multi-Scale Processing (Corner Detection and Feature Description)

In order to detect corners at different scales I create an image pyramid of each photo. I then run the corner detection through feature mapping steps separately on each pyramid level. I then take the set of all points found at all levels and run RANSAC on the full-size image to get only valid correspondences.


This allowed us to find more valid correspondences, and therefore get slightly better homographies. I demonstrate the difference below:

Single Scale Garden Pano:

Multi-Scale Garden Pano:

Difference

Edge of Grass Not Aligned with Single-Scale

Aligned with Multi-Scale

Summary

Part A
  1. The coolest thing that I learned from this project is that different views are so easily mapped between each other with homographies. This makes the whole project possible is is really awesome.
  2. There were some tricky parts such as getting the images to align just right. I had to go into Photoshop and find the exact pixel locations when ginput just wasn't precise enough.
  3. I can't wait to get to automatic correspondence detection. Selecting points in Photoshop by hand is very tedious and time consuming.
Part B
  1. One really cool thing I learned was how to implement the ANMS algorithm, as it intially seemed complex in the paper we worked from.
  2. I also found it was really cool how robust RANSAC was. I didn't think I would have proper corresponding points until I saw how it would always cut down to valid points.