Overview
This assignment used image warping with a “cool” application of image mosaicing. I took photographs to create an image mosaic by registering, projective warping, resampling, and compositing them. Along the way, I learned how to compute homographies, and how to use them to warp images.
Part 1: Shoot the Pictures
I shot multiple photographs so that the transforms between them are projective (a.k.a. perspective). To do this, I shot images from the same point of view but with different view directions, and with overlapping fields of view. Another method I used was to shoot pictures of a planar surface (e.g. a wall) or a very far away scene (i.e. plane at infinity) from different points of view. I acquired my pictures using a Sumsung Galaxy S6 phone camera with the highest allowable resolution setting. I tried to shoot as close together in time as possible, so your subjects don't move on you, and lighting doesn't change too much. Additionally, I used identical exposures and focus locking so as to maximize the image brightness in the overlap region and overlapped the fields of view significantly to between 40% to 70% as recommended. I took a picture of building exteriors/entranceways and forests.
ORIGINAL IMAGES USED FOR RECTIFICATION AND WARPING
Fancy Building Room
Chyrsler Building, New York
Book
ORIGINAL IMAGES USED FOR PLANAR BLENDING
Apartment Building Left View
Apartment Building Right View
Genetics & Plant Biology Building Left View
Genetics & Plant Biology Building Center View
Genetics & Plant Biology Building Right View
Trees Right View
Trees Left View
Recover Homographies
Before warping the images into alignment, I recovered the parameters of the transformation between each pair of images, with the transformation being a homography defined by: p’=H*p, where H is a 3x3 matrix with 8 degrees of freedom (lower right corner is a scaling factor and can be set to 1). This can be described by:
I recovered the homography through a set of (p', p) pairs of corresponding points taken from the two images. Given that
p' is not at infinity, x' and y' can then be expressed as:
I implemented a method to recover the 3x3 homography matrix by using n-by-2 matrices holding the (x,y) locations of n point correspondences from the two input images. In order to compute the entries in the homography matrix, I set up a linear system of n equations (i.e. a matrix equation of the form Ah=b where h is a vector holding the 8 unknown entries of H). This is described below:
I provided more than 4 correspondences since with only 4 points, the homography recovery will be very unstable and prone to noise. This system could then be solved through least squares as follows:
Point matches were provided with a mouse-clicking interface.
Warp the Images
I warped my images using the parameters of the homography. I implemented a method to use the image to be warped and the homography matrix as inputs to warp the image, while avoiding aliasing when resampling the image. Image warping examples are shown below, where the original warped images used are shown above in the first section titled "Shoot the Pictures":
Warped Image 1: Fancy Building Room
Warped Image 2: Chrysler Building, New York
Warped Image 3: Book
Image Rectification
Below are examples of rectified images. To do this, I took a few sample images with some planar surfaces, and warped them so that the plane is frontal-parallel (e.g. the night street examples above).
Note that since here I only have one image that I need to compute a homography for, say, ground plane rectification (rotating the camera to point downward), I defined the correspondences using what I knew about the image. For example, if I knew that the image is square, I can click on the four corners of that image to be rectified and store them in im1_pts while im2_pts I define by hand to be a square, e.g. [0 0; 0 1; 1 0; 1 1]. After selecting the 4 points on the skewed image, I then used warping to determine a reasonable quadrilateral output region for the unskewed points.
Rectified Image 1: Fancy Building Room
(rectifying the image door)
Rectified Image 2: Chrysler Building, New York
(rectifying the Chrysler Building)
Rectified Image 3: Book
(rectifying the book text)
Blend the Images into a Mosaic
I warped the images so they're registered and created an image mosaic, using weighted averaging. All the images
are warped into a new projection, and are added in one by one, slowly growing the mosaic.
The image size is determined for the final mosaic and then warped into that size. Thus, a stack of images define the mosaic. Next, using weighted averaging, the images were blended together to produce a single image by applying simple feathering (weighted averaging) at every pixel. The pixel at the center of each (unwarped) image is set to 1 and it falls off linearly until it hits 0 at the edges.
Planar Projection (using Weighted Average Blending)
Apartment Uncropped Image
Apartment Cropped Image
Genetics & Plant Biology Building Uncropped Image
Genetics & Plant Biology Building Cropped Image
Trees Uncropped Image
Trees Cropped Image
What I Learned:
Implementing homography can be tricky, but it has many interesting applications for smoothly stitching images togther.
I finally have a better understanding as to how image mosaics and panoramas are formed.
