CS194-26 Project 5a: Image Warping and Mosaicing

Michael Park - Fall 2020

Objective

The main objective of this project is to create a mosaic of images through projective warping. This project utilizes image processing techniques borrowed from previous projects. For instance, image warping (forward and inverse warping) is a concept also used to warp triangulates for face morphing in Project 3. Image blending is a concept explored in Project 2, where I generated masks and filters to perform seamless blending of two similar images. This project utilizes these techniques to stitch multiple images to generate a mosaic that encompasses a particular scene.


Shoot the Pictures

First, I took some pictures of scenes with my smartphone camera. While I made sure that AE/AF Lock was toggled for consistency, Samsung phones cameras are known to generate visible image distortions, making it difficult to fully capture the scene. I tried my best to take pictures at the same settings, overlapping the fields of view significantly.

Below are the pictures:


Recover Homographies

To generate warps, a means of transformation is needed. For Project 3, I used affine transformation to warp a triangle to its corresponding pair. While this method was fitting for triangles, it does not quite fit for this task. This is because the images are taken from different perspectives. To match the corresponding features, I had to warp an image to another by changing its perspective, and this is where projective warping is used. In particular, I used homography p'=Hp, where H is a three-by-three matrix with 8 degrees of freedom, and p, p' are vectors composed of the x and y-coordinates of points and their corresponding pairs, as well as their normalizing coefficients. Using this equation, I can transform a point p to its counterpart p' by first taking a matrix product with H, and then dividing the result by the bottommost element (normalizing coefficient) of p'.

To achieve this through Python, I hard-coded the matrix as described by the below image. To retrieve H, I modified the above equation into a least squares-like form. Then, I used np.linalg.lstsq to solve for the eight elements of the homography matrix. Lastly, I appended 1 to the resulting vector, and reshaped it to a three-by-three matrix.


Warp the Images

To be able to use the homography matrix, I first had to convert my image into a usable format. To achieve this, I had to generate a mapping of indices of each pixel of the images. For each pixel, I mapped the x-coord, y-coord, and normalizing coefficient (1 for input images). By then I could multiply the giant matrix by the homography matrix to retrieve a modified mapping of the pixels.

Below are the images after applying homography:


Image Rectification

To test out my code, I took some pictures with planar surfaces, and then warped them with projective warping such that the surfaces are frontal-parellel. In particular, I defined correspondences such that rectangular labels would be warped to be more visible.

Below are the images and their rectifications:


Blend the Images Into a Mosaic

Finally, I took an image and its warped pair and stitched them together to form an image mosaic. One glaring issue I faced while performing this step was that the warped images were cropped off, not displaying pixels outside its original shape. To address this issue, I used the bounding box technique to pad the images such that the warped image will fit entirely within the dimensions of the padded image. I retrieved the coordinates of the corners and forward warped them using the inverse of the homography matrix (np.linalg.inv(H)). To align the warped image with its pair, I had to pad the latter the same way I did for the former. By then I could finaly align the images together.

Blending the images were also tricky to think about. For this project, I used linear blending, taking the weighted average of the two images for the overlapping section. To achieve this, I generated a mask of the resulting mosaic by combining the binarized version of the images (black and white) and assigning a weight to the overlapping section. I experimented with different values of the weights to achieve visually smooth blending.

Below are the results:

While this project was conceptually simple, it was time-consuming in that I had to familiarize myself with how to achieve the effects I wanted with Python libraries. I learned that seemingly fancy features seen in fancy cameras are based on surprisingly simple concepts.