Project 6A: Image Warping and Mosaicing

cs194-26-aeo

Genevieve Tran

Part 1: Shoot and Digitize Images

When taking pictures of a scene, it's important that the camera does not leave it's pivot point. The camera is rotated from a single point, in which the images of the mosaic have 40-70% overlap. This will be important when we define our correspondences between the two images. Here are the pictures I took with my iPhone 7 camera. Unfortunately iPhone automatically adjusts aperture, so the photos when combined are going to be slightly off in coloration. Additionally, since I do not own a tripod, pivoting my phone centered at the camera was quite challenging. All photos were taking near North Berkeley.

Part 2: Recover Homographies

Overview: in order to merge two images into one we first must warp one image into the plane of another. To do this, we must recover the homography that warps between the two images. A homography is a linear transformation, H such that p' = Hp, where p and p' are homogeneous coordinates of the 2 images.

So H is a 3x3 matrix, but we set the 9th entry to 1, so we have 8 degrees of freedom/variables we want to solve for. Computing the exact H is an overconstrained problem, so we approximate H using least squares by solving the equation Ah=b. In order to compute the homography matrix I constructed the following optimization problem.

Part 3: Warp the Images

Overview: this part was considerably trickier. I used the same logic of warping from project 4, except this time we were allowed to use library functions. In order to calculate the shape and the size of the output image in such a way that the whole warped image fits onto the canvas I piped the corners first and created a new transformation matrix to dot it with the original homography matrix so that it also takes care of shifting the warped image by the necessary offests. Using the max and min values of x and y for the piped corners I also found the new output shape to pass into the skimage transformation warp function.

Image Rectification

Overview: At this point in order to make sure both the homography calculation and the warp function behave properly I attempted to rectify a number of images with items that have a square like objects with planar surfaces. I rectified by warping the images so that that plane is frontal-parallel. The original points were chosen manually, and the new points were chosen manually as well, through guess and check to ensure the correct width to height ratios were used. Top three examples are of pitcures I took myself and the last one is a photo I found on the Internet.

Frontal-parallel plane - keyboard.

Frontal-parallel plane - mirror.

Frontal-parallel plane - heater in the corner. Image was cropped to see the heater better.

Frontal-parallel plane - canvas with the hot air balloon.

Part 4: Blend the Images into a Mosaic

Overview: all of the previous steps have been leading to this most challenging part. For all panoramas I shot three images and calculated the homographies of the right and the left images into the plane of the center (middle) image. Before warping images I added an alpha channel to each one in order to do simple weighted averaging later. I proceeded to calculate the output size of the entire panorama and the offests (added to original homographies) for all three images. I then warped each image using the appropriate homography and output shape. Then to combine all three images I simply extracted the alpha channels of each three warped images and added them together. This allowed to to identify where certain pixels had overlapping values from multiple images. Then I averaged those pixels by dividing the summed image (of all three warped images without the alpha channel) by the sum of alpha channels. I've provided three uncropped examples below.

You will notice that the first two examples are slightly blurry. This is because the manually picked correspondence points are not very reliable. Furthermore, the images were taken from approximately the same point but not exact.

I also played around with another blending techinque described on the website. I set the alpha in the middle of an image to 1 and slowly trickled down to 0 off to the sides. Below is an example of such blending based off of images I borrowed from a friend.

Part 5: Lessons Learned

I learned that warping has some very powerful applications and that correspondece points play a big role in precision. It was nice to play around with linear algebra and figure out why certain things were not working as expected because there was always a logical explanation that could be calculated.