Nicole Rasquinha
Computational Photography
Project 6A: Image Warping and Mosaicing
In this project, we learn how to rectify images, in other words, take a plane which is in some non-frontal perspective and rectify it to be front-facing. Furthermore, we learn how to mosaic two images together by shifting the perspective of one to fit the other. Both rectifying and mosaicing rely on the concept of homographies. For the rectifying task, I simply find four points in the image on the plane I wish to rectify that form a square or rectangle. I then transform the image to a set of points which represents an actual square or rectangle on a grid. The over-arching approach of mosaicing is to first determine corresponding points between two images you wish to stitch together. Then, compute a homograhy to transform from one image to another. Recall that the homography matrix, H, is a 3 by 3 matrix but that the bottom right entry is simply a scaling factor and can be set to one. Thus we have eight unknowns. With the four correspondence points, you have eight equations (2 dimensions per point) in order to solve for a system of equations to recovery H. In my code, I use numpy's least squares solver to solve this system. Determine the resulting canvas size by taking the max and min row and column coordinates of the second image's corners and the result of warping the first image's corners. "Paste" the warped image 1 and the shifted image 2 accordingly. Finally, apply some linear blending along the axis of the panorama to smoothen the edge between images.
I'll be there for you... Check out this picture of the apartment building from the classic sitcom Friends.
Would be nice to see what the building looks like from one of the sides. Unfortunately, I didn't walk down that way while strolling through NYC. Luckily we can just rectify the image and nobody will know!
So... looks a little funky. Since we are changing the perspective of the image so drastically, the corners REALLY get warped into this dagger shape. With a little bit of cropping we can get the following image instead.
Great! Now let's fly to Lucerne, Switzerland and check out a few more examples.
From left to right: original image, rectified image, rectified and cropped
I tried to rectify this image of myself obscuring the view of the gorgeous town of Lucerne. I tried to rectify the face of a building very far off in the distance. The further you get from the object, the more you'd have to move the position of the camera to achieve the change in perspective in real life. Also, warping the image a lot causes a lot of interpolation, worsening the appearance. Finally, since there is a subject (me) very close-up in the shot, I have to be warped quite a bit and this warping is especially noticeable. From left to right: original image, rectified image, rectified and cropped
Now it's time to go check out the Swiss Alps!
For the second task in this project, mosaicing, let me take you on a tour of the Swiss town of Bettmeralp and the beautiful Alestch Glacier it boasts access to. Cool note: I actually took all the following images from panoramic-esque videos from my trip. I found this to be a terrific approach to gathering panoramic frames. Let's start at the hotel I stayed at. Here are the original images.
On the left is the transformed image one, and on the right is the mosaic with image two naively pasted on top.
Although the image looks pretty good from a distance, if you look closely you can see a harsh edge between images one and two. To address this problem, I implemented feather blending along the x-axis (weights transitioning from 0 to 1 and vice versa in increments of 1/20) for a more smooth transition between the images. Here is my result:
That's much better, but since I am only feathering along the x-axis, you can still see horizontal edges if you look closely. To remediate this, I add additional feathering such that if you are at an overlapping pixel, you feather according to the closest edge (vertical or horizontal). Here is my final result:
Voila! Now the mosaic is more-or-less seamless! :)
Let's hike onwards to this famous glacier. Is it worth the trek?
As you can see, the horizontal edge is extremely apparent here. Let's apply our additional vertical feathering here as well!
I really like this result. The additional feathering makes a huge difference. And no hike is complete without seeing some happy Swiss cows!For the remaining examples, I will only show the result with both horizontal and vertical feathering.
Can you catch a glimpse of the glacier over yonder?
Our journey has come to close and we find ourselves back in Berkeley. As a cute little bells and whistles extension, I decided to have some fun with this powerful mosaicing tool.Here I am (in my PJs) reuniting with... myself?
Due to my phone's automatic brightness settings, the picture on the left is noticeably brighter than the one on the right. As a quick fix to this issue, I brightened the second image, and re-ran the algorithm. Here is the brightened version of image two and the mosaic result:
I enjoyed this project quite a lot. It was fun to see these images come to life with the added perspectives. I also learned (from personal hiccups) that certain images do not work quite as well. For example, I learned about the relationship between distance and perspective from the Lucerne photo, which I explained in detail in the rectification section. Another small issue I had when stitching together my Berkeley images was difference in brightness. Clearly differences in brightness result in a less realistic output, and it would be nice to implement a mosaicing algorithm that is not sensitive to differences in brightness. During Part B, I believe I will get a chance to explore this further!