CS 294: Project #5A: Image Warping and Mosaicing ΒΆ

Emaad KhwajaΒΆ

Shoot the Pictures ΒΆ

Below are the images used for this project. I took images Street View 1 and Street View 2 in Queens, NYC for warping and mosaicing. These were done by standing firmly in the middle of the street and rotating my body about 30 degrees.

These images were not so interesting for rectification, so I used an image from a Egyptian restaurant in downtown Manhattan for that part.

Recover Homographies ΒΆ

Correspondence points were selected on both street view images (shown in green). These points were used to calculate the homography matrix.

The homography matrix H is calculated to map the coordinates from image 1 to image 2 via least squares. The matrix is shown below (rounded for visual pleasure).

<Figure size 432x288 with 0 Axes>
$\displaystyle H = \begin{bmatrix} 1.00554 \space \space \space -0.03668 \space \space \space 0.0 \\ 0.00074 \space \space \space 1.03878 \space \space \space -0.0 \\ 1979.84344 \space \space \space -18.54425 \space \space \space 1.0 \end{bmatrix}$

Warp the Images ΒΆ

For this situation it made the most sense to fix one image and shift the other to align the corresponding points. The second image was chosen to be fixed. The bounding box of the images is also expanded to accomodate the other image.

Image Rectification ΒΆ

Image rectification on the street would be a little boring, so I chose a more interesting photo. This image is from an Egyptia restaurant in SoHo. On the ceiling are a series of LED signs. I thought it would be cool to rectify that section to get a clearer view of them. I annotated the corner of that square sign with the hexagonal pattern.

The green points correspond to the sign corners, and the orange correspond to a manually designated square of with lengths corresponding to the longest distance between the green points.

<matplotlib.image.AxesImage at 0x1279754c0>
<Figure size 432x288 with 0 Axes>

A new homography is calculated for this image.

Rectifyingn the image, we can see the signs are significantly more clear. We can even clearly make out the arabic now.

The proportions look a bit off despite fitting to a square. This is likely because the original sign was not actually a square, but these results are still very good.

Blending Images into Mosaic ΒΆ

Averageing the warped street view images works, but the results are very strange. We have color variations depending on if there is image overlap.

I generated masks from the warped images by thresholding where the pixel values were greater than 0.

I multiply these masks and their inverses to create 4 sections of the image.

The overlapping sections are averaged together, while the mutually exclusive pieces are taken at full magnitude.

<matplotlib.image.AxesImage at 0x1290beca0>

We do have slight ghosting here. This is likely because I had some accidental degree of translation along with rotation when taking the image (what can I say, there were cars about to come).

We try to mitigate this a bit by implementing a 2-level Laplacian stack. Instead of averaging the overlapping regions, I instead select the View 2 to be on top (Piece 3). Aside from the abrupt end of the white line in the bottom-left corner, these results look pretty good.

<matplotlib.image.AxesImage at 0x12443e250>

Tell us what you've learned ΒΆ

Doing this assignement made it very clear how signficiant very small off-axis effects can have on the prospective image alignment capabilities. While I was not aware of the small translations my hand did, it was enough to significantly shift the focal plane. While trying several iterations of this, I was able to align different parts of the image better by creating bias in that direction (adding more points to make the SSE favor alignment in that region), but I was unable to globally get rid of the doubling.