Sam Zhou
CS194-26-aff
Image Mosaics
The goal of this project is to apply affine transformations to entire pictures in order to produce mosaics of multiple images. Given a set of correspondence points, we can estimate a homography that transforms the first set of points to their corresponding points. This homography can be applied to the entire image to warp it so that the two images look like they smoothly transition between the scenes.
Image Rectifying
First, we take an image and define a few points for an object we wish to rectify. We then manually define a set of correspondence points which signify what we want the object to look like after the transformation.
In this example, we took a picture of a painting from the side and wanted to change it to a head on look at the painting.
In this more extreme example, we wanted to change the perspective to look like we were staring down at the dirt. It worked surprisingly well.
Image Stitching
Now we take two images whih were taken side by side and try to stitch them together into a mosaic. We intentionally take the images so that there is an overlapping part of the scene. Then we click both images to define correspondence points and calculate a homography using those correspondence points. We can use the homography to warp the entire second image and stitch it to the first by creating a new canvas and placing the warped photo on top of the non warped photo.
Here are some examples of photos, their warps, and the final mosaic.
Archery Competition Originals
Second Scene Warped
Archery Mosaic
Jacobs Hall Originals
Jacobs Warped
Piecewise Jacobs
Streetside Shots Originals
Streetside Warp
Stitched Street
Automatic Mosaic
In order to automate this process, we need a better way to define correspondences. We did this by following the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al. First we detect potential points by finding Harris corners. Here are the 500 highest “potential” Harris corners on one of the archery images.
The issue here is that many of the points are clustered very close together. This leads to worse homographies as points far from these clusters are not mapped well. Using Adaptive Non-Maximal Suppression, we were able to pick out a better set of points which were better spread across our scene. Here’s an example on the same image after we used ANMS to find 500 new points.
After we have identified our points, we need to find correspondences between these potential points across two images. We did this by sampling every 5th pixel from a 40x40 square around each potential point and using this 8x8 sample as the point’s defining feature. We found matching pairs by comparing their features’ squared distances from each other and using the closest match. We also tossed away some matches where the first and second closest match were close in score since this indicated there were actually no good matches in reality. This corresponded heavily with points which were found by ANMS but were not in the shaed space between the photos. here are some matching points in the two archery scenes.
Finally, we further refine our set of matching points and define our final homography by using the RANSAC algorithm. This gave us the homography that produced the highest number of matches when we transformed our first set of points to their matching points.From there we proceed as we did before to stitch together the photos. Here are some of the same mosaics done manually, but now with automatic point matching and homography estimation.
Archery Range
Jacobs Hall
Street with a Car
Jacobs at Sunset
If you compare these with the manually selected points, you can see how much better these automatically stitched photos were.
Lessons Learned
Affine transformations can really do a lot. The manual correspondence points were neat to see, but the automatic system was much cooler and also much better in terms of final results. All of the steps took advantage of pretty straightforward procedures and very logical reasoning. It was cool to see it all come together to make an impressive tool.
Sam Zhou
CS194-26-aff
Image Mosaics
The goal of this project is to apply affine transformations to entire pictures in order to produce mosaics of multiple images. Given a set of correspondence points, we can estimate a homography that transforms the first set of points to their corresponding points. This homography can be applied to the entire image to warp it so that the two images look like they smoothly transition between the scenes.
Image Rectifying
First, we take an image and define a few points for an object we wish to rectify. We then manually define a set of correspondence points which signify what we want the object to look like after the transformation.
In this example, we took a picture of a painting from the side and wanted to change it to a head on look at the painting.
In this more extreme example, we wanted to change the perspective to look like we were staring down at the dirt. It worked surprisingly well.
Image Stitching
Now we take two images whih were taken side by side and try to stitch them together into a mosaic. We intentionally take the images so that there is an overlapping part of the scene. Then we click both images to define correspondence points and calculate a homography using those correspondence points. We can use the homography to warp the entire second image and stitch it to the first by creating a new canvas and placing the warped photo on top of the non warped photo.
Here are some examples of photos, their warps, and the final mosaic.
Archery Competition Originals
Second Scene Warped
Archery Mosaic
Jacobs Hall Originals
Jacobs Warped
Piecewise Jacobs
Streetside Shots Originals
Streetside Warp
Stitched Street
Automatic Mosaic
In order to automate this process, we need a better way to define correspondences. We did this by following the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al. First we detect potential points by finding Harris corners. Here are the 500 highest “potential” Harris corners on one of the archery images.
The issue here is that many of the points are clustered very close together. This leads to worse homographies as points far from these clusters are not mapped well. Using Adaptive Non-Maximal Suppression, we were able to pick out a better set of points which were better spread across our scene. Here’s an example on the same image after we used ANMS to find 500 new points.
After we have identified our points, we need to find correspondences between these potential points across two images. We did this by sampling every 5th pixel from a 40x40 square around each potential point and using this 8x8 sample as the point’s defining feature. We found matching pairs by comparing their features’ squared distances from each other and using the closest match. We also tossed away some matches where the first and second closest match were close in score since this indicated there were actually no good matches in reality. This corresponded heavily with points which were found by ANMS but were not in the shaed space between the photos. here are some matching points in the two archery scenes.
Finally, we further refine our set of matching points and define our final homography by using the RANSAC algorithm. This gave us the homography that produced the highest number of matches when we transformed our first set of points to their matching points.From there we proceed as we did before to stitch together the photos. Here are some of the same mosaics done manually, but now with automatic point matching and homography estimation.
Archery Range
Jacobs Hall
Street with a Car
Jacobs at Sunset
If you compare these with the manually selected points, you can see how much better these automatically stitched photos were.
Lessons Learned
Affine transformations can really do a lot. The manual correspondence points were neat to see, but the automatic system was much cooler and also much better in terms of final results. All of the steps took advantage of pretty straightforward procedures and very logical reasoning. It was cool to see it all come together to make an impressive tool.