Photo Mosaics (CS 194-26 Fall 2018 - Project 6-1)

IVAN JAYAPURNA - CS194-26-ABT


Overview (What I've Learned)

The goal of this project was to explore image warping beyond the simple translations we've done so far for 2 cool applications: 1.) Image Rectification and 2.) Image Mosaicing. In this project I captured images on my phone, calculated homography matrices mapping 2 images to each other, applied both forward inverse warping for transformation and finally alpha blended to stitch images together.


my favourite image mosaic result

Through completing this part of the project, I consolidated my understanding of homographies, finally understood the pro's and con's between forward and inverse warping, forward warping is no good because not every pixel in the destination image will be filled! I learned that to accurately predict the size of a result image we can apply a forward warp to the corners of an image first. I think Image Rectification is actually what i personally find coolest about this project, in class we saw how we could see floors of paintings and that inspired me to do similar and look at cool ceilings! Finally I learned that picking points accurately and with a decent spread across the image is crucial to getting a good stitching. Smartly picking points is far more effective than blindly adding more, and I can't wait for part 2 where I'll write code such that I'll never have to pick points again.


Part 1: Shoot Images

Here are the images that I shot using my Samsung Galaxy S5 Phone, that will be used for image mosaics. Note that before actually using these images (~5000 x 3000 px) I resized them to save computation time and file storage space for upload to this site.

brunch_1
brunch_2
doe_1
doe_2
house_1
house_2
smog_1
smog_2


Part 2: Image Rectification

In this part of the project I compute homographies and warp images, by mapping 4 points onto a new flat, projected surface to somewhat replicate a front-parallel viewing of features of interest in the scenes. Although 4 points is unstable, I found that the results they yield are perfectly fine for this portion of the project and it saved me clicking (definitely not for mosaics). If anything extra points made my rectified images worse, as I was very coarsely selecting points in the new image totally by guesswork - more points makes guessing reasonable locations more difficult. Points are selected via ginput manual selection, homographies are solved via least squares, and an inverse warping is done. The images below show 2 cool ceilings and a shifted view into an archway.

Scrabble Original
Scrabble Rectified
Miami Beach Original
Miami Beach Rectified
Kitchen Mural Original
Kitchen Mural Rectified


Part 3: Blend Images into Mosaic

In this core part of the project, I build upon the same core techniques used in the previous parts (Homographies and Image Warping), but this time mapping exact, correspondance points between 2 images. Forward warping on the 4 corners of the original image were calculated to get size of result image, stitching was done by calculating a translational offset, and blending was done with basic alpha blending, which provides surprisingly good results - no noticable artifacts (other than a ghostly phone in the brunch_mosaic that dissappeared between images)! I selected between 20-24 points for each of the 4 pairs of images I shot (Part 1), and warped the left image to fit with the right image for the results shown below.

brunch_1
brunch_2
doe_1
doe_2
brunch_mosaic
doe_mosaic
house_1
house_2
smog_1
smog_2
house_mosaic
smog_mosaic (larger version @ top of page)

Part 2

Part 4: Detect Corner Features in an Image

The first part of automatic point picking is to detect Harris corners, code to detect Harris corners. For this entire process I will be using images I captured in Part 1 of Doe Library. Note that colors are slightly off due to the way cv2 saves images with different colour channels, final results are unaffected, and points are still clearly visible.

Image 1: Harris Corners
Image 2: Harris Corners

Once Harris corners are found, we notice that there are far too many, thus we use Adaptive Non-Maximal Suppression to reduce the number of points to the 500 "strongest" Harris corners in each image. This is done as per the equation in the paper, a constant of robustness of 0.9 was used as in the paper. Resulting points are shown below.

Image 1: 500 ANMS Points
Image 2: 500 ANMS Points

Part 5: Extracting & Matching Features

The next step is to extract features for each points to try match points between image 1 and 2. Windows of 40x40 pixels were taken across the image, subsampled to 8x8, normalized, then flattened. These features were then matched to one another via distance comparison using the dist2 function given. The ratio between the top 2 matches were taken (1-NN / 2-NN) and if the ratio was below a certain threshold I chose to be 0.2, then the point was determined to be a good match. On average this would drop the number of resulting "matching point pairs" from 500 --> ~30.

Image 1: Matched Points
Image 2: Matched Points

Part 6: RANSAC

Finally, to ensure that no anomalous / invalid point matchings were made, I used the RANSAC method to randomly subsample 4 points from Image 1, compute a homography matrix from them, test the H matrix on all the points in Image 1, and count the number of points for which the homography is "valid" by an L2 norm comparison with corresponding points in Image 2 and a threshold I chose to be 0.3. This was repeated for 2000 iterations, and the largest set of inliers was used to compute a final homography matrix.

Image 1: Final Correspondances
Image 2: Final Correspondances

Part 7: Final Auto-Mosaics & Comparison

Finally, with the automatically, computer-generated correspondance points the steps above determined, we repeat the procedure used in part 1 to generate homographies, warp, stitch and blend images together, to generate the results below. For comparison manual results from part 1 are also displayed below alongside automatic results as labeled.

Manual Result
Automatic Result
Manual Result
Automatic Result
Manual Result
Automatic Result

As can be seen results are very comparable, and aesthetically appealing for both automatic and manual methods. For the manual results I somewhat coarsely selected 22-24 points for each image, and on average my code for automatic point selection used ~8 as shown above. One thing to note is that I was unable to reproduce the "house" image mosaic I manually generated in part 1, due to what I believe to be a bug with how harris corners are detected (others had issues with this on Piazza as well), where the entire sky was picked as harris corners, completely ignoring the "min_dist" parameter.

Project Final Takeaways (Part 2)

I think automatic point detection is incredibly cool and powerful. Point selection in Part 1 was incredibly tedious, and also imprecise. As my results above show, computer generated points can achieve the same output results using less than half the points I manually picked. This project also improved my skills at reading techinical papers and translating them into functional code, a skill I am sure will be useful for my final project and other projects in the years to come.