Shoot and digitize pictures
Example 1
I used an iPhone with AE/AF lock when necessary, and resized images to 720x960. Some images are from my photo library and may have been taken with earlier phones, or as screenshots of e.g. Angry Birds AR Isle of Pigs.
Recover homographies
For this part, I coded a code to recover homographies. It would express the problem as a Least Squares. The key innovation is that though the matrix being recovered was 3x3, I rearranged the matrix into a 1x9 vector. This required flattening the output matrix into a vector, and creating individual rows for each subset of the input matrix that would multiply to the output matrix.
The end result is that you can solve for the homography matrix using A = one set of correspondences, x = the transform matrix, b = other set of correspondences. The interesting part is that just with this setup, the bottom row of the matrix will be nearly equal to 0, 0, 1 as expected, without any further constraining of the least squares, problem, because of how homogenous coordinates work.
Warp the images
Example 1
Original image:
Rectified image. The rectified segment was the window
The rectified segment cropped:
Example 2
Original image:
Rectified image. The rectified segment was the inner part of the doorframe (the black square, not the outer cube). It's a bit hard
to see the rectification because the 3d nature of the image makes it look
less rectified than it is.
Blend images into mosaic
For this part, I estimated the size manually using a conservative estimate. That conservative estimate would guarantee that the full image is captured. I would then use magick mogrify -trim to trim out any blank space in all the output images.
One thing I noticed was that having low quality correspondences would reduce the quality of the final mosaic, which could cause difficulties.
For alpha feathering, I used a simple linear function of distance from the center to create a smooth blend. This would only apply when multiple images had pixels in the same location.
Example 1
Original images:
Mosaic image:
Example 2
Original images:
Mosaic image:
Example 3
Original images:
Mosaic image:
What you learned
What I learned from this project is the importance of smoothly interpolating masks, as otherwise the transform looks much worse at the edges. I also learned that not vectorizing code can severely negatively impact performance. Also, I learned that labeling a lot of correspondence points is not useful. It's more important to have two images that are good matches for each other so that the general transform can work in the first place, or its doomed.