Sergei Mikhailovich Prokudin-Gorskii tried to photographed thousands of color pictures without a color camera. He attempted to do this by recording three exposures of every scene onto a glass plate using a red, a green, and a blue filter. He hypthesized that if he could add color to the filters later on, the picture would appear in full color. This project used Prokudin-Gorskii's glass plate images and algorithmically shifted the plates so that the picture could be reconstructed from the 3 filters.
I started with the smaller jpg images, and used SSD with the recommended window size of [-15, 15] pixels both vertically and horizontally. This worked reasonably for all of the photos except for the Monastery picture. I realized that np.roll did not work so well because the edges would interfere with the accuracy of SSD. Thus, I decided to crop the photo by 15px on each side before taking the SSD and was able to generate nice photos. However, the end code changed this value from a flat 15px to 10% of the image so that it would work better with the recursive pyramid. To deal with the bigger images, I kept scaling the photo down by a factor of 9/10 until the image was under 100x100 pixels. I actually started scaling by a factor of 1/2, but only a few of the .tif images ended up aligning properly, so I gradually moved to smaller windows and larger scaling factors to improve both runtime and accuracy. With the factor of 9/10, I ended up using a small window of [-3, 3] on each level of the pyramid, and had about 20 layers. This layering is what allows the individual glass plates to be able to move hundreds of pixels if needed. Once this recursive behavior was implemented, I was able to successfully move all the .tif images accordingly.
The biggest challenge that I faced was trying to understand the shortcomings of SSD, because I consistently got images close, but a little bit off. It wasn't until I realized that np.roll moves the last column or row of a matrix to the beginning that I realized the importance of cropping the edges of the image. After finishing the jpg images, I also thought that simply cropping by 15px or whatever the window size was would fix that issue, but the tif images also were slightly off. I realized that instead of trying to figure out a flat amount of pixels to chop off, it should really be based off of the image size itself, and change it to a percentage really boosted the quality of the final photos.
