Overview
Sergey Prokudin-Gorsky, a Russian photographer in the early 1900s, is most known for his impact in the early days of color photography. At the time, there was no equipment adequate enough for colored photography; however, he had the idea of taking a picture three times, each time using a different colored filter. This can later be recreated by placing the appropriate filter in front of a standard light projector. He was tasked with documenting the Russian Empire; unfortunately, his work was cut short due to the Russian Revolution. While some of his photographs were confiscated or lost, the Library of Congress managed to purchase a decent collection of the remaining ones, and later would clean and release them to the public. In this project, we are to take the black and white photographs and recreate Prokudin-Gorsky's vision of viewing Russia in color.
Approach
After separating the stacked photographs into the individual color channels, we need to find a way to align the images properly on top of each other. For small images, a brute-force search across a range of 30 pixels in each direction is adequate. There are a couple approaches to evaluate the best alignment - the smallest sum of squared differences or the largest normalized cross-correlation. While implementing these functions, I discovered that the border affects the alignment scores more than it should, considering it is not part of the picture. After ignoring 10% of the image on each side, I found that both formulas achieve identical results, with SSD running slightly faster.
The larger tiff images have too bif of a range is too large to efficiently iterate over. For these pictures, I implemented an image pyramid technique to find the best alignment. I downscale the pictures by powers of 2 until the image is small enough to easily work with. After finding the alignment for the small image and with a little bit of math, I can then downscale to the next smallest factor and probe a smaller range of alignments. In essence, the image pyramid employs a binary search technique to find the optimal alignment more efficiently.
Bells and Whistles
Since the images are not aligned perfectly, the borders become an amalgamation of the primary and secondary colors. I implemented automatic cropping to remove these eyesores. Rather than calling complicated border detection formulas, I simply took the mean across the rows and columns and reasoned that the minimum value would be on the border. I then arbitrarily set a threshold to indicate where the border stops and the image begins. After adding a small pixel buffer (0.75% of the dimension) to remove excess artifacts and some more math to figure the impact from the alignment itself, I simply take the maximum border size across all three colors in the four directions and crop that amount. This technique struggles against naturally dark images (e.g. the top border of icon.tif) and pictures with a gradual fading of the border (e.g. train.tif on the top border). I additionally did some preprocessing to remove the white borders from the top and bottom. This change makes the division of color channels slightly more accurate, salvaging a few more rows of the image. Note that this also alters the alignment values in the y-direction compared to the naive implementation.
