In the 1900s, Prokudin-Gorskii gained permission by the tzar to take pictures throughout the vast Russian empire. He recorded exposures of every scene on glass panes using red, green, and blue filters, envisioning the future world of colored images. This project seeks to reproduce the true color images using image alignment techniques.
First, out of curiosity, I tried to see how the image would look if I didn't do any alignment at all.
Next, I tried displacements of rows and columns of the array representations to align the red channel with the blue, as well as aligning the green channel with the blue. This was done via scoring by Normal Cross Correlation to see essentially see which displacements would align best when measured by essentially a matrix dot product. I also cropped the edges of the images before measuring as they were not reflective of the true image. I found that the more cropping, the better results.
I was pretty satisfied and so then I tried to apply this to the rest of the images. But then I found a few problems. The window of displacement [-15, 15] I had initially was not big enough for the tif images. When I widened the window, it also took much more time. So this is when I implemented image pyramid. I scaled down the tif files by half multiple times, applying the normal NCC to the smallest image, and adjusting the calculated shift by displacements of [-2, 2]. At one point, the displacement of [-15, 15] was not large enough for the smallest melons picture, so I changed the displacement to (-50, 50].
The smaller images were more blurry and possibly less aligned because of their limited number of pixels. For example, it wouldn't be possible to shift the pixels by a half, unless new pixels were computed to predict the new pixels between the existing pixels. This would artificially increase the size of the image and allow for more control in the amount of displacement of the channels.
Another observation is that notably, the emir.tif image failed by a large margin. This is because the emir's dress is very particular, which causes trouble with the alignment method that uses NCC. NCC tries to match darker pixels to darker pixels and lighter pixels to ligher pixels. Notice that the blue channel has the emir's darkness and whiteness inverted from the green and red channels. As a result, the positions are very off.
In order to resolve the problem of the emir image, I instead use the Canny edge detector. I use the edge detector to detect the edges, with a dot product without normalization to maximize (because there is no benefit for adjusting for brightness from the result of edge detectors). I still use cropping to decrease error and the image pyramid technique for speed.
The following pictures are the results of the first algorithm. Listed below each image is the displacement (rows, columns) of the green and red channels when aligning to blue. The improved algorithm based off of edge detection did not yield significantly better results for any photo besides emir.tif.
