Colorizing Photos: Images of the Russian Empire

Nancy Li

Overview

Sergei Mikhailovich Prokudin-Gorskii (1863-1944) traveled the Russian empire to photograph people and landscapes, inventing a method to produce colored photos well before the world figured out how to print them. His idea was to record three different exposures of every scene onto a glass plate, and each exposure would be taken using a different filter over the camera lens: red, green, and blue. He even envisioned special projectors in children's classrooms that could project all three exposures on top of one another to show the colorized photo.

Now that technology has come a far way, we can turn Prokudin-Gorskii's monochromatic images of the Russian empire to color using image processing techniques!

Approach

Each photo is separated into its R, G, and B channel images, and needs to be aligned for it to be colored properly. Since an image is represented as a matrix of pixel values, the normalized cross correlation (NCC) or sum of squared differences (SSD) can be used to to calculate the best alignment between the three R, G, and B image matrices. To find the best alignment position, NCC scores are calculated within the shift interval of [-15, 15]. High correlation generally means that the brightness levels of both images are aligned with each other, and so finding the highest NCC score finds the best alignment.

Multiscale Approach

The simple search described above works well for small images under 400x400 pixels, but doesn't work as well for larger images. The search window for alignment is much bigger and it takes far too long for photos as big as 3000x3000 pixels. To avoid this problem, an image pyramid was used: big images are scaled by one-half until it reaches a tenth of its original dimensions, producing a range of images varying in size.

The alignment calculation function is called on each image, and the estimate is applied to the next largest scaled versions of the image. Alignment is calculated again, so at each step of the process the image is shifted by [2*previous x + current x, 2*previous y + current y]. This results in the final full-scale image being correctly aligned while the search window stays at a small constant window, [-15, 15]. However, even with image pyramid applied, large images initially took longer than 2 minutes to align. To solve this problem, the scope of the NCC calculation was narrowed to a smaller center fraction of the original image (to a window of a fifth of the image). Now the alignment and colorization takes ~30 seconds for each large image, and a few seconds for each small image.

Additionally, the Emir of Bhukara image caused a lot of alignment problems (the brightness values of pixels in the blue channel is very different from the green and red). Aligning blue/red to green, and green/blue to red channels was tested, and it was found that aligning blue/red channels to the green worked best--not only for emir.tif but for the other images in the collection as well.

Below are the results and displacement vectors!