Background

Sergei Mikhailovich Prokudin-Gorskii (1863-1944) [Сергей Михайлович Прокудин-Горский, to his Russian friends] was a man well ahead of his time. Convinced, as early as 1907, that color photography was the wave of the future, he won Tzar's special permission to travel across the vast Russian Empire and take color photographs of everything he saw including the only color portrait of Leo Tolstoy. And he really photographed everything: people, buildings, landscapes, railroads, bridges... thousands of color pictures! His idea was simple: record three exposures of every scene onto a glass plate using a red, a green, and a blue filter. Never mind that there was no way to print color photographs until much later -- he envisioned special projectors to be installed in "multimedia" classrooms all across Russia where the children would be able to learn about their vast country. Alas, his plans never materialized: he left Russia in 1918, right after the revolution, never to return again. Luckily, his RGB glass plate negatives, capturing the last years of the Russian Empire, survived and were purchased in 1948 by the Library of Congress. The LoC has recently digitized the negatives and made them available on-line.

Overview

The goal of this project is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, you will need to extract the three color channel images, place them on top of each other, and align them so that they form a single RGB color image. A cool explanation on how the Library of Congress created the color images on their site is available here.

Exahustive Search

Exhaustive Search is a method to align 2 images where we exhaustively search over a window of possible displacements. In particular, we shift the image horizontally by x amount, vertically by y amount, and compare it with the reference image to see if they match using some image matching metric. In this project, we use Exhaustive Search to align green image with blue image, red image with blue image, and stack aligned green, aligned red, blue images together to create a colorful image.

I apply exhaustive search method with a window of [-15, 15] for all .jpg images, and use normalized cross correlation. In addition, I also crop each side of the image by 10% to remove black border before applying search in order to avoid taking the border into account when calculating the metrics. I found that cropping image greatly improves the image quality upon stacking.

Cathedral
green: [2, 5] red: [3, 12]
Tolbolsk
green: [3, 3] red: [3, 6]
Monastery
green: [2, -3] red: [2, 3]

Pyramid Search

An image pyramid is a multi-level image stack, which contains the same image but with different resolutions. The top-level image is the one with the lowest resolution (small enough to run an exhaustive search, as we did in Exhaustive Search). We start from the top-level image and go down the pyramid (the closer an image to the bottom is, the larger its resolution is), updating our estimate as we go. Pyramid Search is prefer over Exhaustive Search for larger images because we need to use smaller window size.

Similar to Exhaustive Search, I also crop each side of all images by 10%. I used a window size of [-5, 5] and normalized cross correlation for matching metric. First, I exhausively search the scaled down by 32 version of the image to obtain the optimal x32, y32. I then move up to the scaled down by 16 version of the image to optain the optimal x16 = x32 * 2 + x and y16 * 2 + y where x and y are in range [-5, 5]. I continue this procedure until the image is of original resolution.

Note that in the case of the Emir of Bukhara (show on right), the images to be matched do not actually have the same brightness values (they are different color channels), so we might have to use a cleverer metric, or different features than the raw pixels. In this project, I just simply use green as base image as opposed to blue like all other images.

Harvesters
green: [16, 59] red: [13, 123]
Icon
green: [17, 41] red: [23, 89]
Train
green: [5, 42] red: [32, 87]
Lady
green: [9, 51] red: [11, 112]
Melons
green: [10, 81] red: [13, 178]
Onion Church
green: [26, 51] red: [36, 108]
Three Generations
green: [14, 53] red: [11, 112]
Church
green: [4, 25] red: [-4, 58]
Self Potrait
green: [29, 78] red: [37, 176]
Workshop
green: [0, 53] red: [-12, 105]
Emir
blue: [24, 49] red: [-249, 95]

My Favorite Images

Painting 1
green: [-4, 7] red: [6, 112]
Painting 2
green: [5, 11] red: [-10, 58]
Painting 3
green: [-2, 3] red: [72, 17]