Overview

Sergei Mikhailovich Prokudin-Gorskii was one of the first pioneers for color photography. In the early 1900s, Prokudin-Gorskii traveled across the Russian Empire documenting the people, places, and things he'd seen. Although there was no way to create color photos at the time, he envisioned his photos someday making it into color print. With this dream in mind, he recorded three exposures of every photo onto a glass plate using red, green, and blue filters. He hoped that his images would eventually find its way into classrooms to help educate schoolchildren about the diversity and history of their home country.

Though Prokudin-Gorskii left Russia after the revolution and never got to see his negatives transform to color photos, many of his triple negative glass plates remain behind. With the advent of digitalized image processing, we can now recreate these color scenes from his RGB glass plates. To do this, we extract the three color channel images, layer them on top of each other, and align them so that they form a seamless RGB image.

Naive Implementation

The naive approach for aligning these images involves calculating the Sum of Squared Differences on shifted positions of the frame to find the best displacement possible. This approach works well for smaller images. First, I divide the triple negatives into thirds to extract the different color channels, then layer the red, green, and blue channels on top of one another. This creates a color image, but the resulting image is blurry and misaligned. To correct this, I search for the best displacements to offset the channel layers at such that the three exposures are aligned. The Sum of Squared Differences (SSD) algorithm is helpful for this, as it helps with figuring out how different two images are by calculating the square of the difference between two pixels on two corresponding channel layers, summed across all of the pixels of the two layers. As I search for the best alignment, I hold the blue layer static and find the best displacements for the red and green layers in relation to the blue layer by rolling the R and G channels (separately) across the B channel, one row and column at a time, and calculating the SSD of a subset of layered pixels at each step of the roll. When the two layered channels align, their SSD is at its smallest, because the differences between pixels layered on top of each other are at its smallest.

Pyramid Implementation

While the naive implementation works well, using the same approach on larger images would take much longer to finish. To speed up the search, I use the same algorithm implementation as before, but take a pyramid approach where I first create an image pyramid to represent the image at multiple scales and then first work with the smaller images up towards the bigger images to build up a rough estimate of alignment. Starting with the original large image, I create this image pyramid by repeatedly scaling down the image by a factor of 2 until I get the image to a minimum threshold size. The minimum threshold size is set to be around the size of the smaller images in the naive approach, which works quickly and accurately. Now that I have a smaller photo and thus less pixels to work with, the search for the best displacement is again very quick; however, this search for alignment is also not as accurate, as the scaled down image has lost some of its original clarity. Therefore, once I finish the image processing on the smallest level, I continue the search process by building back down the pyramid to update the estimate on larger, clearer image canvases. I run the same algorithm on a subset of (-25, 25) pixels on each larger layer of the pyramid until I reach a point in which the displacements are held relatively static, and I've found the displacements for alignment.

Bells and Whistles

Preprocessed Images: Sobel Edges

Even with the SSD algorithm, some images won't align well based off of the RGB channels alone. For example, in the case of the photo of Emir of Bukhara, we see that the color contrast and brightness values are different in each of the separate exposures. Thus, the robe looks vastly different in each of the color channels and have contrasting pixel values. Since the algorithm believes the best displacement is where the summed square difference is the smallest, it may produce a false negative when calculating the SSD of an object that have high pixel variation across channels. Because the robe is mostly blue with very little red (as seen by the richness of the robe in the B versus the R channel), even when the robe is aligned in the B and R channels, the SSD may not be the smallest when the channels are actually aligned. To solve this problem, I decided to preprocess the images and calculate displacements based off of another metric other than the brightness in individual RGB channels. In this case, I decided to use the edges of objects in the photo to use for alignment. The skimage library has a nice filter, called the Sobel Edge filter, that helps us process the images to emphasize edges.

B, G, R (Top to Bottom)

SSD Pyramid Alignment Only

SSD Pyramid Alignment with Preprocessing of Sobel Edges

To improve the algorithm to account for these kinds of images, I preprocess the three channels so that they are now represented by distinct edges rather than the strength of the colors. After preprocessing these images, I again run the SSD algorithm search based off of where the edges are rather than the RGB intensities, and end up with nicely aligned images.

Sobel Edges in G Channel

Sobel Edges in R Channel

Sobel Edges in B Channel

Alignment on Sobel Edges

Extra Images for Fun

Here is an additional set of images I've chosen to colorize and align from the Prokudin-Gorskii Collection of negatives at the Library of Congress. There are over 1,000 glass negatives in the collection, and these caught my eye.