Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

COMPSCI 194-26: Computational Photography & Computer Vision

Professors Alyosha Efros & Angjoo Kanazawa

August 26th, 2021

Nick Kisel

Overview

The digitized Prokudin-Gorskii glass plate image collection, hosted by the Library of Congress. While the negatives are shot from almost exactly the same position and angle by design of the camera, the three negatives are each scanned at a slightly different position and rotation, resulting in subtle misalignments of the combined color image. By cutting the image into three equally sized parts, then shifting each negative by some (x, y) offset, recreating a full color image is possible.

Approach

Align & color the three filtered components using blue as the base color.

Three color channel images are extracted from the glass plate negative image by splitting it vertically into three parts of equal length. To align the three channels, I calculate the optimal displacements of the channels along the x and y axis. I searched over displacements in a 30 pixel wide interval on the x & y axes, scored each one using SSD, and aligned the images using the displacement with the lowest score.

Test different alignment strategies that fit the components together most clearly, and ignore borders & blotches.

First, I discarded the outer twelfth of each image to prevent matching the borders and minimizing the effect of markings & dirt on either side. For smaller images with a width less than 400 pixels, I searched a space of 900 square pixels (a 30x30 range) for the best match.

As for larger images, I used an image pyramid to recursively resize and align progressively larger versions of an image, such that each progressively larger image was four times more finely aligned with the original. Each time an image was resized up, it would be re-aligned in accordance with the aforementioned displacement formula over a range of just 100 square pixels (a 10x10 range around the original scale estimate for the rescaled image). The quick and efficient alignment of the smaller versions of the image serve to make aligning each taller & wider resize more accurate.

Additionally, you'll read about an edge-detection strategy that I included to improve alignment accuracy on sharp images!

Results

SSD & NCC matching

This just lines up color channels in 900 different ways and outputs the result of the best match based on colors in the image. It looks great!

Pyramid strategy & examples

The image pyramid strategy I used for most images decreased the scale of the input image by four per iteration, saving the intermediate photos. Starting from the smallest image, the different channels are aligned, and an "absolute offset" on the original image is estimated from the offset obtained from alignment. As a result, the estimate of the absolute offset becomes four times more fine for every alignment of intermediate images.

Unsatisfied with some of the results which produced afterimages - particularly, Lady, where the red channel's slight left shift causes the eyes to look sleepy, I narrowed the search to just the inner ten twelfths of the image such that the borders weren't included in calculating the difference.

Emir, 58x50

Green: (0, 0) Red: (128, 0)

Emir, 231x200

Green: (0, 0) Red: (112, 16)

Emir, 924x801

Green: (-4, 8) Red: (108, 16)

Emir

Green: (-3, 7) Red: (107, 17)

The following steps describe the estimated position of the green & red channels relative to the blue channel as you scale the image up in size. Scale 1 shows the final estimate.

1 / 64
1 / 16
1 / 4
1

(0, 0)
(0, 0)
(-4, 8)
(-3, 7)

(128, 0)
(112, 16)
(108, 16)
(107, 17)

Large Images

Results

The pyramid algorithm works well for most images, coming within a few pixels of an exact match for most objects if not perfectly matching them. There were some evident difficulties with lady, on which the red channel consistently aligned several pixels to the left of the other two channels, and ended up blurring the image around the lady's head. The same can be said for self-portrait, which struggles with similarly-colored details such as trees, rocks, and bushes.

Additional images

Bells and Whistles

Edge Detection

Using skimage.transform, I used the roberts() edge detection algorithm to trace each of the three negatives before comparing them. The edge detection approach works wonders on sharp images with distinct corners, such as train, workshop, onion_church, & emir. Of course, applying this filter has its tradeoffs: smoother images - namely lady - were disrupted and blurred as a result. Some images, like three_generations, include some sharp corners but less-sharp focal points - thus, the three family members are defocused, but the fence is well-aligned.

Of course, applying this filter has its tradeoffs: smoother images - namely lady - were disrupted and blurred as a result. Some images, like three_generations, include some sharp corners but less-sharp focal points - thus, the three family members are defocused, but the fence is well-aligned.

Auto-contrasting

To improve contrast and utilize the whole color spectrum, I added a preprocessing step: skimage.exposure.equalize_hist, which differentiates colors that are particularly prevalent in an image. In particular, this improved the color of the sky and other washed-out whites in most images. Delightfully, in addition to improving the color, this caused self portrait to line up perfectly without edge detection! I suspect this is because a lot of similar colors appear next to each other, and end up neither being classified as edges nor having a large impact on the SSD sum otherwise. Also notice the well-differentiated tree colors in the scenic background.

Beauty is in the eye of the beholder. It's up to you whether this is a welcome change.

Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

Overview

Approach

Results

SSD & NCC matching

Cathedral

Monastery

Tobolsk

Pyramid strategy & examples

Emir, 58x50

Emir, 231x200

Emir, 924x801

Emir

Onion Church

Large Images

Emir

Harvesters

Icon

Melons

Monastery

Lady

Onion Church

Self-portrait

Three generations

Tobolsk

Train

Workshop

Results

Additional images

Dvorets (garden)

Koloch

Kremlin

Bells and Whistles

Edge Detection

Train: Before

Train, edge detection

Emir

Emir, edge detection

Onion Church

Onion Church, edge detection

Lady

Lady, edge detection

Three Generations

Three Generations, edge detection

Auto-contrasting

Self Portrait

Self Portrait, contrast

Train

Train, contrast

Onion Church, edge detection

Onion Church, contrast

Monastery

Monastery, contrast

Church, contrast

Emir, contrast