Images of the Russian Empire:
The Prokudin-Gorskii Photo Collection (Colorized)

bernie wang | cs194-26-afg | fall 2018

Overview

As early as 1907, Sergei Mikhailovich Prokudin-Gorskii was given permission to capture scenes using red, green, and blue filters that can form color images when aligned. This project aims at automatically aligning, cropping, and adjusting the color of three 3 separate grayscale images that are supposed to represent the red, green, and blue color channels.

Single-scale Implementation

The single-scale implementation only works for smaller images (jpg files). The algorithm works as follows:

Auto-crop all channels.
Use the blue channel as the base channel to align the red and green channels with. Search for the best shift in a [-15, 15] search window for both axes using a metric to compute how well aligned the shifted image is to the base image.
Align the red and green channels to the blue channel, then crop again to get rid of the colored borders that result from overlaying the channels.
Adjust the contrast.

Cathedral unprocessed. (390 × 341)

Cathedral aligned, no auto cropping, and no auto contrast.

Multi-scale Pyramid Implementation

Since the single-scale implementation runs very slowly for larger images like the tif files (~10^6 pixels), an image pyramid is need to find the displacement efficiently. Like the single-scale implementation, we do a limited search at different levels of the image pyramid. The dimensions of the nth layer if one-half of those of the n-1th layer, and the size of the search window is incremented when you move down a layer. The search is called recursively until we end up with roughly a 2 x 2 image, and find the best alignment. We then "improve" the guess going up the recursive call, looking at neighboring displacements about the displacement vector x 2 (since the dimensions have doubled going up a level).

This algorithm replaces step 2 of the single-scale implementation; everthing else stays the same.

Village unprocessed. (3819 × 3270)

Village aligned, no auto cropping, and no auto contrast.

Bells & Whistles

Automatic Cropping

At first, I implemented the cropping feature by finding the edge of the white border, then finding the edge of the black border. However, it was not robust enough, so I then used the canny edge detector to find the border lines. Using a mask to detect a vertical border edge, I compute the cross-correlation for the 10% leftmost rows, and using statistical thresholding, find the rightmost border, even when there are multiple border present. In order to crop all four border only using this function, I rotated the same image 3 times cropping each left border.

Train unprocessed.

Train aligned, cropped, but not contrast adjusted.

However, since borders are detected using thresholding, this implementation of automatic cropping has trouble cropping when there is no border. Lady.tif is an example where the auto cropping cropped too much because there was no border:

Lady unprocessed.

Lady aligned, cropped, but not contrast adjusted.

Automatic Contrasting

Automatic contrasting was implemented by constructing a cumulative histogram, ignoring the bottom and upper 1% of the pixels, and stretching the middle 98% of the pixels to span the entire 8-bit range represented.

Icon unprocessed.

Icon aligned, cropped, and contrast adjusted.

Better Features

Instead of using normalized cross correlation of pixel value, I computed the normalized cross correlation of the absolute value of the images' gradient for the similarily metric. This worked particularly well for emir.tiff, since the pixel values in the area of the blue clothing is different for each color channel. This metric works well because it is independent of color.
Gradients:

Gradient of red channel of emir along axis 1.

Gradient of green channel of emir along axis 1.

Gradient of blue channel of emir along axis 1.

Emir, aligned using NCC and cropped.
r: [90, -317], g: [49, 24]

Emir, aligned using NCC on normalized gradients, cropped, and contrast adjusted.
r: [107, 40], g: [93, 8]

Example Images

Cathedral unprocessed.

Cathedral aligned, cropped, and contrast adjusted.
r: [12, 3], g: [5, 2]

Emir unprocessed.

Emir aligned, cropped, and contrast adjusted.
r: [107, 40] g: [49, 24]

Harvesters unprocessed.

Harvesters aligned, cropped, and contrast adjusted.
r: [124, 11], g: [60, 17]

Icon unprocessed.

Icon aligned, cropped, and contrast adjusted.
r: [90, 23], g: [41, 17]

Lady unprocessed.

Lady aligned, cropped, and contrast adjusted.
r: [120, 13], g: [57, 9]

Monastery unprocessed.

Monastery aligned, cropped, and contrast adjusted.
r: [3, 2], g:[-3, 2]]

Nativity unprocessed.

Nativity aligned, cropped, and contrast adjusted.
r: [7 0], g: [3, 1]

Self portrait unprocessed.

Self portrait aligned, cropped, and contrast adjusted.
r: [175, 37], [79, 30]

Settlers unprocessed.

Settlers aligned, cropped, and contrast adjusted.
r: [14, -1], g: [7, 0]

Three generations unprocessed.

Three generations aligned, cropped, and contrast adjusted.
r: [111, 9], g:[54, 12]

Train unprocessed.

Train aligned, cropped, and contrast adjusted.
r: [85, 29], g:[41, 0]

Turkmen unprocessed.

Turkmen aligned, cropped, and contrast adjusted.
r: [117, 28], g: [57, 22]

Village unprocessed.

Village aligned, cropped, and contrast adjusted.
r: [137, 21], g: [64, 10]

Additional Images

Kafedra unprocessed.

Kafedra aligned, cropped, and contrast adjusted.
r: [135, -5], g: [62, 0]

Crew unprocessed.

Crew aligned, cropped, and contrast adjusted.
r: [57, 30], g: [20, 19]

Drawbridge unprocessed.

Drawbridge aligned, cropped, and contrast adjusted.
r: [77, -38], g: [26, -18]

Lynx unprocessed.

Lynx aligned, cropped, and contrast adjusted.
r: [129, 27], g: [58, 21]

Lilacs unprocessed.

Lilacs aligned, cropped, and contrast adjusted.
r: [96, -24], g: [49, -6]