CS 194-26 Project 1: Images of the Russian Empire

Sergei Mikhailovich Prokudin-Gorskii (1863-1944) traveled across the vast Russian Empire and took thousands of color photographs. Prokudin-Gorskii recorded three exposures of every scene onto a glass plate using a red, a green, and a blue filter. The goal here is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. The difficulty here is that RGB channels on the plates do not immediately align togther, and due to the age of the negatives, there are significant artifacts and corroding of the edges that makes the alignment process harder.

Approach

Small Images

The program reads the entire glass plate image, divides it's height by three, and splits it into thirds, separating the red, blue, and green channels. The image is also converted to a float so that all input images are on the same scale. Next, the red and green channels are aligned to the blue channel by exhaustively searching over a [-15, 15] pixel window of possible displacements and computing the normalized correlation coefficient (NCC) for each displacement, taking the displacement that produced the best score. To counteract the corroded edges and displacements, which negatively influence the alignment, and speed computation, 20% on each side of the images are removed before computing the NCC metric.

Large Images

Because of the dramatic increase in the number of pixels in the larger images, exhaustive search over potential displacements becomes prohibitively expensive because of the window size that must be searched over. In this case, for images with more than 500 pixels in either direction, the image pyramid is used to compute the displacement. The image pyramid algorithm recursively resizes the image to be half the current size until it is less than 500 pixels in both directions. On this smaller image, a full exhautive search over a window of [-15, 15] pixels displacements is performed. When this displacement is returned to the calling function, it is doubled and used as an initial displacement at the higher level. At this level, an exhautive search over a window of [-8, 8] pixel displacements is performed on the larger image, and this displacement is added to the initial displacement. This continues all the way up the image pyramid until the original image is reached (i.e. the initial displacement from the lower level is doubled and fine grain displacement is performed at the higher level to correct inaccuracies in the scaled displacements).

Results

cathedral r = [3, 12], g = [2, 5]	church r = [-4, 58], g = [4, 25]	emir r = [59, 107], g = [24, 49]
harvesters r = [14, 123], g = [17, 59]	icon r = [23, 89], g = [17, 40]	lady r = [12, 116], g = [9, 54]
melons r = [13, 176], g = [10, 81]	monastery r = [2, 3], g = [2, -3]	onion church r = [37, 108], g = [27, 51]
self portrait r = [37, 176], g = [29, 78]	three generations r = [12, 111], g = [14, 52]	tobolsk r = [3, 7], g = [3, 3]
train r = [32, 87], g = [6, 43]	workshop r = [-12, 105], g = [-1, 53]

Other Examples

Here are three additional images from the Prokudin-Groskii collection that were colorized with my algorithm (with and without Bells and Whistles)

Without Bells and Whistles

With Bells and Whistles

Bells and Whistles

Note that the images shown below have all the bells and whistles described in the sections before the one they are in.

Edge Detection

The sobel edge detector is used here to extract the salient edges in each of the channels, and those edges are used to align the images. The image that motivated this was emir. In that case, because the intensity of the red channel versus the green channel and especially the blue channel is very low, the red channel does not align well. Here, we can see a significant improvement.

Auto Cropping

First, 7% on each side of the image is cut to approximately remove the left and right white borders as well as the cross-over from the bottom and top images on the plates. Next, to remove the displaced red and green channels that cross into the borders because of the displacement, pixels equal to the maximum absolute x-displacement and y-displacecment are removed from the left and right, and from the top and bottom of the image respectively. This method does crop off more than is desirable occasionally.

workshop: without auto cropping	workshop: with auto cropping
train: without auto cropping	train: with auto cropping

Auto White Balancing

The scenes are white balanced with a grey world assumption. In other words, the pixels values are scaled such that the average pixel value is grey (with values outside of the allowed range thresholded to black or white if the value is too low or too high respectively). In general, this produced images with colors that appear more realistic.

church: without auto white balance	church: with auto white balance
harvesters: without auto white balance	harvesters: with auto white balance

Auto Contrast

The scences are contrasted using skimage's equalize_adapthist method. This method performs a limited histogram equalization (the amount of contrast is limited by a paramter) over localized regions of the image. In general, this also produced images with more realistic and striking colors, much more so than just from white balancing.

icon: without auto contrasting	icon: with auto contrasting
lady: without auto contrasting	lady: with auto contrasting
onion church: without auto contrasting	onion church: with auto contrasting
workshop: without auto contrasting	workshop: with auto contrasting
emir: without auto contrasting	emir: with auto contrasting