CS 194-26 Project 1

Images of the Russian Empire: Colorizing
The Prokudin-Gorskii Photo Collection

Shreyas Krishnaswamy


Between approximately 1909 and 1915, Sergey Prokudin-Gorskii traveled around the Russian Empire, capturing black and white images with colored lenses. He documented scenes ranging from landscapes to villages to castles. Here, I design a computer program to automatically align, merge, and adjust the negatives, restoring color to these early 20th century moments.


Lake Negatives
From the top: blue, red,
and green negatives from
the original collection
Lake Final
My colorized version


Overview

The collection features three-image sets where each image stores either the red, green, or blue channels. The primary challenge is aligning the manually captured negatives to avoid blurring. Once aligned, I combine the RGB channels to create a color image. Afterwards, I crop each image's borders and remove residue left behind by the imprecise negatives. Lastly, I use contrast stretching to highlight details and sharpen the picture.


Alignment: Naive Approach

I aligned images by comparing the red and green negatives to the blue negative. I shifted the red and green negatives up, down, right, and left by up to 15 pixels and chose the best matching shift. The strategy worked well with low-resolution images but failed to find sufficient shifts for high-resolution images, causing colored blurring.


Naive Approach: Low Resolution

Cathedral
Monastery
Tobolsk


Comparing Negatives

I tested both Sum of Squared Distances (SSD) and Normalized Cross Correlation (NCC) when comparing images. Ultimately, NCC seemed to align images better, so I stuck with it for both the naive approach and the pyramid speedup. I optimized both approaches by ignoring a percentage of the negatives' borders where there are inconsistent frames and other imperfections. Instead I compared the centers of the negatives where the images were more consistent.


Alignment: Pyramid Speedup

For high-res images, I used pyramid scaling to speedup the alignment. Instead of merely widening the [-15, 15] interval from the naive solution, I rescaled the high-res images by halving their resolution repeatedly until their width or height was under 100 pixels. Then, I used a [-15, 15] interval to find an alignment for the low-res negatives. I then repeatedly scaled back up and shifted around the (scaled) previous alignment to find a better alignment for high-res images without naively widening the interval. At each "scaled-up" shift, I tested a [-5, 5] pixel interval around the previous alignment. Overall, this found effective alignments for each sample image from the collection.


Pyramid Speedup: Issues

Some images were still somewhat off. The harvesters in particular had some blurring around a few of the workers. Since our shifts were purely translational in the x and y directions, any misalignment caused by rotation was left unsolved. Moreover, impurities in the negatives themselves can contribute to inaccuracies in the alignment process.


Pyramid Speedup: Outputs and Shifts

Cathedral: G [5, 2], R [12, 3]
Emir: G [49, 24], R [103, 43]

Harvesters: G [60, 16], R [124, 14]
Icon: G [40, 17], R [89, 23]

Lady: G [55, 8], R [110, 12]
Melons: G [82, 9], R [177, 11]

Monastery: G [-3, 2], R [3, 2]
Onion Church: G [51, 26], R [108, 36]

Self Portrait: G [79, 29], R [175, 34]
Three Generations: G [55, 13], R [112, 10]

Tobolsk: G [3, 2], R [6, 3]
Train: G [47, 6], R [87, 32]

Village: G [65, 12], R [138, 23]
Workshop: G [53, -1], R [101, -12]


Automatic Cropping

Although well-aligned, the pyramid scaling outputs leave border residue from slight mislignment and black borders on the colored negatives themselves. To crop the residue, I followed four steps:

  1. I converted the colored pyramid outputs into grayscale and applied a Gaussian blur.
  2. I convolved the blurred images with a Sobel Filter to find sharp changes in color, i.e. edges.
  3. I thresholded the filtered image so anything above 2 * (mean intesity) would be 1 and anything below it would b 0.
  4. I considered any row or column in the binary image with a certain proportion of 1 values to be a straight edge. I scanned near the borders of the image for straight edges, and once I found the first one, I cropped everything from there to the edge.

Automatic Cropping: Edge Detection

Pyramid Output
Cropped Output

Automatic Cropping: Issues

My automatic cropping has issues with over or undercropping under certain circumstances. Since it considers any straight line near the border of an image to be part of the color residue, straight lines that are actually part of the image can be cropped. For instance, the Emir is overcropped since the door creates a natural vertical line that mimics a border, causing the program to crop it. On the other hand, tilted borders are spilled over multiple rows or columns. Since we only look at individual rows or columns when checking for lines, these titled lines escape dection. For instance, the Prokudin-Gorskii self portrait does not get cropped on the left since it's edge is tilted.





Contrast Stretching

The cropped images still have issues with capturing a wide range of colors. Since the computer can display a finite number of colors and the residue borders can often be lighter or darker than the rest of the image, the pictures may reside in a narrow color spectrum, especially now that the bright borders have been cropped. To widen this spectrum and increase contrast, I implemented contrast stretching. First, I split each cropped image back into the RGB channels. Then, I found the 5th and 95th percentiles on the intensity historgram for each channel and rescaled the overall intensities based on these percentiles. The results were somewhat varying. In images like the Icon, the contrast stretching brought out a sense of realism. However, in images like the Fossil, it seems to add too much brightness and obscure details.

Contrast Stretching: Outputs

Cropped Output
Contrasted Output



More Images from the Collection

Altar: G [20, 16], R [59, 17]
Fossil: G [55, 4], R [107, -48]

Lake: G [58, -9], R [126, -15]
Railroad Participants: G [34, -15], R [122, -27]

Resources

To see more of the Prokudin-Gorskii Photo Collection, please visit https://www.loc.gov/pictures/collection/prok/