CS 194-26 Project 1: Image Manipulation and Computational Photography

Images of the Russian Empire:
Colorizing the Prokudin-Gorskii photo collection

Jerry Zhang

Background

Sergei Mikhailovich Prokudin-Gorskii (1863-1944) was a man well ahead of his time. As he traveled across the vast Russian Empire, he took thousands of colored photos starting as early as 1907, including the only color portrait of Leo Tolstoy. His idea was simple: record three exposures of every scene onto a glass plate using a red, a green, and a blue filter. However, there was no way to print color photographs until much later; he left Russia in 1918 prior to his full vision of being able to display the images ever came true. His RGB glass plate negatives, capturing the last years of the Russian Empire, survived the revolution and were purchased in 1948 by the Library of Congress and later digitized. The goal of this project is to take the digitized Prokudin-Gorskii glass plate images and automatically produce a color image with as few visual artifacts as possible.

Exhaustive Search Approach

For the lower resolution images, I used an exhausitve search approach, comparing two color channels at a time at every possible displacement in a [-8, 8] window (in both the horizontal and vertical directions). I scored each possible alignment using the L2 norm metric, also known as the Sum of Squared Differences (SSD), attempting to minimize this value. While experimenting with this method, I realized that the borders introduced significant amounts of noise into this calculation, so I cropped 1/4 along each side, essentially comparing only the middle 50% of each color channel every time. Additionally while I initially tried to align the red and the green channels to the blue channel, I found better results from aligning red and blue channels to the green.

After doing this exhaustive search, I also implemented border removal techniques, searching each color from the edge towards the center for lines that had an average value far greater than or far lower than the average value of all the pixels in the image, or a standard deviation too far from that of the entire image.

Another one of the bells and whistles I implemented was contrast adjustment, utilizing skimage's exposure.equalize_hist method on each color channel. This method transforms the histogram (per color channel) of the image into one with a flat histogram, using the CDF (or accumulated normalized histogram) of the image.

Using this exhaustive search approach with L2 norm and additional optimizations, I was able to produce the following images:

Low Resolution Images - Exhaustive Search
Image Name	Aligned	Aligned with Bells and Whistles	Details
monastery.jpg			Red: [6, 1] Blue: [3, -2] Time: 0.311 sec
tobolsk.jpg			Red: [4, 1] Blue: [-3, -3] Time: 0.298 sec
cathedral.jpg			Red: [7, 1] Blue: [-5, -2] Time: 0.303 sec

Image Pyramid Approach

For larger images, exhaustive search became prohibitively expensive because the pixel displacements were too large for most cases. Searching a small window would not produce good results, and searching a large window would take too long. This image pyramid approach aligns high resolution images by repeatedly downscaling images until they are small enough to run the exhaustive search approach, then scaling the offset found from exhaustive search and fine tuning it on a higher resolution image. This recursive approach speeds up the process and allows the search space to remain small. The parameters I used were:

Maximum pyramid level: 15
Pyramid Scale Factor: 1/2
Exhaustive search window size: [-7, 7]
Maximum image size to run exhaustive search: 400px wide

For border removal, I used the following parameters:

Maximum row / column deviation from average: 0.1
Maximum row / column deviation from standard deviation: 0.2

Using this approach, I generated the following images:

High Resolution Images - Image Pyramid
Image Name	Aligned	Aligned with Bells and Whistles	Details
workshop.tif			Red: [52, -11] Blue: [-53, 1] Time: 36.07 sec
emir.tif			Red: [57, 17] Blue: [-48, -24] Time: 33.61 sec
church.tif			Red: [33, -8] Blue: [-25, -4] Time: 45.29 sec
three_generations.tif			Red: [59, -3] Blue: [-52, -14] Time: 37.57 sec
melons.tif			Red: [96, 3] Blue: [-82, -10] Time: 32.88 sec
onion_church.tif			Red: [57, 10] Blue: [-51, -27] Time: 32.81 sec
train.tif			Red: [43, 26] Blue: [-43, -6] Time: 33.31 sec
icon.tif			Red: [49, 5] Blue: [-41, -17] Time: 28.72 sec
self_portrait.tif			Red: [98, 8] Blue: [-78, -29] Time: 33.96 sec
harvesters.tif			Red: [65, -3] Blue: [-59, -17] Time: 29.44 sec
lady.tif			Red: [61, 4] Blue: [-55, -8] Time: 31.16 sec