CS194-26 Project #1
Colorizing the Prokudin-Gorskii Photo Collection

OVERVIEW

Sergei Mikhailovich Prokudin-Gorskii had a clever idea. He decided to photograph everything he saw in three exposures using a red, green, and blue filter. Even though there was no color photography at the time, his glass plate negatives survived and we are now able to use them to produce colorized images from the past.

Goal: Take the Prokudin-Gorskii glass plate images and colorize them via image processing.
How: Divide each digitized glass plate scan into three color channels (R, G, and B). Properly align them on top of each other to get the colored result!


NAIVE ALGORITHM

I used a simple sum of squared differences (SSD) algorithm to find the best alignment over a [-15, 15] window. The best (x, y) displacement value will be the one that minimizes the SSD value. I ignored the outer 10% of each image and instead chose to consider just the inner portion to avoid having the borders throw off the alignment calculations.Initially, I used the B channel as the reference to align the R and G channels, but realized this resulted in a slight discrepancy for emir.tif. After playing around a bit, I realized that using the G channel as the reference worked better (possibly because of the abundance of blue in the photo). Changing the reference channel didn't seem to affect the alignment of any of the other images. This algorithm worked well for the smaller jpg images, but it proved to be much too inefficient to process the bigger tif images quickly enough. Below are the resulting images:

Cathedral
B: (-5, -2)
R: (7, 1)
Monastery
B: (3, -2)
R: (6, 1)
Nativity
B: (-3, -1)
R: (4, -1)
Settlers
B: (-7, 0)
R: (8, -1)

IMAGE PYRAMID

I needed a much more efficient way to align the bigger images, which contained way more pixels than the jpg images used above. I used the image pyramid recursive approach to halve the size of the image until it was a more manageable size, then find the proper displacement at that level using the SSD algorithm mentioned above. This resulting displacement is multiplied by two each time it is scaled back up a level. We search over a window around this doubled displacement to make any adjustments, repeating this until it returns to its original size, each time finding an updated and more precise displacement for each channel. I manually found that a good window to search for the displacement when scaling back up was 7.
Here are the results for the larger tif images:

Emir
B: (-48, -24)
R: (56, 18)
Emir (B as reference)
G: (48, 24)
R: (102, 54)

*Note: When I place the image obtained from using G as the reference channel (left) next to the image obtained when using B as the reference (right), it is clear that using G as the reference channel results in a clearer and better aligned image!


Harvesters
B: (-60, -16)
R: (64, -2)
Icon
B: (-40, -16)
R: (48, 6)
Lady
B: (-50, -8)
R: (62, 4)
Self Portrait
B: (-78, -28)
R: (98, 8)
Three Generations
B: (-54, -14)
R: (58, -2)
Train
B: (-42, -4)
R: (44, 26)
Turkmen
B: (-56, -20)
R: (60, 8)
Village
B: (-64, -12)
R: (72, 10)

And results from a few other photos that I chose from the collection:

Austrian Prisoners
B: (-3, -1)
R: (9, 1)
Barracks
B: (-4, -1)
R: (10, 1)
Naziya
B: (-1, 0)
R: (2, 0)
Tracks
B: (-28, -6)
R: (78, 4)

BELLS & WHISTLES

Automatic Cropping

Instead of just cropping a predefined margin off of the borders, I tried to detect the actual borders and crop accordingly. I used edge detection on each of the channels on the outer 10% of the image and then averaged out the calculated pixel values for each row/column. If the average was above a certain threshold for any of the channels, that region would be cropped from the final image. I found that a good threshold was 0.1 because this minimized the discolored borders while still maintaining most of the image. However, this threshold can be changed depending on how extreme of a cropping you would want. I decided to go with Sobel edge detection because the Canny method seemed to be a bit too sensitive and crop too much of the edges off.
Here is a comparison of the two methods:
Before Cropping
Sobel Method
Canny Method
As you can see, the cropped image using Canny edge detection crops off a significantly larger portion on the left of the image, likely due to the sharp pink edge. Here is a comparison of predefined margin croppings vs. automatic cropping:
Cropping outer 10% of image: Settlers
Auto Crop. Much better!

Automatic Contrast

Sometimes, an image may appear dull and gray due to lack of contrast. This happens when the image intensities do not occupy the full range [0,1] and instead lie somewhere in the middle. Thus, to fix this I rescaled the intensities of the image so that the minimum intensity is at 0 and the maximum intensity is at 1:
Before:
After rescaling (min/max intensities are at 0 and 1):


Lady: Before
Lady: After
Three Generations: Before
Three Generations: After
Icon: Before
Icon: After
Overall, the images seem sharper, more saturated, and more defined.