Colorizing the Prokudin-Gorskii Photo Collection

CS 194-26 Computational Photography Fall 2018

Guowei Yang  cs194-26-acg 

It's astonishing amazing that Sergei Mikhailovich Prokudin-Gorskii (1863-1944) envisioned that someday in the future, technology will eventually able to process his photography, taken by a black-and-white image camera using three filters (red, blue, green), and reproduce the images of the Russian Empire.


In this project, I will automate the process to colorize the grey-scale images taken by Sergei and add some extra feature to the process, including white balance, image cropping as well as auto contrast adjusting.






According to Wikipedia, "Sergei Mikhailovich Prokudin-Gorskii was a Russian chemist and photographer. He is best known for his pioneering work in colour photography and his effort to document early 20th-century Russia. Prokudin-Gorsky traveled the Russian Empire from around 1909 to 1915 using his three-image colour photography to record its many aspects. While some of his negatives were lost, the majority ended up in the U.S. Library of Congress after his death. Starting in 2000, the negatives were digitised and the colour triples for each subject digitally combined to produce hundreds of high-quality colour images of Russia."


These negatives will serve as our test inputs and we will eventually discover the power of modern image processing technologies.

Context

Process


We are provided the negatives images from U.S. Library of Congress, which each consists of three images:

1) image with blue filter, 2) image with green filter, 3) image with red filter


(1) Blue filtered

(2) Green filtered

(2) Blue filtered

Feed In Image


In order to reproduce the colorized version of the image, we have to understand how computer "recognizes" colored photos. The program (in our case, Python) reads in an image as a 3-dimensional matrix, where the first two dimensions are height and width, and the third dimension is the "color" dimension, where 0 is for red, 1 is green, 2 is blue. We first read in the entire image, and seperat them into the respective color dimensions so that we obtain three 2-dimensional matrices, each corresponding to RGB, and then stack them to another 3D image, Python will automatically color them with respect to the color intensity. However, since each sub-picture may have small offsets due to small camera movements, we have to perform another operation, which is alignment. Otherwise, we would get something like this:




Alignment



In order to align all three color channels, I set the green layer to the "base layer", and find the offset of red and blue layers. With this offset (x, y), I could easily perform a translation operation on each layer so that red and blue align with green layer. To determine the offset, we use a metric that determines "how far" the layers are apart from each other, which is sum of squared differences (SSD). Since the same object in each layer have similar pixel intensity patterns, therefore if two layers are aligned properly, the pixel at the same location in this two layers should have a minimal pixel intensity difference. In this case, minimizing the SSD of the entire image should give us the optimal offset value.


Properly determined offsets could produce all these astonishing pictures:

Enhanced Alignment - Image Pyramid



The naive version of the alignment algorithm exhaustively searches in a window of [-15, 15] pixels when the input size is compressed to be around 400 x 400 pixels. The actual image we got is about 3000 x 3000 pixels, meaning the search window has to be scaled up as well to reach [-120, 120], and the computation of SSD scales up in polynomial time as well. Therefore we have to find a quicker way to optimize the code.



In fact, there is a pyramid algorithm which uses a divide and conquer algorithm that efficiently limits the search space.  An image pyramid represents the image at multiple scales (in our case a factor of 2) and the processing is done sequentially starting from the coarsest scale  and going down the pyramid, updating your estimate as we go deeper. Since we can use the naive version to calculate the most coarsest scale, that limits our search space greatly

Failure Cases


The alignment process worked for all images except this one:




No matter how I tuned the parameters, the red layer could not align with the rest of the layers. (If you look closely, the blue and green layers aligned perfectly). The reason for this is because blue layer is too dominant in this picture, which results in a weak intensity in the red layer. Therefore more than one offset values could minimize SSD.


In order to overcome it, I decided to use edge detection and use that information to align pictures because edge detection removes all colors, leaving only lines.  I used the roberts function from skimage.filters


Final Results


cathedral.jpg

Red Layer: (x = 1, y = 7)

Blue Layer: (x = -2, y = -5)

monastery.jpg

Red Layer: (x = 1, y = 6)

Blue Layer: (x = -2, y = -3)

nativity.jpg

Red Layer: (x = -1, y = 4)

Blue Layer: (x = -1, y = -3)

settlers.jpg

Red Layer: (x = 0, y = -7)

Blue Layer: (x = -1, y = 8)

emir.tif


Red Layer: (x = 17, y = -7)

Blue Layer: (x = -24, y = -49)

harvesters.tif

Red Layer: (x = -3, y = 64)

Blue Layer: (x = -18, y = -59)

icon.tif


Red Layer: (x = 018, y = -41)

Blue Layer: (x = 5, y = 49)

lady.tif

Red Layer: (x = -7, y = -53)

Blue Layer: (x = 3, y = 62)

self_portrait.tif

Red Layer: (x = -29, y = -78)

Blue Layer: (x = 8, y = 98)

three_generations.tif

Red Layer: (x = 0, y = -7)

Blue Layer: (x = -1, y = 8)

train.tif

Red Layer: (x = 17, y = -7)

Blue Layer: (x = -24, y = -49)

turkmen.tif

Red Layer: (x = -3, y = 64)

Blue Layer: (x = -18, y = -59)

village.tif


Red Layer: (x = -13, y = -64)

Blue Layer: (x = 10, y = 73)

Extra Photos

church.tif

Red Layer: (x = -19, y = -16)

Blue Layer: (x = 11, y = 29)

rail.tif

Red Layer: (x = -19, y = -16)

Blue Layer: (x = 11, y = 29)

building.tif

Red Layer: (x = -19, y = -20)

Blue Layer: (x = 10, y = 36)

family.tif

Red Layer: (x = -19, y = -16)

Blue Layer: (x = 11, y = 29)

Bells & Whistles

Auto Crop

Auto Contrast

Since aligning will create those useless boarders, it's better to crop them. The simple algorithm removes the border from the image

Utilizes OpenCV's CLAHE (Contrast Limited Adaptive Histogram Equalization) method to automatically adjust the contrast

Edge Detection


An alternative way to align images is using the edges as the feature, instead of pixel intensity. I used the roberts function from skimage.filters to produce the edge-ed image


python main.py <IMG_PATH> [-hdr] [-bal] [-crop]

How to Run the Program