# CS 194: Colorizing the Prokudin-Gorskii photo collection

## Background

Sergei Mikhailovich Prokudin-Gorskii (1863-1944) won Tzar's special permission to travel across the vast Russian Empire and take color photographs of everything he saw including the only color portrait of Leo Tolstoy. He recorded three exposures of every scene onto a glass plate using a red, a green, and a blue filter. Never mind that there was no way to print color photographs until much later -- he envisioned special projectors to be installed in "multimedia" classrooms all across Russia where the children would be able to learn about their vast country. Alas, his plans never materialized: he left Russia in 1918, right after the revolution, never to return again. Luckily, his RGB glass plate negatives, capturing the last years of the Russian Empire, survived and were purchased in 1948 by the Library of Congress. The LoC has recently digitized the negatives and made them available on-line.

## Overview

In the project, we need to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, we will need to extract the three color channel images, place them on top of each other, and align them so that they form a single RGB color image.

## Naive Appraoch

My program takes a glass plate image as input and produces a single color image as output. To accomplish that, I first divided the image into three equal parts which corresponds to three color channels and then align the red second and the third parts (Green and Red) to the first (Blue), then align Red to green int the end. For a jpg image, to align the parts, I exhaustively search over a window of [-15, 15] pixels displacements, score each one using Sum of Squared Differences (SSD) and normalized cross-correlation (NCC) metric, then take the displacement with the best score. I manually cropped the resulting images in the end for the belowing result secion.

## Image pyramid Approach

For high resolution images, Exhaustive search will become expensive and slow because the pixel displacement will be large. Therefore, I implemented image pyramid alignment algorithm, where I represented the image at scales that are powers of 2 and processed the image at each level. Starting from the coarsest scale (smallest image), I used an adjusted window size to exhastively search for the displacement then move the image based on the rescaled displacement. The moved image then becomes the input of next level of the pyramid. It speeds up the proess and outputs aligned images.

## Auto Crop

Since the edge of the image is white then black, I calculated the mean of each column and each row then set a threthold to identify if the column or row is near the white color(255) or near the black color(0), then crop the image up to that index. I cropped the original image before split into three channels then cropped again before it enters the pyramid search. It works well with the low resolution images, for the high resolution images, some images are overcropped and some still have thin boundary on, because the black boundary is often not pure black but contains some color. Therefore, although I try to find a good threshold value, the result is not perfect.

## Auto Contrast

I used cv2 CLAHE to auto contrast the image. Auto Contrast works well with the below pictures.

## Auto White Balance

I used grey world assumption to estimate the illumination color cast by looking at the average color and comparing it to gray. It works well with the images below.

## More Feature

For the pyramid method, I found out that I could not successfully align the "emir" image, so I used roberts to align the edge of the images, it works well and also improved the quality of some other images as well.