Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

By Cameron Hui, chui3@berkeley.edu

Overview

This project aims to colorize Sergei Mikhailovich Prokudin-Gorskii's collection of photos taken in the early 1900s. His approach took pictures of the same subject with three cameras, each one with a single color filter on them (red, green, and blue filters). He believed that when all images were combined into one image, they would form a full color image. My job was to take the digital negatives of his images (recorded by the US Library of Congress) and combine the negatives to form a color image with as few artifacts as possible.

My approach

I started with the single scale implementation by dividing the image into its respective RGB channels (each negative). Using a sliding window of [-15,15] pixels, I aligned the green channel over the blue channel, then the red channel over the blue channel. I aligned the images by using the np.roll function to move the channels, then used both the L2 norm (AKA Sum of Squared Differences or SSD) and Normalized Cross-Correlation (NCC) formulas to find the best offsets. Looking at my results, I found that the SSD algorithm worked best for aligning the images.

At first, the images weren't aligned the best, even with SSD. This was because the images were skewed by the black and white borders. I realized I could find the ideal alignments by running the alignment algorithm on the center of the images, which would have no borders and also (hopefully) have enough details to find the most ideal displacements. Then I could displace the full images using those calculated displacements for a fully aligned image. So I did that by cropping the image to the middle 50% of the image (cropping 1/4th of the image on all sides). This worked really well for SSD, but not so well for NCC. Because this method almost perfectly aligned my images, I decided to use SSD going forward into my larger scale implementation.

My understanding of the image pyramid method was that I could recursively run my exhaustive search algorithm on a smaller scaled version of the very large .tif images provided by the Prokudin-Gorskii collection. First, I rewrote my colorize function to return the displacements for the red and green channels, then created a recursive function that would call the colorize function on each layer of the image pyramid. However, each layer of the pyramid would refer to the previous layer's displacement when searching for the ideal displacement within the sliding window.

Challenges

Being able to carry over the displacement numbers on each layer of the image pyramid proved challenging, as I needed to keep track of the ideal alignment displacements on each scale of the image, then recalculate the ideal displacements. I ended up rewriting my exhaustive search method for the pyramid method so I could keep better track of the displacement numbers throughout the whole process. In addition, for the pyramid method, my displacements from previous image layers didn't seem to be taken into account when processing the current layer. I thought this could be fixed by changing my algorithm to be aligned with a different color channel than blue as the base, but all combinations I made with other color channels didn't look as good. I think that in my pyramid method, I may have had an error in how I stored the layer displacements from layer to layer.

Upon looking at my resulting images more closely, I realized that most of the really bad offsets on the images were because the images were not properly aligning on the y axis. I decided to investigate closer on my algorithm to see if there could be any changes I could make to align the images better. I realized during this process that my sliding window values weren't updating properly as I went to each layer - instead of checking a window around the scaled version of the previous displacement, it was checking the basic window from [-15,15]. Fixing the algorithm to refer back to the previous displacements helped improve the image quality, but some images were still not aligned correctly. Finally, I tried aligning my images on the green channel again, which improved the alignments much better for images like 'harvesters' and 'icon', but didn't make much changes to some other images.

For the images that didn't work so well with my final algorithm method, I believe it could be an issue with the parameters that I used to align my images, whether it was the sliding window I used or how much I cropped the images to check alignments.

Comparing methods used in my pyramid algorithm

All the previous methods I discussed in my approach and challenges were tested on the 'harvesters' image. The results are shown with labels below:

harvesters1 — UPDATING SLIDING WINDOWS WITH BLUE BASE

harvesters2 — UPDATING SLIDING WINDOWS WITH GREEN BASE

Based on these results I used the updating sliding windows with green base method on all my images, and found that it worked better than my previous methods but were still inaccurate. Some results were better with the previous methods, so the results where other methods are better are compared below.

Results

Exhaustive Search

Here are the L2 images with the 50% crop alignment, and the NCC algorithm right after for comparison.

cathedral l2 — L2 Green offset: (5, 2), Red offset: (12, 3)

cathedral ncc — NCC Green offset: (-15, -13), Red offset: (-15, -15)

monoastery l2 — L2 Green offset: (-3, 2), Red offset: (3, 2)

monoastery ncc — NCC Green offset: (-8, 7), Red offset: (10, 8)

tobolsk l2 — L2 Green offset: (3, 3), Red offset: (7, 3)

tobolsk ncc — NCC Green offset: (12, -3), Red offset: (4, 9)

Pyramid Search

Here are the results for the Pyramid algorithm I made. Some results were better than others depending on the algorithm I used. I have noted which method I used for the ones where my previous algorithms worked better.

emir — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-5, -6), Red offset: (54, 37)

church — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (7, 4), Red offset: (0, -5)

UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-22, -8), Red offset: (45, 23)

lady — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-29, -4), Red offset: (31, 2)

melons — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-29, -4), Red offset: (31, 2)

onion church — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-26, -11), Red offset: (26, 5)

self portrait — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-40, -12), Red offset: (45, 3)

three generations — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-27, -6), Red offset: (0, 0)

train — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-4, -37), Red offset: (58, -44)

workshop — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-28, 1), Red offset: (26, -7)

On other images

I chose to test my blue based L2 pyramid algorithm on 3 more images from the Prokudin-Gorskii collection: The school, Church of the Transfiguration, and Ostrechiny. The results are shown below with their respective offsets. NOTE: Some results worked better with different methods (base images, sliding windows, etc). I have noted which method I used for each below.

transfiguration — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-34, -8), Red offset: (38, 4)

ostrechiny — UPDATING SLIDING WINDOWS WITH GREEN BASE Blue offset: (-8, 2), Red offset: (59, -4)

Bells and Whistles

I really wanted to but uhhhh I was farming for Raiden Shogun oopsies