CS 194 - Project 1

Menu

Colorizing the Prokudin-Gorskii Photo Collection

In this project, I explore coloring the Prokudin-Gorskii photo collection by aligning the black-and-white negatives of photos that were taken with red, green, and blue filters, and various processing methods to make these images look as visually appealing as possible.

Overview

The Prokudin-Gorskii collection is a series of photos that were taken by Sergei Mikhailovich Prokudin-Gorskii on his journey to document the Russian Empire in the 1900's. By taking photos with three exposures through red, green, and blue filters, Prokudin-Gorskii was able to store color information with black-and-white images, since the negative of these images contains the red, green, and blue color information. The following image shows how the channels are stored:

Approach

Aligning the Images

Naively aligning the images

For each image, there are three pictures, where the top, middle, and bottom pictures represent the blue, green, and red channels, respectively. The first step is to get each channel separately from the entire image file, and overlay them on top of each other to form a colored image. Here is the result from overlaying the images:

Aligning images with SSD and NCC

After naively overlaying the image, we see that the image is very blurry, and the borders contain solid streaks of colors. To fix this, we need to put in more work. To start, we will try aligning the images properly. First we try sum of squared distances (SSD):

We then try normalized cross-correlation (NCC):

Using NCC causes the blue channel to be slightly misaligned, and this is especially noticeable on the treetops in the center of the picture. This does not happen as much with SSD, and so we will use SSD to align the images.

Optimizing Runtime

Although the alignment algorithm using SSD works relatively well, larger images took a substantial amount of time to run. To alleviate this issue, we use an image pyramid. In an image pyramid, we scale the image recursively by a factor of 2 until reducing it anymore would result in an image less than 15 pixels wide and 15 pixels tall. When this base case is reached, we then run SSD on the level to find the optimal offset. We then go to the next layer down (which has twice the resolution), multiply the offset we found in the last layer by two, and look in a 15-by-15 area using this offset. By localizing the offsets, we are able to hone in on the optimal offset, and this substantially speeds up our alignment algorithm.

Optimizing Color Alignment

Currently, we have aligned all the red and green channels to the blue channel, however, aligning to other colors may lead to better results. Below, we try aligning to different color channels. First, the red channel:

We then try aligning the red and blue channels to the green channel:

And finally we try aligning the red and green channels to blue:

The differences are subtle, however, aligning the red and blue channels to the green channel appears to have the best results, as the image looks the clearest. In addition, the human eye is most sensitive to colors in the middle of the visible spectrum, so it should make sense that aligning to the green channel should be the best option.

Bells and Whistles

Automatic Cropping

Since the borders of all the images are roughly the same size, it suffices to simply crop a certain amount off of each photo. From testing, it appears that cutting approximately 16% of the width and height of each image suffices. We crop before we align, so that the color borders on the edges of the image do not impact the alignment. Here is the photo before cropping:

And here is the photo after cropping:

As we can see, the color borders are gone, and the blue haze on the treetops are gone.

Automatic Contrasting

To balance contrasting, we run histogram equalization on the image in HSV space. Here is the result after running histogram equalization:

Now, a lot more detail is introduced, and the overall dynamic range is much larger.

Automatic White Balance

To implement automatic white balance, we must use a white-world assumption (i.e. assume the brightest pixel is pure white) or a grey-world assumption (i.e. assume all colors average out to grey). We first try a white-world assumption:

The image appears largely the same, however, there are differences in the white balance if we observe closely enough. This lack of change is likely due to the brightest pixel having a value close to pure white.

Here, we see a substantial difference, as the whites in the picture are completely blown out of proportion. The obvious choice for white-balancing is to use a white-world assumption, and in other images (i.e. in images where the brightest pixel is not close to pure white), the white-balancing is much more noticeable.

Results

Here is the result after running all our processing on the images:

Cathedral
Church
Emir
Harvesters
Icon
Lady

Melons

Monastery
Onion Church
Self Portrait
Three Generations
Tobolsk
Train
Workshop

Additional Images

Here are three additional images from the Library of Congress from the same Prokudin-Gorskii collection that have been aligned and processed.

Bearded Man Portrait
Floodgates
Roses

Offsets

Here are the offsets for the red and blue channel for each image. Each offset is listed in pairs, where the first item is the horizontal offset, and the second item is the vertical offset.

Image Name Red Offset Green Offset
Cathedral (1, 7) (-2, -5)
Church (-8, 33) (-4, -25)
Emir (17, 57) (-24, -49)
Harvesters (-3, 65) (-16, -59)
Icon (5, 48) (-17, -41)
Lady (3, 62) (-9, -49)
Melons (4, 96) (-10, -82)
Monastery (1, 6) (-2, 3)
Onion Church (10, 57) (-26, -51)
Self Portrait (8, 98) (-29, -78)
Three Generations (-3, 58) (-14, -53)
Tobolsk (1, 4) (-2, -3)
Train (27, 43) (-5, -42)
Workshop (-11, 52) (0, -52)
Bearded Man Portrait (7, 48) (-9, -36)
Floodgates (-3, 66) (-16, -56)
Roses (15, 51) (-20, -31)