CS 194-26

Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

Randy Fan



Overview

The purpose of this project is to align digitized glass plates to form colored versions of those images. There were two alignment approaches we were required to implement: single-scale and image pyramid.

Single-Scale Version

The single-scale version simply iterates through a displacement window (I used [-15, 15]) and determines the ideal image displacement in that range by using the SSD function. The SSD (Sum of Squared Differences) function gives us a metric that allows us to score how good the displacement is at matching up one plate with another plate. We want to minimize the result of the SSD in order to improve image matching. For this approach and the one discussed below, I cropped out around 10 percent from each side of the image for preprocessing purposes, which turned out to be quite useful as the borders typically had a lot of noise.

Cathedral. Displacements: Green[2 5] Red[3 12]
Monastery. Displacements: Green[2 -3] Red[2 3]
Tobolsk. Displacements: Green[3 3] Red[3 6]

Image Pyramid

The second approach utilizes an image pyramid, which is a faster search approach that is particularly helpful in aligning large .tiff images. The image pyramid stores the original image plus smaller resolutions/versions of that image. In my implementation of the pyramid, I stored five different resolutions of the same image by recursively rescaling (divide each dimension by two) the original image five times. At every layer, I utilize a difference displacement window, which is determined by [int(15/(5-level)), int(15/(5-level))]. This varying displacement window gives us a more efficient way to determine the ideal displacement of an image because the window essentially shrinks as the resolution goes up and we can update our estimates continually. For example, at the bottom layer (level = 5), we use a displacement window of [-15, 15] and at the top layer (level = 1), we use a displacement window of [-3, 3]. At every layer, we add the previous displacement to the current best displacement. Note that when we update our estimates going up the layers (increasing in resolution), we need to multiply the displacement by two since the resolution dimensions are twice as large. This algorithm worked pretty well for all images except for Emir.tif. This is because the color channels for Emir have varying brightness values, making it difficult for the algorithm to align them properly.



Harvesters. Displacements: Green[16 59] Red[13 123]
Icon. Displacements: Green[17 41] Red[23 89]
Lady. Displacements: Green[9 51] Red[11 112]
Melons. Displacements: Green[10 81] Red[13 178]
Onion Church. Displacements: Green[26 51] Red[36 108]
Self Portrait. Displacements: Green[29 78] Red[37 176]
Three Generations. Displacements: Green[14 53] Red[11 112]
Train. Displacements: Green[5 42] Red[32 87]
Village. Displacements: Green[12 64] Red[22 137]
Workshop. Displacements: Green[0 53] Red[-12 105]

As mentioned previously, Emir.tif was the only image that had issues aligning properly due to varying brightness in its three color channels. Extra credit: I decided to apply Roberts Edge Detection to extract edges from the image, hoping to remove the bias that the brightness was causing. It had great results for Emir.tif, as shown below.

Emir using SSD. Displacements: Green[24 29] Red[-305 93]
Emir using Roberts Edge Detection. Displacements: Green[24 49] Red[40 107]

Result of my algorithm on a few examples of my choice from the Prokudin-Gorskii collection

City of Lodeinoe Pole. Cathedral of Saints Peter and Paul. Displacements: Green[11 26] Red[16 63]
Church of the Savior Not Made by Hands of the Parish of St. Clement in Novaa Ladoga, St. Petersburg Province, Russian Empire. Displacements: Green[24 20] Red[30 31]

Remaining Extra Credit

For extra credit, I implemented automatic cropping, automatic contrasting, automatic white balance, better color mapping, better features (aligning using Roberts Edge Detection, as described above), and aligning and preprocessing data from other sources.

For automatic cropping, I created a while loop that keeps incrementing the amount that needs to be cropped until it breaks a certain condition. My cropping function was image[int(new_height):-1*int(new_height)-1, int(new_width):-1*int(new_width)-1], where the new_width and new_height are width and height multiplied by the specified crop amount (e.g. 0.10). The condition just checks to see if the pixel at the corner has a value greater than or equal to 0.9. I chose this condition because I noticed oftentimes images had border values >= 0.95, and it seemed to work well on the images I tested. The crop amount starts at 0, are increases up to 1.0. Below is an example of automatic cropping being used:

My automatic cropping function chose to set crop_amount to .04
Default of crop_amount = .10

For automatic contrasting, I rescaled the image intensities such that the smallest non-zero intensity is scaled (subtracted) to become zero and the largest non-one intensity is scaled (added) to become one. Initially, I scaled (added) all intensities >= 0.5 by 1 - largest non-one pixel intensity and scaled (subtracted) all intensities < 0.1 by the smallest non-zero pixel. Since the adjustments involved such small numbers, the changes weren't very visually noticeable. To make it slightly more noticeable, I defined another range of intensities [0.6, 0.7] that get its value set to 0.6. This created a small (but noticeable change) as shown belown.

Automatic contrasting applied.
No automatic contrasting applied.

For automatic white balance, I first estimated the illuminant, which I treated as the brightest color. I then multiplied all the intensities by 1/(brightest color). Note I defined brightest color as the largest non-one intensity. The difference is kind of hard to see, but the clouds are slightly whiter with automatic white balancing. Below is a result of this automatic white balancing:

Cathedral with automatic white balance
Cathedral without automatic white balance

For color mapping, I implemented an rgb to hsl converter and a hsl to rgb converter using the instructions described in the following website: http://www.niwa.nu/2013/05/math-behind-colorspace-conversions-rgb-hsl/. I converted the rgb values to hsl. Then, for fun, I increased the Hue to 152 (a greenish color). Finally, I converted hsl back to rgb. Below is a result of this operation for cathedral:

Cathedral with different color mapping

For better features, I applied Roberts Edge Detection on Emir.tif, as described in earlier section.

Here are the results of aligning and preprocessing hst_05397_01_wfpc2_f439w_pc.fits, hst_05397_01_wfpc2_f555w_pc.fits, hst_05397_01_wfpc2_f814w_pc.fots using a linear stretch shifting. I used FITS liberator to convert the images into jpg. After aligning several Hubble images, I could not find a color channel that had a displacement that wasn't all zeros, which I found interesting. Here is the result below:

hst_05397_01_wfpc2_f439w_pc, hst_05397_01_wfpc2_f555w_pc, hst_05397_01_wfpc2_f814w_pc