Project 1 - Colorizing the Prokudin-Gorskii photo collection

CS 194-26, UC Berkeley

Yassin Oulad Daoud

This project involved colorizing images from the collection of the digitized Prokudin-Gorskii glass plate images kept by the Library of Congress. Using image processing techniques in Python, we can automatically produce colorized images from each set of three black and white glass plate images.

To do this, the images must be aligned, since they do not overlap perfectly when cropped from the three digitzed glass plate panels.

Approach

I first wanted to tackle colorizing the lower-resolution (.jpg) images. For these, I could simply take the two color channels I wanted to align, check all the possible offsets within a given window of offsets by pixel (e.g. -20 to 20 pixels along the x- and y-axes), and use the offset that yields the minimum Sum of Squared Distances (SSD) between the RGB values of the channels. This approach worked quite well for the three lower-resolution images, producing a colorized output image in a few seconds.

However, for the higher-resolution (.tif) images, this approach led to unacceptably long turnaround times for colorizing the images. This is because the fixed offset window would include many pairs of x- and y-offsets to try, and the SSD would need to be calculated over the huge channels' pixel matrices for each pair.

To optimize my solution, I implemented my own image pyramid function in order to first calculate offsets at lower resolution versions of the same color channels, thus requiring less computational effort to calculate the overall offsets to use. To make a "step" down the pyramid, I wrote a helper function to scale down an image by a factor of 2 on the x- and y-axes and return an image of one quater the size of the input image (pixel values would be averaged in squares of 4 pixels to produce the output scaled image). I wrote a recursive pyramid function that stepped through the pyramid to some fixed last level (depending on the image size--larger images required about 8 pyramid levels, smaller ones required about 4) and checked each offset from the offset window at that level. Because of the scaling factor, the offset window is just a few pixels along each axis. The offsets calucated at a later (lower-res.) step of the pyramid would be returned and multiplied by 2 to be used as a "starting offset" for the current step (higher-res.).

I tested the above pyramid method on each image, and was able to produce the desired colorizatons for most of the input images. Four of the images, however, did not align properly. I changed the metric I was using to obtain the offsets to use from the SSD to the Normalized Cross Correlation (NCC) of the color channels' pixel values, and was able to then obtain the desired alignments for all but one of the images.

The image I was not able to align properly was "emir.tif". This image seems to have different brightnesses for each of the three color channels, thus the NCC and SSD computed even for the correct offsets were not accepted by the algorithm, which took some incorrect offset instead for alignment. These simpler metrics are based solely on RGB values of each pixel, so the relation between each pixel value and that of the pixels around will not be captured using these methods. A metric that takes into account the relational data between pixels, like gradients or contours, may work better for images like this one.

To trim off the borders of the images, I eyeballed a ratio between the number of pixels that the low and high resolution image borders took up and the width of the entire photo. Then, before performing any computations, I trim off this portion of the pixels from all sides of the image. I found that this approach removed the borders from each image adequately and did not remove too much of the desired parts of the images.

Results

castle.tif

cathedral.jpg

emir.tif

(NOT PROPERLY ALIGNED)

harvesters.tif

icon.tif

lady.tif

melons.tif

monastery.jpg

onion_church.tif

self_portrait.tif

three_generations.tif

tobolsk.jpg

train.tif

workshop.tif

Self-selected images

battlement.tif

peonies.tif

woman.tif

Offsets calculated