The overall goal of this project is to colorize the Prokudin-Gorskii Photo Collection. While Prokudin-Gorskii provides the red, green, and blue-filtered plates for his images, they are not necessarily aligned, which is where the program comes in.

The alignment program determines whether to perform an exhaustive or multiscale search based on the size of the image. We first crop off 10% of the image along each edge to eliminate the black and white borders. The reference plate is the green channel, so the red and blue are shifted relative to it. If the images is smaller than 512 × 512 pixels, exhaustive search is performed.

Exhaustive Search

Exhaustive search is performed using Python’s multiprocessing package. The search window interval is calculated as 10% of the image height and width ranging from a shift of -5% to +5%, respectively. Every combination of shifts is pre-computed and dispatched to multiple CPU threads in chunks of 128. The metric used for this is normalized cross-correlation, using the formula

Where I_1 and I_2 represent the target and reference image after the Sobel edge operator is applied, respectively. The more the images are aligned, the higher this metric is, so this metric is maximized in exhaustive search.

Multiscale Search

For images larger than 512 × 512 pixels, multiscale search is implemented. Both the target and reference image are downscaled by a factor of 1/2, and the algorithm is recursively applied until we reach the base dimension of 512 × 512 pixels, at which point exhaustive search is employed. Once the recursive call returns, we perform an adjustment at the previous scale along a [-2, 2] pixel window in both dimensions. We ensure that aliasing is avoided by setting anti_aliasing=True in the call to skimage.transform.rescale.

Problems

There were various problems when implementing this algorithm.

• The program ran extremely slowly on larger and larger windows and image sizes on exhaustive search. This was solved by employing numpy vectorization whereever possible, as well as using CPU multiprocessing.
• Even when using multiscale processing, the image would not be as aligned as desired. This was solved in a multitude of ways: cropping out the borders of the image, using the green channel as the reference image, and using recursive adjustment.

Three bells and whistles were implemented in this project: Sobel edge scoring, multiprocessing, and auto-contrasting.

Sobel Operator

Before performing the normalized cross-correlation function, a Sobel edge detection operator (namely, skimage.ndimage.sobel) is applied across both images. This is to reduce the confounding effect that differing brightnesses in color have across channels.

Interestingly enough, this has basically no discernible effect on the resultant images as opposed to a non-Sobel operated normalized cross-correlation.

Multiprocessing

Multiprocessing from Python’s built-in package is used in order to speed up calculations. Without multiprocessing, a .tif file takes around 22 seconds to align on an Intel i7-8750H with 6 cores. With multiprocessing, it usually takes around 20 seconds.

Auto-Contrasting via Histogram Equalization

In order to fix the contrast and “brighten up” some of the images, histogram equalization is performed on each one via the skimage.exposure.equalize_hist function. This technique uses the concept of histogram equalization to push down middling-intensity colors in order to boost the present of the darkest and lightest parts of any given image.