CS194-26 | Colorizing the Prokudin-Gorskii Collection
Allen Zeng, CS194-26-aec

Large outputs rescaled to less than 1500px height due to 25MB limit.

Algorithm In Depth

First, the three channels are obtained. Note that the original three-panel image has borders around each panel. So, 7% of each side of each channel is cropped out. This value was empirically chosen to balance border removal and keeping the image intact.

Then the Green channel is aligned to the Blue channel. And then the Red channel is aligned to the Blue channel. For the alignment, NCC was used. While aligning two channels, the assumption is made that two channels are no more than +/-15% displaced in translation, relative to the channel size. Empirically, that factor is generous in displacement. Furthermore, we only examine the inner 70% of the images (without cropping the original any further). This ensures that the plate borders and the physical blemishes of the plate edges do not affect alignment.

So the search space over channel $A$ and channel $B$ is the double for-loop over displacements [-15%, 15%] vertically and [-15%, 15%] horizontally. (This factor works well for small images, but not for large images. Image pyramids are used to optimize this part.) The translated channel $A$ that receives the best NCC score is returned, as the best alignment to channel $B$.

Once Green and Blue are aligned, and Red and Blue are aligned, the three colors are stacked into a single image. The outputs are shown below.

Image Pyramids

While searching over +/-15% displacement works for small images, it is inefficient for larger images. (For example, something over the size of 1000x1000 pixels.) To solve this inefficiency, image pyramids are implemented with a factor of 2.

Image pyramids are created for each channel. Say for channel $A$, the original image size is 2000x2000 pixels. A list of scaled-down images are produced based on that large image. In my implementation, I scale until both height and width are below 200 pixels. So for example, I would end up with a list of scaled images $$[(2000\times2000),(1000\times1000),(500\times500),(250\times250),(125\times125)].$$ This way, I can initially limit the alignment search space to the smallest image. The search becomes much faster. After the alignment is found for the smallest image, we search again in the image one step larger. In the one-step-larger search, we can do a rough alignment given by $2 \times $(the previous image's alignment). Note that pyramid has a scale factor of 2. Also, note that due to the rough alignment and the factor of 2, this current roughly aligned image is at most +/-2 pixels away from the optimal alignment to channel $B$ at this level. So at the current level, the search space can be limited to +/-2 pixels in each direction. (Just to be safe, I set the search space to +/-3 pixels. But empirically, all relative displacements are actually within [-2,2] pixels. Except for the Emir picture.)

After recursively finding the image alignments in the pyramid, the original image is efficiently aligned. This is much better than naively searching over [-15%, 15%] of 2000 pixels: 600x600 pixel search space.

Again, after alignment, the large original channels are stacked to create the output image.

Aligned Images (Images Clickable)

All translational displacements are relative to Blue channel (top image). All of these displacements are in terms of absolute translation in pixels (not relative per level).

* small input images run on single-scale search (no pyramid processing necessary)

Key: channel[Vertical pixels, Horizontal pixels]

* Cathedral G[5 2] R[12 3] Runtime: 0.4774 secs	Emir G[49 24] R[45 -303] Runtime: 22.2336 secs	Harvesters G[59 16] R[124 13] Runtime: 20.4103 secs	Icon G[40 17] R[89 23] Runtime: 21.7114 secs
Lady G[48 9] R[112 11] Runtime: 9.9525 secs	* Monastery G[-3 2] R[3 2] Runtime: 0.5933 secs	* Nativity G[3 1] R[7 0] Runtime: 0.7132 secs	Self G[78 29] R[176 37] Runtime: 19.6527 secs
* Settlers G[7 0] R[14 -1] Runtime: 0.5585 secs	Generations G[53 14] R[112 11] Runtime: 18.7443 secs	Train G[42 5] R[87 32] Runtime: 20.0464 secs	Turkmen G[56 21] R[116 28] Runtime: 21.6017 secs
Village G[65 12] R[137 22] Runtime: 21.0091 secs	Plaque G[62 19] R[134 26] Runtime: 19.7466 secs	House G[9 19] R[35 38] Runtime: 18.4928 secs	Closet G[36 -18] R[102 -15] Runtime: 19.07636 secs

Going Further (Extra Credit)

Contrast Stretching

I also implemented a function which does Contrast Stretching (AKA Normalization) on any given channel. This Contrast Stretching function takes in 2D matrix of uint16 values, and remaps them such that the visual contrast of the channel is more noticeable.

Algorithmically, the matrix of uint16 values is first flattened into an array. Then the cumulative histogram of the array is calculated and normalized to [0.0,1.0]. The x-axis of the cumulative histogram has 65536 bins ranging from [0, 65535], and the y-axis is [0.0,1.0]. Define map $T(x)$ to be that cumulative histogram. Then for each pixel in the original channel (each element in the original matrix), the pixel is remapped to $y=T(x)\times65535$.

This remapping causes the channel to have better contrast. The non-cumulative histogram of the remapped image is uniform, and the cumulative histogram is a straight ramp.

As an example, here is the Contrast Stretching of the Red channel of Cathedral. Before and after.

Contrast Stretching on RGB RGB Stretching

For the "RGB Stretching" set of images, I applied the Contrast Stretching function above to all three channels BEFORE alignment.

The alignment on the images turned out to be approximately the same as in the original algorithm. (Except Emir).

But the most noticeable difference is the change in colors. Overall, the colors seem much more vivid, saturated, and balanced. Additionally, the black-and-white contrast of the images are more noticeable well. On the whole, the set of images is cooler and more blue in comparison to the origin set. A few of the images are unrealistically colored however.

Under the assumption Prokudin-Gorskii used pure RGB filters, the RGB output images in the original algorithm has strong yellows and sepia tones. This suggests that while the RGB assumption is decent, Prokudin-Gorskii did not actually have pure RGB filters.
Due to the yellow biased images under the RGB model -- if the image does not have a good distribution of blue colors, the RGB Stretching function would make the output image too blue. (Blue is the complement to yellow.) Images with sufficient blue in them turn out very well though. See "Self" image vs "Lady" image.

Specific comparisons below.

Contrast Stretching on Value of HSV HSV Value Stretching

Seeing that alignment did not change with the RGB Stretching, for the "HSV Value Stretching" set I did processing AFTER alignment. After aligning the images using the original algorithm, I converted the RGB image to HSV space. Note that the Value parameter in HSV corresponds to black-and-white contrast of images. Then I performed Contrast Stretching on that Value parameter. Essentially, this attempts to improve black-and-white contrast without changing the hues in the image. As shown below, this worked well.

Contrast Stretching on SV of HSV HSV Saturation and Value Stretching

For the "HSV Saturation and Value Stretching" set, I did the same thing as HSV Value Stretching, but I also called the Contrast Stretching function on the Saturation parameter of HSV.

This change produced more realistically colored images than just Value Stretching. That saturation of colors seem to be better balanced, meaning nothing is too unsaturated or too saturated. But the images also seem less vivid overall.

Alignment Over Scale and Rotation 5D Transform

Based on the technology available to Prokudin-Gorskii, not all three panels in the image plates were perfectly captured at the exact same point in 3D space. As a result, the photos in each panel may be slightly out-of-scale with each other. (I.e. different zoom levels.) Furthermore, when the plates were transfered to a digital format, some of the panel photos may have been slightly rotated.

To account for this, we increase the dimesionality of the search space to include vertical scaling, horizontal scaling, and rotation, in addition to the original vertical and horizontal translation. For scaling, I assumed channel $A$ is no more than +/-5% scaled than channel $B$ in each direction. For downscaling channel $A$, I padded with gray (0.5) values; and for upscaling $A$, I had to crop out edge pixels. For rotation, I assumed channel $A$ is no more than +/-5 degrees rotated than channel $B$.

I took advantage of the recursive image pyramid approach to speed up the 5 dimesional search, but the large .tiff images took about half an hour each to process.

In terms of alignment, the results were worth it though: