CS 194: Project 1
Images of the Russian Empire:
Colorizing the Prokudin-Gorskii Photo Collection
Introduction
(From Project Description) The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, you will need to extract the three color channel images, place them on top of each other, and align them so that they form a single RGB color image.
Methodology
Alignment
To begin the process, each negative was divided into thirds along its vertical axis and separated into three images, each representing one color channel (red, green, or blue). These three images were then separately stored in NumPy arrays, and passed pairwise into an alignment method.
The photos were repositioned using a six-level pyramidal search method, starting with coarse alignment and ending with the finest pixel-wise adjustments. Each level increases the resolution of the photos being matched by a factor of two, starting with photos at ο»Ώ of their original resolution, then iteratively halving the compression until you are matching the full-resolution images.
At each level of the pyramid, the images are aligned according to a predefined loss function. The alignment with the lowest loss function is then selected and used as the starting point for the comparison at the next level.
Cost and comparison
The loss function used is based on the classic sum of squared differences (SSD) loss between the pair of channels being aligned. However, because the real-life subjects of these pictures are not in grayscale, the color channels are not always perfectly correlated. A perfect example of why this causes problems when using the naΓ―ve SSD loss is the photo of the Emir of Bukhara:
Notice the stark differences between the intensities of each of the raw color channels in this image. This causes SSD loss to sometimes miss the alignment that seems obvious to us humans, because the blocks of pixels that need to be aligned between the different images are not similar in intensity.
So, instead, my loss function takes the current alignment and then shifts it by +1 in both the horizontal and vertical directions. Then, these two shifted versions of the image are subtracted from the original and the absolute values of the resulting images are summed. This is a little complicated to explain in English, so I will include the corresponding snippet of my code below:
xb = np.abs(x - np.roll(x, 1, axis=0)) + np.abs(x - np.roll(x, 1, axis=1))
yb = np.abs(y - np.roll(y, 1, axis=0)) + np.abs(y - np.roll(y, 1, axis=1))
return np.sum(np.sum((xb / np.mean(xb) - yb / np.mean(yb)) ** 2))
This results in a makeshift edge detector, which attempts to align not the blocks of color, but the edges between blocks of color, thereby avoiding the issue of different color channels having different intensities.
Auto-Cropping
This alignment, however, creates distortions at the fringes of the pictures, as the edges of shifted color channels no longer align correctly. This results in bands of color that are not representative of anything in the real world, such as those seen in the following image:
To fix this, I algorithmically crop out these bands of color to return an image that is more representative of the actual subjects of the photograph. This is achieved by another clever use of our pseudo-edge detector technique, in this case by summing down the rows and columns of the image and looking for the largest one-step gradients in each. Because the crop lines are generally straight up and down, they show up in this summed and shifted domain as large spikes in the gradients. By isolating these spikes, and finding the largest ones near the tops and bottoms of the images, we are able to easily identify and remove these awkward bands of color.
Auto White Balance
The last step in my pipeline is to attempt to subtly fix the white balance of the cropped pictures. One simple way to do this is to set the brightest pixel in the scene to white, and adjust all of the other pixels accordingly.
To achieve this, I simply identify the brightest pixel in the image and take the difference between that pixel's three color channels and perfect white, then factor that difference onto every other pixel while clamping all pixels to a maximum of 1.0. Doing so subtly adjusts the color balance of the image so that whites are whiter and colors are more evenly matched, resulting in the images you see below.
Gallery
Emir
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment:
Red Green Blue
<24, 49>
<40, 107>
156β3091 222β3386
0.01884489 0.02391089 0.02244602
Church
Green Offset:
Red Offset:
Crop:
(top to bottom) (left to right)
Auto White Balance Adjustment:
Red Green Blue
<4, 25>
<-4, 58>
89β3067 103β3413
+0.01939422 +0.04950027 +0.05072099
Monastery
Green Offset: <2, -3>
Red Offset: <2, 3>
Crop:
(top to bottom) 15β331
(left to right) 21β377
AWB: None
Cathedral
Green Offset: <2, 5>
Red Offset: <3, 12>
Crop:
(top to bottom) 13β330
(left to right) 14β372
AWB (RGB): +0.01176471 +0 +0.0588235
Harvesters
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<18, 60>
<14, 124>
200β3180 200β3493
+0.05229267 +0.02877852 +0.02053864
Icon
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<17, 41>
<23, 90>
189β3204 225β3612
+0.05072099 +0.05285725 +0.05911345
Lady
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<9, 56>
<14, 113>
239β3161 188β3528
+0.01785306 +0.04377813 +0.01443503
Melons
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<10, 80>
<13, 177>
267β3241 195β3515
+0.01918059 +0.01918059 +0.03874266
Onion Church
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<26, 51>
<36, 108>
248β2907 376β3609
+0.03599603 +0.04293889 +0.04293889
Three Generations
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<13, 53>
<9, 111>
240β3045 268β3411
+0.05232319 +0.11184863 +0.07902647
Tobolsk
Green Offset: <3, 3>
Red Offset: <3, 7>
Crop:
(top to bottom) 9β331
(left to right) 21β381
AWB (RGB): None
Train
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<0, 41>
<30, 86>
215β3158 184β3571
+0.0476997 +0.02937362 +0.02839704
Workshop
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<0, 54>
<-12, 105>
251β3198 178β3391
+0.01933318 +0.0183566 +0.02244602
Self Portrait
Green Offset:
Red Offset:
Crop: (top to bottom) (left to right)
Auto White Balance Adjustment: Red Green Blue
<29, 78>
<37, 175>
194β3138 335β3656
+0.03199817 +0.03105211 +0.03419547
Additional Examples
Domes
Samarkand
Flowers