πŸ“·

CS 194: Project 1

Images of the Russian Empire:

Colorizing the Prokudin-Gorskii Photo Collection

Introduction

(From Project Description) The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, you will need to extract the three color channel images, place them on top of each other, and align them so that they form a single RGB color image.

Methodology

Alignment

To begin the process, each negative was divided into thirds along its vertical axis and separated into three images, each representing one color channel (red, green, or blue). These three images were then separately stored in NumPy arrays, and passed pairwise into an alignment method.

The photos were repositioned using a six-level pyramidal search method, starting with coarse alignment and ending with the finest pixel-wise adjustments. Each level increases the resolution of the photos being matched by a factor of two, starting with photos at 132\frac{1}{32}ο»Ώ of their original resolution, then iteratively halving the compression until you are matching the full-resolution images.

At each level of the pyramid, the images are aligned according to a predefined loss function. The alignment with the lowest loss function is then selected and used as the starting point for the comparison at the next level.

Cost and comparison

The loss function used is based on the classic sum of squared differences (SSD) loss between the pair of channels being aligned. However, because the real-life subjects of these pictures are not in grayscale, the color channels are not always perfectly correlated. A perfect example of why this causes problems when using the naΓ―ve SSD loss is the photo of the Emir of Bukhara:

Notice the stark differences between the intensities of each of the raw color channels in this image. This causes SSD loss to sometimes miss the alignment that seems obvious to us humans, because the blocks of pixels that need to be aligned between the different images are not similar in intensity.

So, instead, my loss function takes the current alignment and then shifts it by +1 in both the horizontal and vertical directions. Then, these two shifted versions of the image are subtracted from the original and the absolute values of the resulting images are summed. This is a little complicated to explain in English, so I will include the corresponding snippet of my code below:

xb = np.abs(x - np.roll(x, 1, axis=0)) + np.abs(x - np.roll(x, 1, axis=1))
yb = np.abs(y - np.roll(y, 1, axis=0)) + np.abs(y - np.roll(y, 1, axis=1))
return np.sum(np.sum((xb / np.mean(xb) - yb / np.mean(yb)) ** 2))

This results in a makeshift edge detector, which attempts to align not the blocks of color, but the edges between blocks of color, thereby avoiding the issue of different color channels having different intensities.

Auto-Cropping

This alignment, however, creates distortions at the fringes of the pictures, as the edges of shifted color channels no longer align correctly. This results in bands of color that are not representative of anything in the real world, such as those seen in the following image:

To fix this, I algorithmically crop out these bands of color to return an image that is more representative of the actual subjects of the photograph. This is achieved by another clever use of our pseudo-edge detector technique, in this case by summing down the rows and columns of the image and looking for the largest one-step gradients in each. Because the crop lines are generally straight up and down, they show up in this summed and shifted domain as large spikes in the gradients. By isolating these spikes, and finding the largest ones near the tops and bottoms of the images, we are able to easily identify and remove these awkward bands of color.

Auto White Balance

The last step in my pipeline is to attempt to subtly fix the white balance of the cropped pictures. One simple way to do this is to set the brightest pixel in the scene to white, and adjust all of the other pixels accordingly.

To achieve this, I simply identify the brightest pixel in the image and take the difference between that pixel's three color channels and perfect white, then factor that difference onto every other pixel while clamping all pixels to a maximum of 1.0. Doing so subtly adjusts the color balance of the image so that whites are whiter and colors are more evenly matched, resulting in the images you see below.

Gallery

Emir

The Emir of Bukhara has more drip than Hurricane Ida.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment:

Red Green Blue

<24, 49>

<40, 107>

156–3091 222–3386

0.01884489 0.02391089 0.02244602


Church

The motion of the water and the defects in this film almost make this picture look like it was Deep Dream'ed (minus all the random fish and dogs).

Green Offset:

Red Offset:

Crop:

(top to bottom) (left to right)

Auto White Balance Adjustment:

Red Green Blue

<4, 25>

<-4, 58>

89–3067 103–3413

+0.01939422 +0.04950027 +0.05072099


Monastery

The domes on this monastery almost put the Hagia Sofia to shame.

Green Offset: <2, -3>

Red Offset: <2, 3>

Crop:

(top to bottom) 15–331

(left to right) 21–377

AWB: None


Cathedral

Green Offset: <2, 5>

Red Offset: <3, 12>

Crop:

(top to bottom) 13–330

(left to right) 14–372

AWB (RGB): +0.01176471 +0 +0.0588235

Saint Basil's lesser known sister cathedral, Saint Thyme's

Harvesters

The two people on the center-left were disco queens before it was cool.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<18, 60>

<14, 124>

200–3180 200–3493

+0.05229267 +0.02877852 +0.02053864


Icon

The intricate design of this shrine pales in comparison to the sheer amount of gold that was used to create it.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<17, 41>

<23, 90>

189–3204 225–3612

+0.05072099 +0.05285725 +0.05911345


Lady

Photograph me like one of your French girls.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<9, 56>

<14, 113>

239–3161 188–3528

+0.01785306 +0.04377813 +0.01443503


Melons

This man is not Toph. This man is Melon Lord.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<10, 80>

<13, 177>

267–3241 195–3515

+0.01918059 +0.01918059 +0.03874266


Onion Church

Shrek's foremost house of worship.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<26, 51>

<36, 108>

248–2907 376–3609

+0.03599603 +0.04293889 +0.04293889


Three Generations

Don't talk to me or my son or my son's daughter ever again.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<13, 53>

<9, 111>

240–3045 268–3411

+0.05232319 +0.11184863 +0.07902647


Tobolsk

Green Offset: <3, 3>

Red Offset: <3, 7>

Crop:

(top to bottom) 9–331

(left to right) 21–381

AWB (RGB): None

Tobolsk's record low temperature is -61.2Β° F, which is just a few degrees shy of the temperature that Southern Californians think Berkeley is.

Train

i like trains.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<0, 41>

<30, 86>

215–3158 184–3571

+0.0476997 +0.02937362 +0.02839704


Workshop

I had to workshop this caption a whole bunch of times before I thought it was any good.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<0, 54>

<-12, 105>

251–3198 178–3391

+0.01933318 +0.0183566 +0.02244602


Self Portrait

This looks like it could be the cover of the hottest album of 2021.

Green Offset:

Red Offset:

Crop: (top to bottom) (left to right)

Auto White Balance Adjustment: Red Green Blue

<29, 78>

<37, 175>

194–3138 335–3656

+0.03199817 +0.03105211 +0.03419547


Additional Examples

Domes


Samarkand


Flowers