For historical context and instructions, see the project specs here.
The goal of this project is to take the black and white images that Projudin-Gorskii took, and combine them to form colored images. His photos were captured in a very unique fashion. He would take 3 images of the same exact scene, but would filter out a specific color for each of the images. This allows the 3 images to capture the color information long before color photography was possible.
In order to produce the colored versions of his images, we will use some image processing techniques to split, align, and then reproduce the colored image.
Each of his images begins as a plate of 3 stacked images.
In order to get each channel separately, we can simply slice the image into 3 pieces, getting us each channel individually.
We can then assign each frame to the corresponding color and stack them.
And bingo bwamo! we now have a colored image!
Unfortunately, there is a glaring issue. The iamges are not aligned, causing for a rather poor quality image. Let's see what we can do about that.
In order to align the images, we will begin with one color channel, that we will shift around to align to another color channel.
My algorithm iterates through a 30x30 window (for small jpegs) and attempts to determine which of the following alignments are the best.
To compare different alignments, normalized cross-correlation (NCC) is used, which is simply a dot product between two normalized vectors. This measure allows us to see how similar two corresponding alignments are.
When images are lined up correctly, the pixel values will be relatively similar based on brightness and the contents of the image, leading to a high ncc. Then we can pick that alignment, and stack the translated versions of the 3 color channels.
Let's see the results!
monestary.jpg R = (2,3) G = (2,-3)
cathedral.jpg R = (3,12) G = (2,5)
tobolsk.jpg R = (3,7) G = (3,3)
While exhaustively searching through possible alignments works for the smaller jpeg images, this will not work for larger images, where displacements can be hundreds of pixels, instead of just a few pixels.
In order to combat this, we will use a pyramid alignment technique, in which we first scale down the image and align it roughly first. Afterwards, we slowly look at larger copies of the image, and make more fine alignment adjustments, until we are at the orginal image size. At this point, our image will be mostly aligned, which was done at lower resolution saving on processing time. Then we can make the alignment on the large image, but only have a very small search window since the earlier steps have guaranteed our image is already somehwat aligned.
My algorithm processes the image in 5 levels, scaling down the image by 1/2 each time (up to 1/16th scale).
Starting at the lowest levels, I search a window size of 2^(current level). This allows us to make large moves on the smaller scaled images, and then as we make our way up, narrow down our search window and make finer alignment adjustments. For example, on the 1/16th scale, if we make an shift the iamge 1 to the right, I keep note of that and that eventually corresponds to a 16 pixel shift on the full resolution picture.
By breaking down the search into multiple layers, this vastly improves the processing time compared to exhaustive search. This allows the alignment for each photo to be completed in ~20 seconds.
church.tif R = (-3,57) G = (4,24)
harvester.tif R = (18,123) G = (19,59)
icon.tif R = (23,91) G = (18,41)
lady.tif R = (9,113) G = (6,51)
melons.tif R = (13,179) G = (3,87)
onion_church.tif R = (37,107) G = (26,49)
self_portrait.tif R = (37,175) G = (29,77)
three_generations.tif R = (11,108) G = (15,49)
workshop.tif R = (-12, 106) G = (0, 53)
train.tif R = (32,85) G = (6,42)
emir.tif R = (-276, 157) G = (23, 48)
uh oh. It looks like for some reason, the emir photo did not align correctly. Why.
If we look at the original picture, we can see that the coat colors are completely different in each color channel. In one, it is dark, while in the other, it is white. This will cause our alignment to mess up due to the relative brightness of each color in the coat. What can we do about this?
Rather than relying directly on pixel values, which appears to have not worked in this case, we will find the gradient of each color channel, and align the pictures based on the gradients. This allows us to try to match up the features and textures of the pixture, rather than relying on the relative brightness of colored pixels.
The gradient measures how much of a change there is between pixels within the image. Taking the gradient tens to highlight edges and features within the image
Here is the gradient for the red channel of the image
This way, it doesn't matter if the coat is white or dark. Our alignment algorithm can match up the features of the details on the coat.
We can thus reuse our pyramid alignment, and get a much better alignment for the emir picture
emir.tif R = (41, 106) G = (23, 48)
As a result of shifting different color channels around, it is often the case that we end up with large bands of a solid color on the edge of the image. In order to remove these, I have written an auto cropping technique that looks at each edge, and removes layers of the image where the entire row or column has a low variance in it's different color channels. This indicates that there might be a bar of solid color there that we can safely remove, while keeping the important parts of the iamge intact.
Before
After
Before
After
Unfortunately, Prokudin-Gorskii was unable to see his images in color, and thus couldn't tell if the white balance was correct or not. To help with this, I wrote a simple technique for attempting to automatically white balance the images.
The technique works by assuming that the average scene is grey, and scaling all the pixels in the iamge to match that.
Before
After
We will take some additional photos from the collection and run the same process on them to see them in color.
animal.tif R = (28,130) G = (22,59)
peasant_girl.tif R = (19,11) G = (10,-15)
floodgate.tif R = (28,26) G = (18,9)
