In this project, I attempt to colorize several photographs taken by Sergei Mikhailovich Prokudin-Gorskii (1863-1944). Prokudin-Gorskii created many RGB glass plates of things around Russia, and by aligning them, we are able to see a color photo of the scene. We do this by stacking 2D arrays that represent each component of the image on top of each other, effective finding how much red, green and blue are needed in each pixel.
The main issue lies in the alignment of the glass plates. I try to create an algorithm that can automatically align the plates using some metric. For this project, I chose to use the L2 norm, or Sum of Squared Differences (SSD), to score possible displacements for the pixels of the images, and I take the smallest value, aligning the image with the associated vector. I align both the green and the red plate onto the blue plate. Specifically, I search a range from -15 to 15 pixels to see which alignment fits best. A single iteration of this search basically sums up my naive, single scale implementation.
However, complications arise for large images. The TIFF images are simply too large to do an exhaustive search over that large range. To combat this, I used an image pyramid. I create a predetermined number of layers for the pyramid, with each layer being the image scaled down to a smaller size (I scale down by half each time). Since I chose to have 10 layers in my pyramid, the smallest image ends up being 2^10 = 1024 times smaller than the original. This allows me to have a smaller range to search for that best alignment vector, simply because there are less pixels. I chose to search within a range of -5 to 5. I then scale up this alignment vector to the next image, and apply it at that level. Since this application reduces the distance that the images are from being truly aligned, I can retain that small range of search for this larger size image. I then adjust my alignment vector using the new calculations, and repeat for bigger sizes. By doing so, I effectively reduce the time it takes to align and colorize these photos.
Below are my results.
Single-Scale Algorithm Results on Smaller, JPG Images
Notes: the single scale algorithm works well for these small images. Cathedral's blue plate seems to be slightly high, possibly due to how the photograph is laid out (the grass details would make it hard for the algorithm to score the true alignment vector well).
|
Image Pyramid Algorithm Results on All Images
Notes: Due to their small size, the JPGs actually suffered from the reducing of size, since the vectors at lower fidelities only served to throw off the true vector that could've been found naively. However, most TIFFs enjoyed favorable results. Icon was my favorite, since the algorithm lined up the details of the gold sculpture and wall extremely well, resulting in a clean photograph. Emir, on the other hand, failed miserably; the different brightnesses (color channels) between the R, G, and B plates negatively affected the scoring algorithm. The same argument applies to the Village photograph, which also was not quite aligned correctly. Despite this, only 2 out of the 11 photographs provided did not turn out as well, so I am happy with the current algorithm and configuration of variables.
Also, I'd like to note a few details I found interesting. In the Harvesters photo, you can see that some women are clear, implying they stood very still for the photo. However, several women did not line up very well at all, but because it is specific to these very few women, I suspect that they must have moved a lot during the taking of the photograph. Also, in the Self-Portrait photo, I can see red-blue artifacts in the background. These might imply some being came through the area when those plates where taken, but not when the green plate was shot. Also, the long duration of the photo taking time results in a very beautiful blurry water effect.
|
|
|
|
|
|
|
|
|
|
|
Additional Images from the Collection
These are some additional images I chose from the collection to show the algorithm's results. They all work well. I chose Gospel because I really liked the gold intricacies in Icon, and wanted to see those results again. I also gave my algorithm an extremely difficult image: Forest. The amount of green detail, as expected, confused my algorithm with many false positives for the correct alignment vector. That, combined with how intricate the image is, leads to a failed output for this particular image.
|
|
|
|
Vectors and Times
I've listed here my outputted data for all the alignment vectors found by the algorithm, as well as the execution time to find that alignment vector. Vectors are in (x, y) format; x being the number of pixels to roll the array horizontally, and y the nubmer of pixels to roll vertically.
Monastery: (2, -3) for green plate, (2, 3) for red plate. Took 2.5276710987091064 seconds.
Cathedral: (2, 1) for green plate, (3, 8) for red plate. Took 2.903214931488037 seconds.
Tobolsk: (2, 3) for green plate, (3, 6) for red plate. Took 2.932835102081299 seconds.
Village: (-5628, -5760) for green plate, (-5606, -5641) for red plate. Took 19.35067105293274 seconds.
Harvesters: (-5521, -5641) for green plate, (-5524, -5578) for red plate. Took 18.86025905609131 seconds.
Train: (-5567, -5686) for green plate, (-5541, -5640) for red plate. Took 20.170985221862793 seconds.
Self Portrait: (-5568, -5664) for green plate, (-5559, -5566) for red plate. Took 19.30407404899597 seconds.
Three Generations: (-5510, -5623) for green plate, (-5512, -5567) for red plate. Took 18.833855152130127 seconds.
Lady: (-8284, -5622) for green plate, (-8282, -5549) for red plate. Took 18.826510667800903 seconds.
Icon: (-5563, -5700) for green plate, (-5558, -5650) for red plate. 19.36525011062622 seconds.
Emir: (-5499, -5630) for green plate, (-9293, -5565) for red plate. 21.13472318649292 seconds.
Workshop: (-5523, -5618) for green plate, (-5534, -5570) for red plate. Took 19.35067105293274 seconds
Onion Church: (-5504, -5623) for green plate, (-5494, -5566) for red plate. Took 18.914446115493774 seconds.
Melons: (-5569, -5646) for green plate, (-8355, -5550) for red plate. 19.526010036468506 seconds.
Bashkir Woman: (-5606, -5706) for green plate, (-7038, -7858) for red plate. Took 18.681334018707275 seconds.
Forest: (-8408, -8587) for green plate, (-5606, -5641) for red plate. Took 19.35067105293274 seconds
Church: (-5519, -5641) for green plate, (-5514, -8424) for red plate. Took 18.352840185165405 seconds.
Gospel: (-8373, -5642) for green plate, (-8415, -5563) for red plate. Took 19.06008505821228 seconds.