Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection
Overview
Sergei Mikhailovich Prokudin-Gorskii traveled all across the Russian Empire and took color photographs of a great variety of things. He was able to take color photos by taking three exposures with red, green, and blue filters. In this project, we attempt to combine these sets of three negatives computationaly while adjusting for unattractive attributes added due to age and/or the digitization process.
Approach
Preprocessing
For all approaches, the images are loaded in and split in to the three red, green, and blue negatives using the code provided in the starter. Additionally, 10% of the pixels on each side (top, bottom, left, right) are cropped off. This preprocessing step is done because I experimentally found it to cause much better results. That's likely the case because we SSD (the metric used) heavily penalized "misalignment" of the borders since they were black, with misalignment in quotes because the borders aren't a consistent black.
Naive
The naive approach involved searching over an entire 30 pixel by 30 pixel window of displacements. The displacement that is chosen is that which has the lowest SSD score. As this is the naive approach, the SSD score is calculated using every pixel in the image.
Naive Results
![Cathedral/0.067 minutes/ Red [12,3], Green [5,2]](./naive-images/cathedral.jpg "Cathedral")
Cathedral - Red [12, 3] Green [5, 2] - 0.067 minutes
![Monastery/ 0.059 minutes/ Red [3, 2], Green [-3, 2]](./naive-images/monastery.jpg "Monastery")
Monastery - Red [3, 2] Green [-3, 2] - 0.059 minutes
![Tobolsk/0.062 minutes/ Red [6, 3], Green [3, 3]](./naive-images/tobolsk.jpg "Tobolsk")
Tobolsk - Red [6, 3] Green [3, 3] - 0.062 minutes
Image Pyramid
The results of the naive approach are good, but it takes far too much time when it comes to the larger .tif images. Therefore we utilize an image pyramid approach. The process is as follows:
- Create a series of images per negative that start from an approximately a 60 pixel by 60 pixel image to an image of the original size, where each consecutive image is double the size of the previous.
- Search over a 30 pixel by 30 pixel window for the optimal alignment for the smallest image.
- For each following stage of the pyramid, take the previous optimal offset, multiply by two to account for scaling, then adjust by searching additional displacements in a smaller window (efficiency 1). Calculate displacement scores from a restricted window (efficiency 2).
See code for specific parameters.
Image Pyramid Results
harvesters.tif / 0.236 minutes / Red offset: [124, 15]. Green offset: [59, 18]
icon.tif / 0.234 minutes / Red offset: [90, 23]. Green offset: [41, 18]
lady.tif / 0.237 minutes / Red offset: [118, 13]. Green offset: [52, 7]
melons.tif / 0.236 minutes / Red offset: [182, 14]. Green offset: [85, 10]
onion_church.tif / 0.229 minutes / Red offset: [108, 37]. Green offset: [50, 26]
self_portrait.tif / 0.240 minutes / Red offset: [175, 37]. Green offset: [77, 29]
three_generations.tif / 0.219 minutes / Red offset: [111, 12]. Green offset: [53, 15]
train.tif / 0.232 minutes / Red offset: [86, 32]. Green offset: [42, 7]
village.tif / 0.237 minutes / Red offset: [137, 22]. Green offset: [64, 12]
workshop.tif / 0.232 minutes / Red offset: [105, -12]. Green offset: [53, -1]
emir.tif / 0.231 minutes / Red offset: [-4, -859]. Green offset: [48, 24]
As you can see, this image was not able to be aligned due to the specific attributes of the negatives - his outfit is very specifically one color, so one of the three negatives has pixels where the others don't, leading it to be aligned with a different area that has a lot of the pixels (eg. the background). We fix this in the "Bells and Whistles" section.
Additional Images from the collection
Note: these are aligned with the edge detection method from Bells and Whistles.
optional1.tif / 0.242 minutes / Red offset: [40, 15]. Green offset: [13, 14]
optional2.tif / 0.233 minutes / Red offset: [79, -14]. Green offset: [32, -1]
optional3.tif / 0.221 minutes / Red offset: [41, 35]. Green offset: [18, 28]
Bells and Whistles
Emir in particular was not able to be aligned properly using the simple image pyramid process. To fix this, we utilize simple edge detection in the form of a Sobel fitler after the cropping process and before the pyramid generation process. This fixes Emir because now the problems caused by his outfit color don't apply because they are not an edge.
emir.tif / 0.2505 minutes/ Red offset: [107, 41]. Green offset: [49, 24]
Fixed Emir