In the early 1900s, Sergei Mikhailovich Prokudin-Gorskii travelled across the Russian Empire taking various photographs of people, buildings, landscapes, etc. While there was no way to print color photographs at that time, Prokudin-Gorskii captured these pictures by taking three exposures of the scene using red, green, and blue filters, envisioning that there would be a way to combine these filters into a fully colorized picture in the future. The goal of this project is to finish what he started.
Each scene is provided as one large image file of the three filters which are ordered blue-green-red in descending order. After isolating each exposure and converting to float32, the filters are aligned relative to the green filter using a multi-scale processing scheme called an image pyramid (which defaults to single-scale processing if the image width is less than 350px). At each scale of the pyramid, the algorithm searches over a user-specified range of x and y displacements and identifies the "best" alignment/displacement as that which minimizes the L2 distance between the images (SSD). All three filters are then aligned w.r.t to these displacement vectors (x,y) to produce the final image.
Initially, the optimal alignment was found using an exhaustive search across all possible displacement vectors (x, y) in the range [-15, 15] w.r.t the blue filter. This process took up to 2 minutes for the large .tif files and generally performed well, barring a few instances such as Emir where the alignment was perceptually off and produced slight red/blue hues around the outline of the person.
The following images were all colorized with a displacement search window of 15.
Blue Shift: (0,0) ; Red Shift: (1, 4)
Blue Shift: (-2,3) ; Red Shift: (1, 6)
Blue Shift: (-3,-3) ; Red Shift: (1, 4)
Blue Shift: (-12, -64) ; Red Shift: (10, 73)
Blue Shift: (-26, -51) ; Red Shift: (10, 57)
Blue Shift: (-14,-53) ; Red Shift: (-3, 58)
Blue Shift: (-17, -40) ; Red Shift: (5, 48)
Blue Shift: (-24, -49) ; Red Shift: (17, 57)
Blue Shift: (-16,-59) ; Red Shift: (-3, 65)
Blue Shift: (-9, -51) ; Red Shift: (3, 61)
Blue Shift: (-29, -78) ; Red Shift: (8, 98)
Blue Shift: (0,-53) ; Red Shift: (-11, 52)
Blue Shift: (-10, -81) ; Red Shift: (4, 96)
Blue Shift: (-5, -42) ; Red Shift: (27, 43)
Blue Shift: (22, -52) ; Red Shift: (-33, 55)
Blue Shift: (-7, -21) ; Red Shift: (-6, 28)
Blue Shift: (-9, -14) ; Red Shift: (-1, 2)
After an initial alignment, most if not all pictures still have 'color artifacts' near the edges due to offsets during alignment, faulty exposures, or degradation of the film. Being able to automatically detect these artifacts and crop the picture accordingly would produce a nicer result in most cases. Using image gradients, these artifacts are detected as horizontal/vertical lines of gradients whose absolute average value across the line lies above a certain threshold, and the image is cropped until the deepest occurence of such a line.
In the age of smartphones, fancy cameras, and social media, it is common for everyday photos to be auto-contrasted or modified by applying some sort of filter. Usually, the channel adjustments result in more "realistic" or "higher quality" photos that more closely resemble what we see. Given the older technology that was used to take these photographs, it would be interesting to see how they may look under a more mondern "lens". The following adjustments were made according to the methodology described here.