Images of the Russian Empire - CS194-26 FA 2018 Project 1

Neil Ambar Gupta, cs194-26-aef

Project Idea

Prokudin-Gorskii took photographs of the old Russian empire landscapes in which he used 3 exposures of the same subject (with red, green, and blue filters) onto glass. Our project was to take those filters and convert them into a colorized image by aligning and superimposing the filters.


Example:

Trio BGR Filters
Conversion Result

The project's goal was to take the three filtered exposures and combine them to produce one colored image with precision and accuracy. This was done via aligning exposures as so above on the right. Then the alignments were superimposed and presented in color format as output. With varying format intensity and resolution, this was challenging but multiple approaches were implemented: naive alignment and pyramiding.

Basic Naive Approach

How I did it

The standard approach for small images was simple for this project. The first step was to isolate all 3 filters of an image, and i personally first crop 100 pixels from the edges to avoid discrepancies from the edges. Then I fixed the blue filter as the base filter and translated the green filters within a [-20, 20] vertical range and aligned where the two filters had the greatest Normalized Cross-Correlation value as matrices. The same goes for the red filter on the blue. Superimposition was done on all 3 filters after aligning and the resulting color images were outputted.

Nativity --> G:[3, 1] R:[8, 0]
Cathedral --> G: [5, 2] R:[12,3]
Monastery --> G: [-3, 2] R:[3, 2]
Settlers --> G:[7, 0] R:[15, -1]
Similarity method - Normalized Cross-Corrleation (NCC):

I used frobenius norm to calculate the NCC of two matrices (in our case the NCC of two filters). It is the dot product of two matrices (after normalizing) that would achieve filter relation desire. This was chosen because I found this process more preferable to SSD. Formula shown below:


Pyramid Approach

How I did it

In this situation, I used image pyramiding where at different resolutions and scaling up I implemented the naive approach at each layer. First I had to start at the smallest layer using (1/2)^depth as a scaling factor. In this case I used three s.t the smallest layer was 1/8th the size of the original filters and implemented a large version of the naive window [-30, 30] to start off with. As each layer increased size I formally imposed the naive window to a fixed [-4, 4] for alignment. Note this is done for both other filters on top of the main base filter at each layer, and each layer requires realignment it is on in relation to the base layer (e.g. 1 pixel adjustment on the 1/8 layer is the same as 8 pixels on the original layer). Once all the layers are done, with total alignments accumulated, we super impose the filters and retrieve our color filter. Once again, I used frobenius norm again to calculate the NCC of two matrices for filter similarity.


Emir --> B:[-48, -24] R:[54, 18]
Harvesters --> B: [-60, -18] R:[66, 0]
Icon --> B:[-42, -18] R:[48, 6]
Lady --> B:[-54, -6] R:[60, 0]
Self Portrait --> B:[-78, -24] R:[96, 6]
Three Generations --> B:[-54, -12] R:[60, 0]
Train --> B: [-42, -6] R:[42, 24]
Village --> G:[-66, -12] R:[72, 12]
Turkmen --> G:[-54, -18] R:[60, 6]
Problems

In certain images with strong blue presence, the red filter was off alignment especially with emir.tif, thus i decided to use green as the base filter (adding functionality for using any filter as base) and translating blue and red relative to it. It worked well for all images and I had no problems.

Extra Images

Some images I did for fun:

Church --> B:[-72, -18] R:[78, 0]
Samarkhand --> B:[5, 2] R:[12,3]
Statue --> B:[-12, -6] R[72, 6]