Fall 2018

CS194-26 - Project 1

Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

Regina Ongowarsito - cs194-26-aeq

09/05/2018

Overview

The Prokudin-Gorskii photo collection constitutes a variety of picture "negatives" produced by a special set up that captures the target subject through 3 differently colored lenses, intended to be used to make a colored image of the subject by overlaying the 3 resulting pictures in an aligning manner. In this project, we attempt to create a python program that automates the matching and aligning process so that each captured picture overlays on top of each other and reproduces a colored picture of the scene.

Technical Approach

Error-calculation

Two different error-calculating approaches are pursued in this project -- Sum of Squared Differences (SSD) and normalized cross-correlation (NCC),. Both produces an error-connotating value that can then be used to identify the most matching overlay setup between all films: by picking the specific displacement that creates the least amount of error. In the case of SSD, this is connotated by the smallest value, and in the case of NCC, this is connotated by the largest value.

Matrix shifting/displacement

When shifting/displacing matrices corresponding to a 1-color negative, the numpy function .roll() is not used due to the function having a behavior of rolling over values 'shifted out' of the matrix into the other side. A "matrix shifter" function called padCut(matrix, x, y, fillValue) is instead implemented -- the function "shifts a matrix" by pushing its entries horizontally depending on the x input and vertically using y input and filling those distances with the fillValue input.

Issues

1.

Initial implementation of the matching algorithm has both the aforementioned fillValue input of the padCut() function call being -1 and the error-calculation belimited to only matrices with values that are not -1 (essentially, masking the calculation to only parts of the original matrix still "visible" after shifting the matrice) in addition to normalizing the error by how many pixels were included in the calculation. However, this is prone to producing color-layer "match-displacement" calculations that "prefers" a lower number of pixels that still remain in viewing window, due to the lower number producing a higher possibility of a smaller number. This is especially impactive when using SSD and the pyramid-search, as there are less pixels to count over.

Further experimentations led to the removal of the masking and the setting of the fillValue input to 0.5, as 0.5 worked best for all pictures -- further study implies that 0.5 works best due to 0.5 being a "neutral" value, sitting right in the middle of the grayscale range. In contrast, setting the fillValue to a 0 will push the error-value calculation towards displacements that places darker parts of a color layer on top of the "empty"/fillValue-filled spots, while setting the fillValue to a 1 will push the error-value calculation towards displacements that places lighter parts of a color layer on top of the "empty"/fillValue-filled spots.

Before, fillValue set to -1, mask on.

After, fillValue set to 0.5, mask off.

2.

Each 3-color negatives come with a border that can affect calculations significantly, especially if the the 3-color negatives are split apart to 1-color negatives in a way that the borders don't align exactly. To fix this, simply cutting the border improves matching calculations significantly.

Before, without border cropping.

After, with border cropping.

3.

Alignment order (G to B then R to B vs G to B then R to G, etc.) matters, especially in situations where the color glass pictures have wildly different contrast levels. Most of these alignments used a specific order for alignment: G to B, and R to B. While this worked for most cases, it particularly fails in emir.tif.