CS 194-026 Project 1

Project 1: Images of the Russian Empire
Colorizing the Prokudin-Gorskii photo collection

Overview: Sergei Mikhailovich Prokudin-Gorskii (1863-1944) captured images of the Russian Empire. His idea was simple: record three exposures of every scene onto a glass plate using a red, a green, and a blue filter. His RGB glass plate negatives, capturing the last years of the Russian Empire, survived and were purchased in 1948 by the Library of Congress (LoC). The LoC has recently digitized the negatives and made them available on-line. The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, three color channel images need to be extracted, placed on top of each other, and aligned so that they form a single RGB color image. However, the full-size glass plate images are very large, so the alignment procedure will need to be relatively fast and efficient.

Singlescale Alignment (Naive Approach): The images were divided into three equal parts and rolled such that the second and the third parts (G and R) were aligned to the first (B). The channels were aligned as the image was exhaustively searched over a window of possible displacements and scored using the normalized cross-correlation (NCC): (image1./||image1|| and image2./||image2||). The displacement with the best score was used to align the image parts. Using the singlescaling approach, images were produced as follows:

singlescale alignment, no crop G: [5 2], R: [12 3]

singlescale alignment, no crop G: [35 26], R: [-31 23]

singlescale alignment, no crop G: [34 -16], R: [35 35]

singlescale alignment, no crop G: [35 18], R: [35 25]

singlescale alignment, no crop G: [35 10], R: [35 35]

singlescale alignment, no crop G: [-3 2], R: [3 2]

singlescale alignment, no crop G: [3 1], R: [8 0]

singlescale alignment, no crop G: [23 35], R: [24 16]

singlescale alignment, no crop G: [7 0], R: [15 -1]

singlescale alignment, no crop G: [4 11], R: [34 6]

singlescale alignment, no crop G: [35 5], R: [16 -22]

singlescale alignment, no crop G: [35 15], R: [-20 -35]

singlescale alignment, no crop G: [35 35], R: [35 -20]

Multiscale Alignment: Due to the exhaustive singlescale horizontal and vertical circular shifts used to align the red and green channels with the blue channel, this search will become prohibitively expensive if the pixel displacement is too large. This is the case for high-resolution glass plate scans. To improve upon speed of the algorithm, I used a faster search procedure, the image pyramid. I used the image pyramid to represent the image at multiple scales which were scaled by a factor of 2. Starting from the coarsest scale (smallest image at the top of the pyramid) and proceeding sequentially to larger images on subsequent scales, the shift offset at each level was estimated with a scaling of the prior pyramid level, with the window search size scaled by that factor. Multiscaled images are shown below:

multiscale alignment, no crop G: [5 2], R: [12 3]

multiscale alignment, no crop G: [105 42], R: [49 24]

multiscale alignment, no crop G: [59 19], R: [123 19]

multiscale alignment, no crop G: [41 18], R: [89 24]

multiscale alignment, no crop G: [51 6], R: [111 10]

multiscale alignment, no crop G: [-3 2], R: [3 2]

multiscale alignment, no crop G: [3 1], R: [8 0]

multiscale alignment, no crop G: [77 29], R: [175 37]

multiscale alignment, no crop G: [7 0], R: [15 -1]

multiscale alignment, no crop G: [49 15], R: [108 10]

multiscale alignment, no crop G: [42 6], R: [85 32]

multiscale alignment, no crop G: [56 22], R: [117 30]

multiscale alignment, no crop G: [64 13], R: [136 23]

BELLS & WHISTLES

Better Features: Due to discrepancies in the exposure times of the RGB filters, the light intensities of the channels can vary. Depending on the resolution and light exposure of the image, these discrepanies can lead to improper alignment of the filters. To overcome this problem, various feature and edge detection methods can be used to better align the images. I used the Sobel filter method to detect edges and their orientation as it is computationally very efficient and yields reasonably accurate alginments. Using the Sobel filter technique, I computed the gradient approximation of image intensity through convolution of an integer filter moved both horizontally and vertically. The sobel filter functions by using two 3 by 3 kernels G_x and G_y defining the horizontal and vertical derviative approximations. The kernels are convolved with each 3 by 3 subsections of the original image to weight their intensities for gradient approximation. With comparison of the weighted kernels to the original image moving in horizontal and vertical directions, the best RGB filter alginment offset using the mean squared error metric is computed. Images are then aligned according to this offset.

Better Transformations: To further improve the image gradient approximation for better alignment, I used a third dimension for depth alignment. Using a depth proportional to the logarithm of image height and weight, the sobel filter was reapplied to compute the best offset for this dimension.

The sobel filter method with an added dimension for depth alignment to help better align 'emir.tif' is shown below:

Snow

multiscale, using Sobel Filter, no crop G: [105 42] R: [49 24]

Automatic Cropping through Border Detection: To detect the border of the image, the relative window image scan should not only capture the essential features of the image, but should also cut off the irrelevant boundaries. Converting the image to grey scale, horizontal and vertical image boundaries are detected by scanning the pixels within a range of the search window, separately for left, right, top, and bottom borders. Each row and column of the images are scanned and the maximum pixel luminance difference within that range for each scan is obtained. This is then compared to a threshold proportional to the sum of the pixel luminance differences within that range and the window search size to determine if the border is to be cropped and from which index. Below are the results of cropping the images with improvements where indicated:

crop improvement

Additional Images from the Prokudin-Gorskii Collection:

Below are additional image results using multiscale alignment, cropping, and sobel filtering:

Krestianskiia Dievushki G: [-14 10] R: [11 19]

Roses G: [53 1] R: [114 -24]