CS194-26: Image Manipulation and Computational Photography

Programming Project #1: Images of the Russian Empire

Allyson Koo

Project Overview

This assignment involved manipulating the Prokudin-Gorskii glass plate images to automatically produce reasonably accurate colorized versions of the images. The source of these glass plate images were created by a photographer, Sergei Mikhailovich Prokudin-Gorskii, who photographed the Russian Empire by recording three exposures of every scene he desired onto glass plates using three different filters: a red, a green, and a blue. This assignment involved using the digitized versions of these glass plate images, extracting the different color channel images, stacking them, and finding the alignment that properly produced a single RGB color image.

Implementation

Single-Scale Algorithm

In this project I utilized the suggested libraries, scikit-image and numpy, to extract the color channel information from the digitized images and to manipulate the resulting matrices of numbers. To find the proper alignment, I first implemented a simple single-scale algorithm that tried all possible displacements within a user-specified window (in this case, 15 pixels) and measured which alignment was best by using the L2 norm. I aligned the green and the red images to the blue image. The following images were produced with this naive implementation:

JPG Images

Cathedral.jpg Offset: Green: [1, -1] Red: [7, -1]

Monastery.jpg Offset: Green: [-3, 2] Red: [3, 2]

Nativity.jpg Offset: Green: [3, 1] Red: [7, 1]

Settlers.jpg Offset: Green: [7, 0] Red: [14, -1]

One issue that arose with this naive implementation was with the Monastery.jpg image. The corrected image has been displayed above, but when just running the simple algorithm on the raw input, the following image had originally been produced:


Original output (imperfect alignment)


Original image

As shown, the image was imperfectly aligned by running the naive algorithm on the raw input. This was fixed by cropping the edges (5% off each side) of the image. This helped to produce the correctly colored version of the Monastery.jpg file displayed in the JPG images section above. A possible reason why this was an issue with this picture is that the actual contents of the picture are very light in comparison to the black border (as shown in the original image). Thus, the algorithm could be focusing more on aligning the black borders rather than the actual contents of the picture. Cropping the border off (5% from each edge) allows the algorithm to match the actual contents of the picture instead, producing the better colored image shown in the JPG Images section.

Image Pyramid

For larger images, the single-scale implementation proved to be too slow due to the number of different possible alignments required. Thus, to speed up the algorithm and make it feasible to use on larger images, I implemented a coarse-to-fine image pyramid. First, I started by aligning a vastly scaled down version of the image (i.e. 2^-5 scale of the original). Then, once the small version of the image is properly aligned, I used the calculated displacement as a starting point for the next layer of the image (one that is twice as big as the previous one). I did this recursively until finally the full-sized image was properly aligned.

TIF (Larger) Images

Emir.tif Offset: Blue: [49, 24] Red: [0, -200]

Harvesters.tif Offset: Green: [60, 16] Red: [124, 13]

Icon.tif Offset: Green: [40, 17] Red: [89, 23]

Lady.tif Offset: Green: [53, 8] Red: [117, 10]

Self_portrait.tif Offset: Green: [78, 28] Red: [176, 36]

Three_generations.tif Offset: Green: [54, 11] Red: [112, 9]

Train.tif Offset: Green: [43, 5] Red: [87, 31]

Turkmen.tif Offset: Green: [56, 19] Red: [115, 26]

Village.tif Offset: Green: [64, 11] Red: [137, 21]

One issue that arose with the generic image pyramid implementation was with the Emir.tif. Originally, the following image was produced:

Original output (incorrect alignment)

The failure of the image pyramid algorithm to properly align this image was due to the fact that the images had different brightness levels. The blue filter was significantly darker than the others; causing issues when we tried to align the other colors to the blue. This problem was solved by trying to match the red and blue colors to green instead; since the green was of medium intensity and thus could be correctly matched with both the red and blue to produce the correct output shown above in the TIF (larger) images section.

Additional Images

Cliff_reflection.tif Offset: Green: [11, 46] Red: [29, 73]

River.tif Offset: Green: [18, 16] Red: [89, 28]

Round_hill.tif Offset: Green: [40, 4] Red: [131, 6]

Bells & Whistles

Better features: I tried aligning several of the images using edges (using skimage.filters) instead of using RGB values. While many of the images produced were of similar quality to those produced by matching RGB, there were a few that were noticeably improved by using the edge matching instead of color matching:

Non-cropped Monastery.jpg output using RGB alignment

Non-cropped Monastery.jpg output using edge alignment

In this example, the output on the left was produced by running the simple single-scale algorithm on the non-cropped raw input using RGB alignment. The image on the right was produced by running the same algorithm, but passing in the result of detecting horizontal and vertical edges instead of the raw color data and aligning based on that. This had a noticeable improvement for this case.

Cropped Emir.tif output using RGB alignment

Cropped emir.tif output using edge alignment

In this example, the output on the left was produced by running the image pyramid algorithm on the cropped raw input using RGB alignment. The image on the right was produced by running the same algorithm, but passing in the result of detecting horizontal and vertical edges instead of the raw color data and aligning based on that. This also had a noticeable improvement for this case.

Automatic contrasting: I tried to adjust the contrast of several of the images using the function skimage.exposure.equalize_adapthist. This produced the following noticeable improvements on several images:

Monastery.jpg without contrast adjustment

Monastery.jpg with contrast adjustment

Nativity.jpg without contrast adjustment

Nativity.jpg with contrast adjustment

Self_portrait.tif without contrast adjustment

Self_portrait.tif with contrast adjustment