Overview

In this project an algorithm is implemented that, given a 3-channel image, produces a color image as output. The algorithm is based on Sergei Mikhailovich Prokudin-Gorskii's method of photographing three grayscale exposures of a scene onto a glass plate using red, blue, and green filters and combining the images to create the color image. First a simple single-scale version is implemented, which searches over a user-specified window of displacements, and then a coarse-to-fine pyramid speedup for larger images is implemented.

Naive Implementation for Low Resolution Images

In the naive implementation each set of three images is first cropped on the left and right borders to remove unnecessary white and black space. Then, the set of images is split into three images, each representing one of three red, blue, and green filtered images. The images are further cropped on the top and bottom to remove unnecessary white and black space by detecting changes in pixel brightness. Finally, the images are aligned atop one another with blue, green, and red filter order.

Border Detection

For each side of the image, the border is detected by taking the difference between the previous pixel and the current pixel for each row or column containing the potential border. If this difference is greater than threshold 0.2, the last pixel was white, and there has been no significant change in brightness for the last five pixels, then a border has been found. If this difference is greater than threshold 0.2, the last pixel was black, and a border has already been detected, then the pixel index for the end of the border has been found, and the image is cropped at this index.

Alignment

The green filtered image is aligned to the red filtered image by shifting the green filtered image [-15, 15] pixels in the x and y axes and finding the best possible fit. The best fit is determined by Sum of Squared Distances, which is minimized.

Before and After

Offsets are [x, y] coordinates listed in order of green and then red shifts.

Offsets: [2, -4], [3, 3]
Offsets: [2, -14], [2, -8]
Offsets: [3, -5], [3, -2]

Image Pyramid Implementation for Higher Resolution Images

In the optimized implementation first images are cropped on the borders. Then, images are rescaled down four levels, and the optimal displacement is searched for in the entire image in the coarsest resolution. This alignment is propagated to the next level in resolution and the alignment values are multiplied by two, giving a set of displacement values to search through at this next level. The process is repeated until the first level of original resolution is reached. This process is faster since the set of values to search through is constrained by the values given at lower resolutions.

Before and After

Offsets are x, y coordinates listed in order of green and then red shifts from lowest to highest resolution image pyramid levels.

Green Filtered Image Offsets:
[1, 3]
[1, 5]
[2, 10]
[4, 20]
[8, 40]
Red Filtered Image Offsets:
[1, 3]
[1, 5]
[2, 10]
[4, 20]
[8, 40]
Green Filtered Image Offsets:
[0, 3]
[0, 6]
[0, 12]
[0, 24]
[0, 48]
Red Filtered Image Offsets:
[0, 3]
[0, 6]
[0, 12]
[0, 24]
[0, 48]
Green Filtered Image Offsets:
[1, 3]
[1, 5]
[2, 10]
[4, 20]
[8, 40]
Red Filtered Image Offsets:
[1, 3]
[1, 5]
[2, 10]
[4, 20]
[8, 40]
Green Filtered Image Offsets:
[2, 2]
[0, 3]
[0, 6]
[0, 12]
[0, 24]
Red Filtered Image Offsets:
[2, 0]
[4, 0]
[8, 0]
[16, 0]
[32, 0]
Green Filtered Image Offsets:
[0, 3]
[0, 6]
[0, 12]
[0, 24]
[0, 48]
Red Filtered Image Offsets:
[-1, 3]
[-2, 6]
[-4, 12]
[-8, 24]
[-16, 48]
Green Filtered Image Offsets:
[0, 0]
Red Filtered Image Offsets:
[0, 0]
Green Filtered Image Offsets:
[0, 0]
Red Filtered Image Offsets:
[0, 0]
Green Filtered Image Offsets:
[0, 0]
[0, 0]
[0, 0]
[0, 0]
[0, 0]
Red Filtered Image Offsets:
[-1, 0]
[-2, 0]
[-4, 0]
[-8, 0]
[-16, 0]
Green Filtered Image Offsets:
[0, 0]
[0, 0]
[0, 0]
[0, 0]
[0, 0]
Red Filtered Image Offsets:
[-1, 0]
[-2, 0]
[-4, 0]
[-8, 0]
[-16, 0]
Green Filtered Image Offsets:
[0, 0]
[0, 0]
[0, 0]
[0, 0]
[0, 0]
Red Filtered Image Offsets:
[0, 0]
[0, 0]
[0, 0]
[0, 0]
[0, 0]

Custom Images

Custom examples downloaded from the Prokudin-Gorskii collection. Run using naive algorithm.

Before and After

Offsets are [x, y] coordinates listed in order of green and then red shifts.

[2, 10] , [3, 15]
[1, 15], [1, 15]
[5, -15], [9, -7]

Bells and Whistles

Automatic Cropping

For the naive implementation, automatic cropping was implemented by detecting image borders. See above section for details.