Yuhong Chen

Background

Sergei Mikhailovich Prokudin-Gorskii was a man truly ahead of his time. As early as 1907, he was convinced that color photography would be BIG. He travelled around the Russian Empire and took pictures of everything he saw, even though there was no way to print the color photographs.

The idea is simple: he took photographs of the same scene using a red, a green, and a blue filter. These produced glass negatives that recorded the brightness of the three color channels. If one were to align the three color channels in the right way, the color image would be reconstructed. In this project, we're going to do just that.

Approach

The basic assumption for aligning the color channels is that the aligned pixels in each channel would have a similar brightness. So by comparing the different channels and taking the Sum of Squared Differences, also known as the L2-norm, we can find the best offset for alignment by choosing the one with the least difference in brightness between channels. We call this approach the L2-loss approach.

For small images with about 400x400 pixels, the offset for each axis is usually less than 15. So we can easily do an exhaustive search for such images. But for larger images with more than 3000x3000 pixels, such a search becomes computationally expensive. To fix this, we adopt a pyramid search method, searching at smaller scales first, which allows us to search over large areas quickly, and then switching to larger scales to locate the exact offset.

Results

cathedral.jpg G:(2, 5), R:(3, 12)

monastery.jpg G:(2, -3), R:(2, 3)

lady.tif G:(9, 51), R:(11, 112)

melons.tif G:(10, 81), R:(13, 178)

harvesters.tif G:(16, 59), R:(13, 123)

icon.tif G:(17, 41), R:(23, 89)

Bells and Whistles

L2-loss on Edge Maps

After using the naive L2-loss, I decided to try using edge maps for alignment. I used the Roberts Edge Detection algorithm to generate an edge map of each image, and then applied L2-Loss to the edge maps. Most of the images were already pretty well aligned, so there wasn't much visible improvement for them, but the result for emir.tif aligned using the edge map was much better than the result from just L2-Loss:

aligned with plain L2-loss

emir.tif G:(24, 49), R:(55, 103)

aligned with L2-loss computed on the edge map

emir.tif G:(24, 49), R:(40, 107)

Zooming in on Emir's face, you can see that the plain L2-loss result created a halo-like artifact due to the red layer being misaligned, whereas the edge map computed loss produced a much clearer face.

However, using edge maps was detrimental when aligning images with a natural landscape background, as those images generated noisy edge maps that interfered with alignment. This effect is visible with the results for self_portrait.tif.

aligned with plain L2-loss

self_portrait.tif G:(29, 78), R:(37, 176)

aligned with L2-loss computed on the edge map

self_portrait.tif G:(29, 78), R:(0, 1)

Adaptive Histogram Equalization

As some of the pictures in the collection appeared to be over-exposed, I wanted to apply automatic contrast adjustment to the images. I used the "equalize_adapthist" function available in the skimage library to apply Contrast Limited Adaptive Histogram Equalization to the input images. The results yielded more natural looking images for some inputs, but generated a pastel-like texture for others.

For example, the output for workshop.tif looked more realistic after Adaptive Histogram Equalization, but the output for onion_church.tif gained a more artificial feeling.

Without Adaptive Histogram Equalization