Project 1: Images of the Russian Empire

Michael Gibbes, cs194-26-abn

Part 1: Single-Scale Implementation

My first implementation uses the brute force way of colorizing the Prokudin-Gorskii photos. Below are several low-res originals from his collection:

These images are organized in BGR format, where the top photo is blue filtered light, etc. I aligned the three channels automatically by applying a sum of squared differences (SSD) equation:

ssd = sum_i(sum_j((im1[i][j] - im2[i][j])**2))

I test an offset of plus or minus ~20 pixels, using the alignment with the smallest SSD.

However, it seemed I was missing something important. Look what happened when applying the same algorithm to a picture of a monastery.

It turns out that the aligning algorithm is correct, but sometimes it fits to darker portions of the photo. In this case, the edges of the photo.

I clipped off 10% of the pictures to remove the black and white borders, which seemed to do the trick.

That concludes my single-scale search for aligning the RGB channels of these photos. The next part will be focusing on a method for making this process more efficient.

Part 2: Multi-Scale Implementation

The implementation with single-scale is costly when iterating over images with more than 400x400px sizes -- exponentially so. One practical optimization is simply to scale down the image, like a pyramid.

For example, here are the pyramids corresponding to one image:

Once pyramidized, the key to optimal search is traversing the pyramid from top to bottom. I set my search box to be the current level height + 1. This value is arbitrary but works surprisingly well. At the largest image level (0), the search is no more than 1 pixel, or iterating over the image 9 times. When traversing down a level (i.e. searching larger images), I start from the offset of the upper level times 2 to account for the scale. Search takes no longer than 10 seconds on average and is accurate in general.

Here are a few more examples of this algorithm applied purely.

Per some suggestions that arose from discussing with some other students in this class, I realized that the green channels were actually better as an alignment base. You can see the difference in the photos of the emir.

Part 3: Bells and Whistles

Auto Contrast

For auto-contrasting, my process involves shifting the entire image down until the minimum value is 0.0, and then stretching it out so that the maximum value is 1.0. The effect is more noticeable on darker areas, like the main dome of this church.

Auto Balance

To enable automatic white balancing, I use the following process

1. Approximation of the illuminant... Take the average color to receive an R', G', and B', expressed as floating points between 0 and 1.
2. Calculate the proportion that each float dominates the illuminant to get 3 integers. For example, proportion r = R' / (R' + G' + B').
3. Multiply every pixel by the inverse of its proportion (r, g, or b) multiplied by 3. A pixel containing exactly the average color should now be precisely balanced as a shade of gray.

This method works quite well for images taken inside or with artificial lighting such as this sword display in Zlatoust.

It fails when an image already contains a rich, saturated color, like this very blue sky in which the algorithm has a distinct bias for red.

And one more example...

Final: Results on All Images

Here are the final results on running my algorithm over every image, plus a few of my own I found from the Library of Congress. Underneath each image is its alignment.