CS194-26 Project 1: Images of the Russian Empire

Overview

The goal of this project was to align three seperate images that represent the B, G, and R channels of a single color image. Since these images were taken seperately before the invention of color photography, they are given as three seperate areas on a single sheet of film, as shown below.

Approach

In order to find the best method of aligning the three image channels, I tried various correlation functions and methods of searching the solution space.

Correlation / Error Functions:

• Sum of Squared Differences (SSD), a simple metric where we try to minimize the Frobenius norm of the difference between two channels.
• Normalized Cross-Correlation (NCC), a metric where we try to maximize the dot product between two channels after first normalizing them.

Search Algorithm:

• Local search, where the error is calculated within an NxM search area and the images are translated to find the optimal offset.
• Pyramid search, which is the same as local search, except an image pyramid is first constructed. This allows us to more exhaustively search the downsized images and move up the pyramid, further searching within the promising areas of higher resolution images.

The difference between the error functions that I tested turned out to be quite minor, while the search algorithms showed a huge difference. The problem is nearly intractable using the local/naive search method, while constructing an image pyramid allows the search to be effectively exhaustive while taking a fraction of the time (less than 30 seconds for each image). In terms of how well the images align, it seems that NCC is a better error metric to use, as SSD exhibits some minor misalignments in a few of the images I tested.

A major improvement to the algorithm comes from recognizing that the edges of the channel images are quite noisy, so I modified the error functions to crop the shifted images before computing the error. This allows all but one test image to align well with the following simple algorithm:

1. Load the image and crop it evenly into the three channels.
2. Construct an image pyramid for all three channels.
3. Run a pyramid search using the NCC error function between the b and g channels, cropping the edges of each image. Do the same for b and r.
4. Combine the three images into a single color image.

The results for this on the sample images are:

Offsets: b = (0, 0), g = (5, 2), r = (12, 3)

Offsets: b = (0, 0), g = (49, 24), r = (269, -210)

Offsets: b = (0, 0), g = (60, 17), r = (124, 13)

Offsets: b = (0, 0), g = (41, 17), r = (89, 23)

Offsets: b = (0, 0), g = (55, 8), r = (117, 11)

Offsets: b = (0, 0), g = (82, 11), r = (178, 13)

Offsets: b = (0, 0), g = (-3, 2), r = (3, 2)

Offsets: b = (0, 0), g = (51, 27), r = (108, 36)

Offsets: b = (0, 0), g = (79, 29), r = (176, 37)

Offsets: b = (0, 0), g = (53, 14), r = (112, 11)

Offsets: b = (0, 0), g = (3, 3), r = (6, 3)

Offsets: b = (0, 0), g = (43, 6), r = (87, 32)

Offsets: b = (0, 0), g = (65, 12), r = (138, 22)

Offsets: b = (0, 0), g = (53, 0), r = (105, -12)

As shown above, the only example image that is not aligned properly with this method is emir.tif, most likely due to the colorful nature of the Emir's clothing.

Bell / Whistle

In order to fix Emir, I tried using an edge filter based on the Prewitt operator, which aligns the image based on the edges. This produces:

Offsets: b = (0, 0), g = (49, 24), r = (107, 40)

This method works very well on Emir without causing any issues in the other example images.

More Test Images

For the test images, I chose a few random ones from the Prokudin-Gorskii collection:

Offsets: b = (0, 0) g = (31, 20) r = (82, 35)

Offsets: b = (0, 0) g = (25, -13) r = (71, -21)

Offsets: b = (0, 0) g = (79, -12) r = (165, -34)