Project 1 - Images of the Russian Empire

In this project, I take Prokudin-Gorskii's color channels and combine them into color images using image processing techniques. In addition, I also implement simple image improvement algorithms to boost the quality of the output.

Approach

Part 1: Color Matching

The overall idea behind combining the color channels involves finding the optimal offset for each channel, such that the scans match up well. The naive approach is to simply perform a simple convolution-style algorithm with two channels, where every offset is attempted and the best offset is selected. I use the normalized cross correlation as my metric for gauging match quality, although in my tests the sum of squared differences did just as well.

While this works well on small images, this does not scale well, especially in 2 dimensions, so I turn to a more complex pyramid approach for larger images. I first downsize each channel to roughly about 30 x 30 pixels, and solve on this coarse scale using the naive approach. I then solve on progressively finer scales, by propagating the calculated offset from the previous level and searching only in the vicinity of the propagated offset. I use a factor of 2 for my image pyramid in my tests, which seems to work well.

Initially, I had some problem matching select images - however, simplying cropping a small amount from the border of each channel seems to solve that issue.

Part 2: Bells and Whistles

In the below examples, the left image is the before, the right is the after.

Automatic Cropping

Many of the final matched pictures have unpleasant colored borders, from either the original scans or alignment artifacts. I first detect the borders by identifying the regions of the channels that match poorly. I take the difference between the channels, and then apply a thresholding operation to accentuate the borders. I then identify the rows and columns that differ significantly, and crop them from the image.

Automatic Contrasting

Many of the matched images appear to have poor contrast or appear blurred out, so image quality is vastly improved with introduced contrast. We do this via histogram equalization, specifically OpenCV's CLAHE function.

Automatic White Balancing

All of the matched images appear to have a heavy brownish-yellow tint. In order to combat that, we perform white balancing, using the average color as the illuminant, and shifting that illuminant to gray, making the images appear much more natural.

Better Features

The naive approach for features to use in color matching is simply the channel's brightness values. However, each channel will always have slightly different brightness values, causing alignment issues on some images. A better approach is to use image edges as features. In my case, I take the Laplacian of the image in order to acquire the edges.

Better Transformations

The color matching algorithm above only finds a (X, Y) displacement for matching. However, in the real world, other changes like rotation or even perspective shifts can cause issues. Thus, we need to somehow find a 3x3 matrix, also known as a homography, that maps one channel's positioning to another channel's positioning. We do this by using an algorithm called SIFT to find interesting features in each channel. Next, we match the features of each channel against each other, and calculate the homography using the locations of matching features. We can then simply apply the homography, and get a better match. Unfortunately, for some images the homography didn't seem to work as well, so all the other examples are done using the above pyramid matching.

Noise Reduction

Many of the input images appear very noisy, so I decided to try applying noise reducing functionality in order to combat that. I use a wavelet denoising algorithm, specifically skimage's denoise_wavelet function. These functions work by transforming the RGB data into a wavelet domain, where Gaussian noise is much more easily recognizable and therefore easily removed. The difference isn't the most noticeable - the image is roughly adjusted by about 2 percent.

Solar Dynamics Observatory

We can also combine channels from another source. In the below example, we pull different images of the sun in ultraviolet wavelengths, perform color correction and combine them into one photo. On the left you can see the source channels, and on the right you can see the result: the top right matched the top left at offset (-1, -13), bottom right matched at (-17, -18), and bottom left matched at (-16, -23).

Results

Example Images

Green: (5, 2), Red: (12, 3)

Green: (-3, 2), Red: (3, 2)

Green: (3, 1), Red: (8, 0)

Green: (7, 0), Red: (15, -1)

Green: (49, 24), Red: (106, 41)

Green: (60, 18), Red: (123, 14)

Green: (39, 16), Red: (89, 23)

Green: (56, 9), Red: (119, 13)

Green: (77, 29), Red: (175, 37)

Green: (53, 13), Red: (113, 10)

Green: (42, 8), Red: (86, 33)

Green: (57, 22), Red: (117, 29)

Green: (65, 11), Red: (137, 21)

Selected Source Images

Green: (54, 20), Red: (115, 25)

Green: (58, 11), Red: (139, 19)

Green: (-3, 22), Red: (46, 45)