Sergei Mikhailovich Prokudin-Gorskii was a Russian photographer who pioneered the art of color photography. Back in 1907, the method he concocted of taking color images consisted of recording three exposures of a particular scene through red, green, and blue colored filters. Upon overlaying the three images with the correct color channels, the full-colored image would pop right out.

In this project, we are presented with the three exposures, as such:

We aim to detect the best alignment of the three grayscale channels such that when pieced together, the three exposures form as sharp of a color image as possible.

Main Project

Initial Methods

I first tried a naive method of overlaying each of the red and green exposures on top of the blue and computing a sum of squared differences (SSD) value to evaluate the alignment over a [-15, 15] window in X and Y. This worked decently for some images but very poorly for others, which either had massively different exposures between the channels (think of a blue sky in "monastery" taking up half of the image) or had large margins that impacted the SSD metric.

Here's an example that did not work with naive methods:

Improvements

In order to get a good alignment (excluding bells and whistles) with faster performance, I took the following steps:

• Cropped out 10% of the margins on each side to remove any harsh borders, which would provide inaccurate metrics when computing the SSD.
• Devised an image pyramid, recursively downscaling the image and at each depth performing a constant (much smaller) number of comparisons using the SSD.

Bells and Whistles

Better Features

I used a simple Sobel kernel and convolved the image to detect edges. This was particularly useful for images like "emir" and "monastery", which had large splotches of blue which brought the blue channel heavily out of alignment with the others. Running the alignment algorithm on the edge-filtered image finally produced a good alignment.

Here's what "emir" looked life after the sobel filter (this is scaled down and contrasted for clarity). The differences are much less pronounced in just the blue channel. While the patterns may still look different, it's now only outlines, not entire filled sections, that somewhat vary between the channels.

This is the result of preprocessing with the Sobel filter:

Automatic Cropping

This was the most time consuming part for me, as I experiemented with various methods before settling on one that worked best. Here are some examples of what I tried:

• Variances on rows of pixels
• Pixel intensity extremes
• IQR on rows of pixels
• Gradients between rows

This was difficult due to the fact that any metric I used comparing pixel variation could be too greedy and chop off the plain blue sky in "monastery", for example. Any gradient methods I would also have to be mindful of natrually occurring vertical and horizontal lines such as the ones from the door in "emir".

In the end, I opted to crop away any borders in addition to any bright artifacts formed from channel alignment. I combined a couple of methods in order to crop just the right amount from each picture with one algorithm (required some parameter tuning):

• Downscale any large images to below 400x400 pixels (affects next step)
• Apply Sobel filter to downsized images to get highly contrasted edges
• Enhance contrast and scale to (0, 1)
• Begin scanning at the row index which is 10% the image height, and work back upwards. Upon discovering a sharp edge (mean intensity falls above a threshold), crop everything including and beyond that row.
• Rotate image 90 degrees and repeat for each side

Here is the result, before and after cropping:

Automatic Contrasting

For contrasting, I just used a simple linear scaling of pixel intensities to fit between 0 and 1. This did not change much for many pictures, like Emir, who already has a white turban and a glorious black beard.

Results Gallery

Here is a gallery of all the results, along with their computed x/y shifts.