Introduction
Sergei Mikhailovich Prokudin-Gorskii (1863-1944), a Russian photographer, as early as 1907, believed that photography was the wave of the future. He won Tzar's special permission to travel across the vast Russian Empire and took color photographs of everything he saw. He took photos of people, buildings, landscapes, railroads, bridges, thousands of color pictures! However, the time when he took the photographs, color cameras didn't exist. So he recorded three exposures of every scene onto a glass plate using a red, a green, and a blue filter and envisioned that this could produce colorful photos. However, his plan was never materialized. Luckily, his RGB glass plate negatives, capturing the last years of the Russian Empire, survived and were purchased in 1948 by the Library of Congress.
In this project, we will take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, we will extract the three color channel images, place them on top of each other, and align them so that they form a single RGB color image.
Method
Single-Scale Implementation: Naive Approach
Given the glass plates, we will first divide the picture into three parts for the R, G, B color plates. If we just stack the plates together, we will get a colorful but very blurry image. We can tell that obviously the three color channels do not properly align. In order to align the color channels, we need to move each channel a little bit. In the project, we will use the blue channel as the reference channel and match the green and red channels to it. In order to find the best displacement that will give us the best image result, we will align the images using some image matching metric. In the naive implementation, we will try the two methods of the Sum of Squared Differences (SSD) distance and normalized cross-correlation (NCC) to measure the heuristics since they are very simple.
Sum of Squared Differences(SSD)
$$||img_1 - img_2||{^2_2}$$
The sum of squared differences measures the difference between two color channels. The smaller the calculated value, the better the alignment.
Normalized Cross-Correlation(NNC)
$$\frac{img1}{||img1||{^2_2}}\cdotp\frac{img2}{||img2||{^2_2}}$$
The normalized cross-correlation calculates the correlation between two color channels. The bigger the calculated values, the more correlated the two channels are, the better the alignment.
When we use the SSD as the metric, the green channel displacement is (-6, 0) and the red channel displacement is (9, 1). However, even though the algorithm is working, the output image is still very blurry.
|
It's not hard to notice that the borders of the image might be harming our result. So we will align using only the internal 80% of the image by cropping out 20% of the border when we align. We can see from below that the result is a lot more precise.
|
|
Image Pyramid
In the single-scale implementation, we searched over a small range of displacement [-15, 15]. However, for higher resolution images, even searching over a small range can be very computationally expensive. Thus, we will implement a image pyramid to decrease the search range. Image pyramid speeds up the process by rescaling images and have us work with smaller copies of the image. We will first compute how many levels our image pyramid will have. We will represent an NxN image as a pyramid of $$1x1, 2x2, 4x4, ..., 2^kx2^k$$ images. Our algorithm will first recurse on the smallest scale, find its optimal alignment and down to the bigger images and do the same.
Provided Examples
|
|
|
|
|
|
|
|
|
|
|
|
|
Other Examples
|
|
|
|