In this project, we were tasked with combining the red, green, and blue filtered photos taken by Sergei Mikhailovich Prokudin-Gorskii in the 1900s of various scenes.
Because the various color channels were not all taken from the exact same position, the primary challenge of this project was to align the frames of the 3 channels.
To align the photos, we must somehow "match" the various frames. For this project, we must align the red and green channels with the blue channel.
I did this by using a normalized-cross-correlation (NCC) metric, which computes the similarity between two arrays, each normalized to eliminate any skew due to magnitude or variance differences. For this "matching" metric, a higher score is better. This metric is computed over a sliding window of displacements for each frame, and the best displacement (highest NCC score) is chosen as the final offset before merging the 3 frames.
For large images, naively doing this actually does not work, because computing the NCC over and over again for a 4000 by 3000 resolution photo is extremely expensive, especially if its over a [-20, 20] window (1600 times), which I chose. Instead, performing alignment for larger resolution photos was done using an image pyramid.
Since computing NCC 1600 times for high-resolution images is inefficient, I instead used an image pyramid. For this pyramid, I scaled images down by a factor of 2 until its width was under 400 pixels.
At this lowest resolution, it is relatively fast to compute the NCC across a 40x40 sliding window. As we go up in resolution, our new "center" is scaled up by a factor of 2. Then, we explore a new sliding window from [-2, 2] (25 checks), which is fine, even at the highest resolution.
All the images had 10% cut off from the borders before processing, since the black borders can affect the results of NCC, but logically should not contribute. Furthermore, any filler values created due to image shifting were ignored as well.
The below is added as an image because html doesn't support LaTeX...
As stated prior, the only difficulty was getting the "emir" image aligned. This is because, as can be seen below, the color channels look very different. In particular, the dress looks extremely bright on the top (blue) channel, gray in the middle (green) channel, and nearly black in the bottom (red) channel. This leads to problems when using NCC, which gives the output image shown on the right below. We can see that the red channel is not properly aligned to the blue and green channels, likely as a result of its lack of "signal" on the dress compared to the blue channel (which is nearly max signal/white on the dress).
On the other hand, the image preprocessed for edges looks similar in all channels. While the edges are certainly much stronger on the blue channel, they can still be clearly seen in the red and green channels. As a result, the matching algorithm on the edge preprocessed images works fine.
Offsets are specified according to the amount of shifting to the right (x) and down (y).
| Image name | Using NCC | Offsets (x,y) | Using Sobel with NCC | Offsets (x, y) |
| castle |  |
Red: (4, 98) Green: (3, 35) |
 |
Red: (4, 98) Green: (4, 35) |
| cathedral |  |
Red: (3, 12) Green: (2, 5) |
 |
Red: (3, 12) Green: (2, 5) |
| emir | |
Red: (-246, 99) Green: (24, 49) |
|
Red: (40, 107) Green: (24, 49) |
| harvesters |  |
Red: (14, 124) Green: (17, 60) |
 |
Red: (14, 124) Green: (17, 60) |
| icon |  |
Red: (23, 89) Green: (17, 41) |
 |
Red: (23, 90) Green: (17, 42) |
| lady |  |
Red: (12, 117) Green: (9, 55) |
 |
Red: (13, 120) Green: (9, 56) |
| melons |  |
Red: (13, 178) Green: (11, 82) |
 |
Red: (13, 177) Green: (10, 80) |
| monastery |  |
Red: (2, 3) Green: (2, -3) |
 |
Red: (2, 3) Green: (2, -3) |
| onion_church |  |
Red: (36, 108) Green: (27, 51) |
 |
Red: (35, 107) Green: (25, 52) |
| self_portrait |  |
Red: (37, 176) Green: (29, 79) |
 |
Red: (37, 176) Green: (29, 78) |
| three_generations |  |
Red: (11, 112) Green: (14, 53) |
 |
Red: (9, 111) Green: (12, 54) |
| tobolsk |  |
Red: (3, 6) Green: (3, 3) |
 |
Red: (3, 6) Green: (3, 3) |
| train |  |
Red: (32, 87) Green: (6, 43) |
 |
Red: (29, 85) Green: (2, 41) |
| workshop |  |
Red: (-12, 105) Green: (0, 53) |
 |
Red: (-12, 105) Green: (-1, 53) |