In this project, we will be colorizing the glass plate negatives taken by Prokudin-Gorskii across the Russian Empire from over 100 years ago and using image processing techniques to align and colorize them.
We will accomplish this by first splitting the glass plate negatives into their 3 subimages, each representing an image taken with a particular color filter (B, G, and R respectively). We will call these split up subimages color channels.
Because each color channel image is very slightly different from others (i.e. due to camera or image movements), we cannot simply naively project the 3 color channels on top of one another. Rather, we need to determine the pixel displacements in the x and y dimensions that result in the most optimal color image result.
How we define the "optimal alignment" matters. In this case, we chose to use the SSD, or sum of squared differences metric, where we compute the pixel-by-pixel differences between one image and another, square them, and sum them up across the entire image to generate this metric (note that the lower the SSD is, the more aligned the images presumably are i.e. not a lot of difference across the images). The key here is that we are focusing on comparing the "alignment" of 2 images / color channels at a time (R and B then G and B or B and G and then R and G). In general, throughout our experimentation, we have found the SSD metric to be rather effective as a metric guide for aligning the color channels.
For the smaller JPEG images, we were able to run an exhaustive search algorithm which simply
exhaustively analyzes a "window" of possible pixel shifts to align the images and outputs
the (x, y) displacement vector with the lowest SSD across the two color channels we're comparing
at that given point in time.
For these smaller JPEGs, we found that we only needed to search within a [-15, 15] pixel window
in order to determine fairly good alignments for our images. We've included the results of our
exhaustive search algorithm on the JPEG images in our examples (along with their optimal displacement vectors)
in the Results on Example Images section below.
For the larger, higher-resolution TIF images, the displacement vectors would be large enough such that it was no longer
feasible to run exhaustive searches at that scale of window size. We needed a better, faster approach.
Rather than exhaustively search a large window of the initial image, the approach we decided to take relied on
constructing an image pyramid of different image resolutions (varying by factors of 2) and search window sizes. That is, we built a recursive
function that downsized the image until we hit a base case (e.g. the image getting too small or hitting a defined limit of recursive layers)
and at each stage, we ran an exhaustive search window that was "informed" by the image one level coarser than it.
The key idea here is that we would still be doing exhaustive searches, but instead of just naively considering one window centered on (0, 0),
through recursion we would consider multiple windows at multiple scales, the result of each alignment helping to inform where we should start the
next window that we ran. One key difficulty I encountered with this was deciding how to scale up / down the search window sizes as we were scaling up /
scaling down the image size. Course staff helped me here in encouraging me to think about how we'd likely want smaller changes in the displacement vector
later in the recursion than earlier, which helped me come up with a dynamic search window adjusting approach.
We've included the results of our image pyramid search algorithm on the TIF images in our examples (along with their optimal displacement vectors)
in the Results on Example Images section below. We've also included the results of this algorithm on a few other images (not in the project's
example set) from the Prokudin-Gorskii collection in the Results on Additional Images section below.
In this section, we will show the results of our algorithms on the images in our example set as well as
state the displacement vectors we used to align each pair of color channels before projecting them together.
We will also explain some of the difficulties encountered in aligning some of the images underneath each respective image.
Aligning the green channel with the blue channel: (2, 5)
Aligning the red channel with the blue channel: (3, 12)
Aligning the green channel with the blue channel: (2, -3)
Aligning the red channel with the blue channel: (2, 3)
Aligning the green channel with the blue channel: (2, 3)
Aligning the red channel with the blue channel: (3, 6)
Aligning the green channel with the blue channel: (4, 24)
Aligning the red channel with the blue channel: (-4, 58)
This one was a little difficult because I noticed that later calls to my alignment functions were messing with a perfectly good alignment I found on a coarser level of the image. It was through iterating on this that I got the idea of dynamically adjusting window sizes to be smaller at higher-res stages (and sometimes even not running alignment on the highest resolution, skipping that stage altogether and trusting the next coarsest level).
Aligning the blue channel with the green channel: (-24, -48)
Aligning the red channel with the green channel: (18, 56)
Emir is rich in color and strong with blues, which made it a challenging task to find the best alignment. Through some experimenting, I found that aligning on the green layer (rather than the blue layer, as I had done previously) resulted in better outputs. I also noticed that the most relevant parts of the image (his face, his body & dress) were the most relevant parts to focus on aligning, which gave me the inspiration to run the alignment on cropped versions of the images (even if I would stack the full images), making sure the most relevant information was being prioritized with the SSD alignment algorithms.
Aligning the blue channel with the green channel: (-18, -60)
Aligning the red channel with the green channel: (-2, 64)
The large multitude of faces made aligning this image challenging. I experimented with various different window sizes before I had gotten this one down. Solving Emir also helped resolve this.
Aligning the blue channel with the green channel: (-18, -40)
Aligning the red channel with the green channel: (6, 48)
Aligning the blue channel with the green channel: (-8, -50)
Aligning the red channel with the green channel: (4, 58)
Aligning the green channel with the blue channel: (10, 82)
Aligning the red channel with the blue channel: (12, 178)
Many of the melons are similar in color and could result in misleading results from SSD computations. What helped me solve this one was being more restrictive on the permitted window size I was using as well as forcing fewer layers of recursion (preventing my algorithm from essentially overthinking).
Aligning the blue channel with the green channel: (-26, -52)
Aligning the red channel with the green channel: (10, 58)
Aligning the green channel with the blue channel: (29, 78)
Aligning the red channel with the blue channel: (37, 176)
The many rocks, grasses, and natural details had made aligning this image challenging. I employed a similar approach to melons, where I focused on a more constrained window size to explore to determine the optimal offsets.
Aligning the blue channel with the green channel: (-14, -50)
Aligning the red channel with the green channel: (-2, 58)
Aligning the blue channel with the green channel: (-6, -42)
Aligning the red channel with the green channel: (26, 44)
Aligning the blue channel with the green channel: (0, -52)
Aligning the red channel with the green channel: (-12, 52)
Here, we feature the results of our algorithms on some other images in the Prokudin-Gorskii collection that were not in our example set. I purposefully selected high-res TIFs for each of these images, as I had found the JPEG versions of these examples were so low in resolution that the image was a bit unclear. Enjoy :)
Aligning the blue channel with the green channel: (-6, -32)
Aligning the red channel with the green channel: (2, 48)
Aligning the blue channel with the green channel: (4, -24)
Aligning the red channel with the green channel: (-4, 76)
Aligning the blue channel with the green channel: (0, 0)
Aligning the red channel with the green channel: (34, 0)