CS194-26 Project 1: Images of the Russian Empire

Gurkaran Singh Goindi

In the early 20th century Sergei Mikhailovich Prokudin-Gorskii provided a precursor to color images. His idea was to photograph the same scene three times on a glass plate: first using a red filter, then using green filter, and finally third using a blue filter. He envisioned projectors to be able to display color images using his glass plates, but was never able to do it. Although he was never able to visualize and share his photos in color, we can use modern computational photography techniques to do so.

In this project, in order to more accurately align the images, I decided to crop 30% of the image, mostly from border areas, and then run the various matching algorithms on the remainder of the image. Once the requisite filter offsets were obtained, I would apply them to the original image.

Part 1: Exhaustive Search

Exhaustive Search is feasible for all of the smaller images, specifically the jpgs in the provided image folder. Since the image dimensions are smaller, the possible pixel shifts are much smaller, meaning most shifts will fall within the range of [-20, 20]. So we can use exhaustive search and try all possible shifts, returning the best shift according to the SSD. The algorithm rolls the image in both the x and y dimensions, trying every shift in the range of [-20, 20] for each dimension and caclulating the SSD with respect to the green channel. Although there wasn't significant difference between aligning with respect to the green and blue channels, because I picked the green channel to align with respect to for Part 2 of the project, I decided to keep things uniform here as well. Here are the smaller jpg images, and their returned offsets.

Cathedral

Red Offset: (1, 7), Blue Offset: (-2, -5)

Tobolsk

Red Offset: (1, 4), Blue Offset: (-3, -3)

Monastery

Red Offset: (1, 6), Blue Offset: (-2, -3)

Part 2: Pyramid Search

For larger images, exhaustive search is infeasible. Due to the images' huge sizes (over 2000x2000pixels in most cases) the shifts in alignment tend to be a lot more than 20pixels in either direction. Therefore I used the idea of image pyramids, whereby each color is iteratively scaled down by a factor of 2, until the image is of height < 250 pixels. Then, I would recursively try to find the displacement using exhaustive search on each level of the pyramid, starting from the lowest resolution to the highest resolution, and each round multiply and add the displacement of the previous round to the next round. Eventually, I would find the displacements, apply them using np.roll, and the image would be ready.

I decided to the align the images with respect to the green channel, as I realized in high-contrast such as Emir, the blue was much more visible and highly contrasted, meaning that any other filter as a base wasn't quite giving the result. Any other issues I had with alignment, were somewhere alieviated when I decided to find the displacements on a cropped image (cropped 30%), and then applied the displacement to the original image.

A general explanation for misalignments can be found at Melons.

Church

Red Offset: (-16, 32), Blue Offset: (0, -2)

Emir

Red Offset: (16, 48), Blue Offset: (-32, -48)

Harvesters

Red Offset: (0, 8), Blue Offset: (-16, 64)

Icon

Red Offset: (0, 6), Blue Offset: (-16, -48)

Lady

Red Offset: (0, 8), Blue Offset: (-16, -48)
There is a slight issue in Lady's alignment, I believe this is because the blue here is much more poignant than the other colors, thereby making the blue color have too much of a displacement.

Melons

Red Offset: (0, 10), Blue Offset: (-16, -80)
The issue of misalignment is greater here, probably because the colors are a lot more nuanced. Interestingly, higher contrast images such as Emir, Kurmy, Church, Train, and Onion Church have come out much clearer as per my program. I'm not sure of the reason for this, but I attribute this to an overactive blue filter, as in all the other misaligned images (such as Self Potrait, Lady, and Harvesters) the blue offset is significantly higher than the red offset, which is often near 0.