Colorizing the Prokudin-Gorskii Photo Collection

CS 294-26 • Fall 2020 • Project 1

Shubha Jagannatha




Overview

Prokudin-Gorskii is a Russian photographer who worked towards color photography in as early as 1907. In this project, I've taken his collection of photos which captures various people and places in the Russian Empire and worked to synthesize three images taken with red, green, and blue filters into a single color image. The difficulty in this project is in developing an algorithm to align these photos.


Approach


Sum of Squared Differences

Two popular methods in quickly aligning photos which have a relatively small amount of alignment correction needed are:
  1. Sum of Squared Differences (SSD)
  2. Normalized Cross Correlation (NNC)
In this project, I've implemented SSD which is simply taking the offset which minimizes the sum of the squared differences between two images. In my approach, I took pairs of images (red + blue or green + blue), specified a range of offsets to check (e.g. -15 to 15), and specified an axis to move my image on (vertical or horizontal). Then, I would shift my image according to the specified offset, calculate the ssd for that offset, and keep track of the image and offset with the minimum ssd so far. I did this functionality twice: once along the vertical axis and once along the horizontal axis. After finding the r, g, and b images with aligning horizontal and vertical offsets, I stacked the images to arrive at the final color images.

Here are all of the jpg images before and after they were successfully aligned.


Before
After, Offset R-B: (-1, 8), G-B: (-1, 1)
Before
After, Offset R-B: (2, 3), G-B: (2, -3)
Before
After, Offset R-B: (3, 6), G-B: (3, 3)

Coarse-to-fine Pyramid Speedup

To handle much larger images in the .tif format, a coarse-to-fine pyramid speedup was needed. This consists of taking multiple versions of the image with each new image a fraction (alpha < 1) smaller than the previous image. Each new image size is a "level" of the pyramid and in my implementation, I typically used 10 levels with an alpha of 0.5. Starting from the smallest level, I searched for the offset needed to align the images using the user inputted start and end ranges. Then, I moved on to the next smallest level by dividing the previous offset by alpha and searching the range of pixels surrounding that offset (prev_offset + start, prev_offset + end). I repeated the pyramid aligning twice, once for the vertical offset and once for the horizontal offset.

Here are some highly successful images which resulted from the pyramid alignment. The last four images were heavily cropped (by 200 px) which gave a much better output.

Provided Course Images




Offset R-B: (4, 98), G-B: (3, 35)
Offset R-B: (121, 22), G-B: (76, 16)
Offset R-B: (12, 178), G-B: (9, 82)
Offset R-B: (7, 106), G-B: (5, 52)
Offset R-B: (91, 32), G-B: (41, 4)
Offset R-B: (111, 11), G-B: (8, 45)
Offset R-B: (194, 3332), G-B: (4, 150)
Offset R-B: (13, 123), G-B: (16, 57)
Offset R-B: (175, 36), G-B: (30, 82)
Offset R-B: (36, 108), G-B: (26, 51)


Extra Images

Offset R-B: (17, 8), G-B: (9, 2784)
Offset R-B: (20, 129), G-B: (18, 60)



Failure Cases: The Emir

One apparent failure case is the Emir. My image alignment tool struggles to align this image because the value of the r, g, and b images varies significantly across images. This is mostly due to his blue clothing which has a very light value in the b image and has a very dark value in the r image (shown below). This difference means the images do not have the minimum ssd when they are properly aligned, so the output is misaligned.
Blue Filter Image
Red Filter Image