CS 194-26: Project 1 - Colorizing Prokudin-Gorskii

Background

Sergei Prokudin-Gorskii's RGB-filtered photographs of Russia offer an in-color glimpse of the early 1900's across the vast Empire. Although the technology to mesh the results from the R, G, and B filtered images into color did not exist at the time of Prokudin-Gorskii, we can use modern day simple image processing techniques to obtain the color image.

Alignment

Due to inconsistencies (ex: subject moves during portrait, shifts in background) between filtered images, there is a need for alignment between channels before combining to obtain our colored result. Treating alignment as a simple (x,y) translation problem, we can use a couple of methods to optimally align images.

Exhaustive Search

For smaller images (.jpg format), we can use a simple exhaustive search method to align our images. The middle 80% of the image was first cropped to ensure borders did not affect alignment. Over a [-20, 20) pixel window of displacements, filtered images were translated and matched to a base image using the Sum of Squared Distance metric [sum((image-base)**2)]. For these images, the blue filter was used as the base. Exhaustive Search with SSD was sufficient for proper alignment of smaller JPG images (shown below)

Green Offset: [3, 3] Red Offset: [3, 6], ~1s

Green Offset: [2, -3] Red Offset: [2, 3], ~1s

Green Offset: [2, 5] Red Offset: [3, 12], ~1s

Pyramid Speedup

However, for larger images (.tif format), our simple exhaustive search algorithm is too slow (~5 mins per image). Therefore, we need to implement a faster search procedure - using an image pyramid. An image pyramid consists of the same image resized to an extremely coarse resolution and then sampled at higher resolutions all the way to its original scale. By starting at the most coarse image and moving our way up to the original, as well as shrinking our window size as we move, we can minimize unneccessary information processing and best estimate image offset. Similar to the exhaustive approach, the middle 80% of the image was first cropped to ensure borders did not affect alignment. Results along with execution time are included below.

Green Offset: [4, 25] Red Offset: [-17, 0], 38s

Green Offset: [16, 60] Red Offset: [13, 124], 46s

Green Offset: [17, 40] Red Offset: [23, 89], 52s

Green Offset: [8, 55] Red Offset: [11, 122], 43s

Green Offset: [10, 82] Red Offset: [12, 176], 46s

Green Offset: [26, 52] Red Offset: [36, 108], 44s

Green Offset: [29, 79] Red Offset: [36, 175], 46s

Green Offset: [14, 54] Red Offset: [11, 111], 45s

Green Offset: [6, 44] Red Offset: [32, 87], 40s

Green Offset: [-1, 53] Red Offset: [-12, 104], 42s

Green Offset: [24, 49] Red Offset: [-384, 14], 39s.

Selected Images

Retrying Emir

Emir encountered poor results during pyramidal alignment. This was mitigated by using a different base channel, green, whose results are shown below.

Red Offset: [17, 57] Blue Offset: [-24, -49], 38s

Retrying Church

Church encountered poor results during pyramidal alignment. My assumption is due to the interference of the sky in alignment, which is a little blurry/unclear edges. This was mitigated by using the cropped central 60% of the image, whose results are shown below.

Green Offset: [4, 25] Red Offset: [-4, 58], 36s

B&W

Auto-Contrast Borders

On certain images, contrast seems low. Therefore, by applying autocontrasting, we can increase the quality of said images. This is demonstrated on melons and lady below. Note the aliasing along the border of the lady image - although the overall image quality has improved, due to the borders autocontrast is not optimized. With autocropping of borders, this autocontrast should be significantly better.