CS 194-26

Project 1: Images of the Russian Empire

Colorizing the Prokudin-Gorskii Photo collection

Chelsea Ye, cs194-26-agb

Overview

The goal of this assignment is to create colored images given digitized three black-and-white Prokudin-Gorskii glass plate images taken with red, green, and blue filters. We are expected to align the three-color images with as few visual artifacts as possible using image processing techniques.

Approach
Exhaustive Match

For exhaustive search, I shifted red and blue plates pixel by pixel to find the best match with green plate. The search is done over a window the offsets of [-50, 50] pixels. Each match is scored using Sum of Squared Differences (SSD) distance and take the offset with the lowest SSD.

Pyramid Match

When processing high-resolution glass plate scans, exhaustive search will consume too much time as the pixel displacement is too large. In this case, I implemented pyramid search to increase the efficiency.
I compressed the image to multiple scales with compressing factor of 2 until the smallest layer is less than 64 x 64 pixels. Then exhaustive match is done sequentially starting from the last level (the coarsest scale), update the offset, move to the next level (with higher resolution) and start searching from the previous offset. Repeat match until it arrives the original layer.
The initial scoring search with offsets in range [-20, 20] to ensure successful alignments at lowest dimension. Beyond the initial scoring, the search window is limited to [-3, 3] instead, greatly reducing computational cost.

Problems

I started pyramid approach with matching green and blue plates to red at first. As I ran the pyramid algorithm on the images, I found out it did not achieve a good performance on some of the images. Images such as emir had great differences in brightness among three color plates, which makes them difficult to match. Specifically, blue plate oftenly has a lower brightness while the red plate has the greatest brightess. Thus I switched the approach to matching red and blue plates to the green one instead, and it did come out with much better performance.

Bells & Whistles

I applied automatic cropping by edge detection to make the image more alignable. I use Canny edge detector to detect borders across three color channels, crop out the borders, and normalize the image size across three channels.
After color match, I applied a final cropping to the composite colored image to remove any extra border left. The final images came out neatly in this way.

Results

The results are shown in two categories: (1) applying pyramid match only, and (2) applying auto-cropping by edge detection before matching. Results include all example images and three extra images from Library of Congress.

File Without Edge Detection With Edge Detection
emir
blue: (-7, 3) red: (9, 116)

blue: (-42, 47) red: (17, 57)
harvesters
blue: (3, -118) red: (-3, 65)

blue: (-24, 28) red: (-6, 65)
self_portrait
blue: (2, -50) red: (7, 98)

blue: (-48, 50) red: (12, 98)
icon
blue: (-16, -42) red: (5, 48)

blue: (-39, -42) red: (6, 48)
train
blue: (7, -111) red: (1, 111)

blue: (-20, 29) red: (21, 43)
onion_church
blue: (-22, -52) red: (10, 57)

blue: (-32, 69) red: (12, 57)
three_generations
blue: (-5, -52) red: (0, 57)

blue: (-20, 48) red: (-6, 57)
lady
blue: (6, -57) red: (-10, 44)

blue: (-13, 21) red: (3, 46)
workshop
blue: (5, -53) red: (-11, 51)

blue: (-28, 18) red: (2, 51)
tobolsk
blue: (-2, -3) red: (1, 4)

blue: (0, 1) red: (-1, 4)
village
blue: (7, -144) red: (-7, 103)

blue: (-46, -94) red: (24, 8)
cathedral
blue: (1, -1) red: (0, 7)

blue: (-5, 0) red: (0, 0)
monastery
blue: (0, 6) red: (1, 6)

blue: (0, 5) red: (0, 6)
melons
blue: (-4, -83) red: (3, 96)

blue: (-19, 41) red: (-3, 96)
forest
blue: (7, -34) red: (-5, 80)

blue: (-38, 15) red: (5, 80)
milan_cathedral
blue: (-12, -57) red: (11, 51)

blue: (-16, 39) red: (4, 50)
church
blue: (-10, -179) red: (8, 124)

blue: (-7, 50) red: (-3, 71)