CS 194-26 Fall 2020, Project 1: Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

April Sin

Project Overview

In 1907, Sergei Mikhailovich Prokudin-Gorskii (1863-1944) [Сергей Михайлович Прокудин-Горский] took color photographs of everything he saw. He accomplished this by using RGB filters and obtaining 3 glass plate negatives, RGB respectively. To view the colored images, we need to align and overlap the three plates. To improve results, it is also a good idea to make adjustments such as cropping, contrasting, white balance, color mapping, etc. The process of reproducing the color image is to be automated in this project.

Project Approach

Alignment

In this project, we are given input files with channels in the order BGR (as opposed to the conventional RGB). For simplicity, I aligned G and R channels to the B channel. My general approach is to have a fixed channel size of height h and width w . Then, we use an algorithm to find the best coordinate for the left-upper corner of each channel to which I call start .

1. Direct Mapping

The first algorithm I adopted is to simply divide the image by three. So the start for BGR would be (0, 0), (h, 0) and (h*2, 0). Then stace them together using numpy.stack. This method gave pretty reliable results with low computation time.

2. Simple Algorithms

I implemented both Sum of Squared Differences (output_ssd) and normalized cross-correlation (NCC) algorithms. I select a window that is smaller than the channel size to create sub-matrices for G and R channels. The sub-matrices are initialized to either 0 or infinity to suit the metric. This also help with handling out of bound windows.

Image Pyramid

At first, I tried applying pyramid to the full input image that contains all three channels. It gave a very coarse alignment probably due to extra borders in the input image. Then I was going to use border detection to remove borders then apply image pyramid. However, since I was not able to implement that, I opted for another approach. Instead, I use the displacements of the basic algorithm as starting estimates for the three channels, then create three separate matrices corresponding to each channel. Then, use the three matrices to calculate an estimate alignment.

Bells and Whistles

Adjustments

I added fix exposure which scales the brightest pixel as white and darkest pixel as black.
Both grey-world and white-world auto white balance (AWB) functions were implemented, but I tested that grey gives better results.
I attempted to write an algorithm to detect hard borders for cropping but was not able to sucessully crop.

Improving the Alignment Algorithm

I used Laplacian from openCV to compare the gradients of channels.
I used output_canny edge detection from openCV.

After Thoughts

Difficulties

Buliding from scratch is really fun, but it took some time to learn about new Python packages and to present the project as a whole nicely. Since there are so many parameters, I struggled to come up with a set of standardized arguments to pass into my alignment algorithms. Initially I wanted everything to be adjustable, but it made it really hard to test results as well as the debugging followed. This contributed to not being able to align the channels as good as I wanted.

Wish List

I would like to implement adjustments before and after alignment. Maybe having a few repeated iterations of adjust, align, re-adjust, re-align can help find the best displacement values. I would also like to experiment more on adjustments that can be made to make the results look better.

Project Results

The displacements are (dy, dx) relative to the blue input image, presented in the order for G, and R channels.

Some pictures of adjustment are omitted since the same result is computed with canny edge detection.

Example Results - Low Resolution Images

	adjustments	canny edge detection
monastery		"Offset(row=3, col=2) Offset(row=-3, col=2)"
tobolsk	"Offset(row=6, col=3) Offset(row=3, col=2"
cathedral		"Offset(row=12, col=3) Offset(row=5, col=2)"

Example Results - High Resolution Images

	adjustments	canny edge detection
workshop	"Offset(row=120, col=-8) Offset(row=64, col=0)"
emir		"Offset(row=120, col=32) Offset(row=56, col=16)"
three_generations		"Offset(row=120, col=8) Offset(row=64, col=16)"
castle		"Offset(row=112, col=0) Offset(row=40, col=0)"
melons		"Offset(row=120, col=-24) Offset(row=96, col=8)"
onion_church		"Offset(row=120, col=32) Offset(row=56, col=24"
train	"Offset(row=96, col=24) Offset(row=48, col=0)"	"Offset(row=96, col=24) Offset(row=56, col=0)"
icon		"Offset(row=104, col=24) Offset(row=48, col=16)"
self_portrait		"Offset(row=112, col=-8) Offset(row=88, col=24)"
harvesters		"Offset(row=120, col=0) Offset(row=72, col=16"
lady	"Offset(row=0, col=-16) Offset(row=-80, col=-8)"	"Offset(row=64, col=-16) Offset(row=64, col=0"

Extra Results - Self-selected images

	adjustments	canny edge detection
makhrovye	"Offset(row=6, col=1) Offset(row=1, col=1)"	"Offset(row=6, col=1) Offset(row=1, col=1)"
kapri		"Offset(row=10, col=-1) Offset(row=4, col=-1)"
cotton		"Offset(row=104, col=-16) Offset(row=88, col=24)"