CS194-26 Project 1: Colorizing the Prokudin-Gorskii photo collection

Jia Ze Yu, cs194-26-ade

2018-09-05

1. Overview

This project extracts the three color channel images, place them on top of each other, and align them so that they form a single RGB color image.

2. Approaches

For the baseline naive implementation, I first crop away the margins that are filled with noises and exhaustively search over a window of possible displacements (of size [-15,15] pixels). Then, I score each one using the Sum of Squared Differences(SSD) image matching metric and take the displacement with the best score. This works well for smaller jpeg images but for larger tif images, this method becomes too computationally expensive. Thus, I implemented the image pyramid search procedure by representing the image at multiple scales (scaled by a factor of 2).

3. Results

These are the results generated from main.py. Each picture is attached with the information of the horizontal shift of the green channel (g_x), the vertical shift of the green channel (g_y), the horizontal shift of the red channel (r_x) and the vertical shift of the red channel (r_y) in pixels.

Baseline Exhaustive Search

Cathedral.jpg (g_x, g_y) = (5, 2), (r_x, r_y) = (12, 3) Monastery.jpg (g_x, g_y) = (-3, 2), (r_x, r_y) = (3, 2)
Nativity.jpg (g_x, g_y) = (3, 1), (r_x, r_y) = (8, 0) Settlers.jpg (g_x, g_y) = (7, 0), (r_x, r_y) = (14, -1)

Pyramid Search

Harvesters.tif (g_x, g_y) = (59, 16), (r_x, r_y) = (123, 13) Icon.tif (g_x, g_y) = (41, 17), (r_x, r_y) = (89, 23)
Lady.tif (g_x, g_y) = (51, 9), (r_x, r_y) = (111, 12) Self_portrait.tif (g_x, g_y) = (78, 29), (r_x, r_y) = (176, 37)
Three_generations.tif (g_x, g_y) = (53, 14), (r_x, r_y) = (112, 11) Train.tif (g_x, g_y) = (42, 5), (r_x, r_y) = (86, 32)
Turkmen.tif (g_x, g_y) = (56, 21), (r_x, r_y) = (116, 28) Village.tif (g_x, g_y) = (64, 12), (r_x, r_y) = (137, 22)

3. Bells and whistles

I used an edge detection technique called Sobel filter to further improve the alignment since Pyramid Search alone is insufficient to align emir.tif well.

Emir.tif

Pyramid Search (g_x, g_y) = (49, 24), (r_x, r_y) = (78, -712) Sobel (g_x, g_y) = (49, 24), (r_x, r_y) = (105, 41)

Before/After Sobel

Harvesters.tif (g_x, g_y) = (59, 16), (r_x, r_y) = (123, 13) Harvesters.tif (g_x, g_y) = (56, 11), (r_x, r_y) = (118, 10)
Icon.tif (g_x, g_y) = (41, 17), (r_x, r_y) = (89, 23) Icon.tif (g_x, g_y) = (39, 16), (r_x, r_y) = (89, 23)
Lady.tif (g_x, g_y) = (51, 9), (r_x, r_y) = (111, 12) Lady.tif (g_x, g_y) = (57, 9), (r_x, r_y) = (121, 13)
Self_portrait.tif (g_x, g_y) = (78, 29), (r_x, r_y) = (176, 37) Self_portrait.tif (g_x, g_y) = (74, 25), (r_x, r_y) = (175, 37)
Three_generations.tif (g_x, g_y) = (53, 14), (r_x, r_y) = (112, 11) Three_generations.tif (g_x, g_y) = (58, 16), (r_x, r_y) = (115, 11)
Train.tif (g_x, g_y) = (42, 5), (r_x, r_y) = (86, 32) Train.tif (g_x, g_y) = (40, -1), (r_x, r_y) = (85, 28)
Turkmen.tif (g_x, g_y) = (56, 21), (r_x, r_y) = (116, 28) Turkmen.tif (g_x, g_y) = (58, 21), (r_x, r_y) = (118, 29)
Village.tif (g_x, g_y) = (64, 12), (r_x, r_y) = (137, 22) Village.tif (g_x, g_y) = (64, 10), (r_x, r_y) = (137, 22)

I then added auto-contrast.

Before/After Sobel + Autocontrast

Cathedral.jpg (g_x, g_y) = (5, 2), (r_x, r_y) = (12, 3) Cathedral.jpg (g_x, g_y) = (5, 2), (r_x, r_y) = (12, 3)
Monastery.jpg (g_x, g_y) = (-3, 2), (r_x, r_y) = (3, 2) Monastery.jpg (g_x, g_y) = (-3, 2), (r_x, r_y) = (3, 2)
Nativity.jpg (g_x, g_y) = (3, 1), (r_x, r_y) = (8, 0) Nativity.jpg (g_x, g_y) = (3, 1), (r_x, r_y) = (8, 0)
Settlers.jpg (g_x, g_y) = (7, 0), (r_x, r_y) = (14, -1) Settlers.jpg (g_x, g_y) = (7, 0), (r_x, r_y) = (14, -1)
Emir.tif (g_x, g_y) = (49, 24), (r_x, r_y) = (78, -712) Emir.tif (g_x, g_y) = (49, 24), (r_x, r_y) = (105, 41)
Harvesters.tif (g_x, g_y) = (59, 16), (r_x, r_y) = (123, 13) Harvesters.tif (g_x, g_y) = (56, 11), (r_x, r_y) = (118, 10)
Icon.tif (g_x, g_y) = (41, 17), (r_x, r_y) = (89, 23) Icon.tif (g_x, g_y) = (39, 16), (r_x, r_y) = (89, 23)
Lady.tif (g_x, g_y) = (51, 9), (r_x, r_y) = (111, 12) Lady.tif (g_x, g_y) = (57, 9), (r_x, r_y) = (121, 13)
Self_portrait.tif (g_x, g_y) = (78, 29), (r_x, r_y) = (176, 37) Self_portrait.tif (g_x, g_y) = (74, 25), (r_x, r_y) = (175, 37)
Three_generations.tif (g_x, g_y) = (53, 14), (r_x, r_y) = (112, 11) Three_generations.tif (g_x, g_y) = (58, 16), (r_x, r_y) = (115, 11)
Train.tif (g_x, g_y) = (42, 5), (r_x, r_y) = (86, 32) Train.tif (g_x, g_y) = (40, -1), (r_x, r_y) = (85, 28)
Turkmen.tif (g_x, g_y) = (56, 21), (r_x, r_y) = (116, 28) Turkmen.tif (g_x, g_y) = (58, 21), (r_x, r_y) = (118, 29)
Village.tif (g_x, g_y) = (64, 12), (r_x, r_y) = (137, 22) Village.tif (g_x, g_y) = (64, 10), (r_x, r_y) = (137, 22)