﻿

Colorizing the Prokudin-Gorskii Photo Collection

OVERVIEW

In this project, I produced color images from the Prokudin-Gorskii glass plate images by dividing the glass plate images into their three color channel images: B, G, and R. These channels are then aligned and stitched together to create a single RGB color image with as few visual artifacts as possible. Here is an example of what my code accomplishes:

 blue channel (input) green channel (input) red channel (input) colored image (output)

METHODOLOGY

INITIAL APPROACH

My initial logic for aligning these 3 channels was to:

1.     Find the best displacement between the green and blue channel over a [-15,15] window using some image matching metric (I used sum of square distances, as the computation was faster than L2 norm)  and align the green channel relative to the blue channel

a.     the best displacement is defined as that which gives the minimum possible sum of square distances between pixel values between the two channels

2.     Find the best displacement between the red channel and the aligned green channel over a [-15,15] window and align the red channel relative to the aligned green channel.

However, this exhaustive search only worked well for smaller images (.jpg files) but was neither efficient nor thorough enough for larger images.

IMPROVEMENTS

1.    To (somewhat) ignore the image borders, I manually cropped the images to account for only the center 85% of the image.

2.    To more efficiently and accurately calculate the appropriate displacement, I implemented an image pyramid.
I sequentially calculated the appropriate displacement over a [-15, 15] window over multiple scales (scaled by a factor of 2)– scaling from the coarsest image to the actual image size, aligning the actual image using the pixel displacement scaled to the appropriate factor relating the smaller image and the actual image.  This significantly reduced runtime and produced good alignments, even for the Emir of Bukhara.

3.    (Bells and Whistles) To align well regardless of color channels and RGB similarity, I aligned the images using canny edge detection (reference: https://scikit-image.org/docs/dev/auto_examples/edges/plot_canny.html).
Comparing the edges rather than pixel values allows for more accurate alignment, and the images matched do not need to have the same brightness values. Though the difference is just noticeable, alignment using edge detection worked better than alignment using raw pixel values.

The difference after applying canny edge detection is just noticeable. Comparing the appearance of the eyes in these two pictures eyes reveals edge detection worked better– there is no redness in the eyes in the image on the right, the eyes are well defined because the image is better aligned.

 Emir of Bukhara using SSD on pixel values Emir of Bukhara using SSD on edges

OUTCOMES & OFFSETS

 stitching without alignment alignment using SSD on raw pixel values alignment using SSD on edges monastery.jpg Green dx: 2, dy: -2 Red dx: 2, dy: 4 Green dx: 2, dy: -4 Red dx: 4, dy: 2 tobolsk.jpg Green dx: 2, dy: 2 Red dx: 2, dy: 6 Green dx: 2, dy: 2 Red dx: 2, dy: 6 cathedral.jpg Green dx:2, dy: 4 Red dx: 2, dy: 10 Green dx:2, dy: 4 Red dx: 2, dy: 10 workshop.tif Green dx: 0, dy: 52 Red dx: -10, dy: 104 Green dx: -2, dy: 52 Red dx: -12, dy: 104 emir.tif Green dx: 24, dy: 48 Red dx: 40, dy: 48 Green dx: 24, dy: 40 Red dx: 40, dy: 108 three_generations.tif Green dx: 14, dy: 54 Red dx: 12, dy: 112 Green dx: 18, dy: 56 Red dx: 12, dy: 116 castle.tif Green dx: 2, dy: 34 Red dx: 2, dy: 98 Green dx: 4, dy: 36 Red dx: 4, dy: 100 melons.tif Green dx: 10, dy: 82 Red dx: 14, dy: 178 Green dx: 10, dy: 80 Red dx: 12, dy: 176 onion_church.tif Green dx: 26, dy: 52 Red dx: 36, dy: 108 Green dx: 28, dy: 52 Red dx: 36, dy: 108 train.tif Green dx: 4, dy: 42 Red dx: 30, dy: 86 Green dx: 8, dy: 40 Red dx: 34, dy: 82 icon.tif Green dx: 16, dy: 40 Red dx: 22, dy: 88 Green dx: 16, dy: 42 Red dx: 22, dy: 90 self_portrait.tif Green dx: 28, dy: 78 Red dx: 36, dy: 176 Green dx: 30, dy: 80 Red dx: 38, dy: 178 harvesters.tif Green dx: 16, dy: 60 Red dx: 14, dy: 124 Green dx: 16, dy: 60 Red dx: 14, dy: 124 lady.tif Green dx: 8, dy: 48 Red dx: 12, dy: 110 Green dx: 8, dy: 56 Red dx: 12, dy: 118

Aligned using canny edge detection

 Loparev cardboard factory Green dx: 2, dy:24 Red dx: -6, dy: 92 Exit from the yard of the Church of Saint George Green dx: 24, dy: 72 Red dx:38, dy: 154 Fishing settlement Green dx: -32, dy: 8 Red dx: -70, dy: 30

 On the island of Capri Green dx: -16, dy: 32 Red dx: -24, dy: 78 Detail of the Por-Porog waterfall Green dx: -10, dy: 34 Red dx: -18, dy: 122

-