Colorizing the Prokudin-Gorskii Photo Collection
Matthew Sie | SSID: 26250823 | cs194-26-aep
The goal of this project was to align the three color channel images of multiple glass plate images in order to produce a single RGB color image.
I began with a naive approach of creating a displacement window of [-15, 15] pixels in both the x and y directions. At each displacement, I scored the two color channel images being compared using the Sum of Squared Differences metric. The displacement that returned the lowest score was the final offset used to align the two color channels. This socring was done to align the green channel to the blue and the red channel to the blue. Then, the three channels were stacked on top of each other to produce the final image.
One of the early issues I encountered was the extraneous scoring calculations caused by the black border in some of the images. To remove the effect of the borders on the scoring, I cropped each image channel to only pass in the middle 80% of the image into the scoring method.
While this method worked well on the smaller images (cathedral.jpg, monastery.jpg), it was too computationally expensive for the larger images. Therefore, I implemented an image pyramid algorithm, in which I iteratively scaled down the large (cropped) images by 50% until it got below a 300 pixel threshold height. At this lowest resolution image, the scoring method was run with a maximum displacement window, an ideal displacement was returned, multiplied by 2, and applied to the subsequent next-higher resolution image. This recursive algorithm was run on each layer, iteratively cutting the displacement window in half, until it reached the highest resolution image.
The scoring method, combined with the image pyramid algorithm, worked well to correctly align most of the images. The results are shown in the table below under "Results".
The final issue involved images with which the scoring method and image pyramid alone did not suffice to align the image, such as "emir.tif". This I corrected using Canny Edge Detection, which I will discuss in the "Bells and Whistles" section below.
| Image Name | Unaligned | Aligned | Offset (Green) | Offset (Red) |
|---|---|---|---|---|
| cathedral.jpg | |
|
(4,2) | (12,2) |
| monastery.jpg | |
|
(-2,2) | (2,2) |
| nativity.jpg | |
|
(2,0) | (8,0) |
| settlers.jpg | |
|
(6,0) | (14,0) |
| emir.tif | |
|
(50,24) | (106,40) |
| harvesters.tif | |
|
(60,18) | (124,10) |
| icon.tif | |
|
(38,16) | (90,22) |
| lady.tif | |
|
(56,10) | (120,12) |
| self_portrait.tif | |
|
(78,30) | (174,36) |
| three_generations.tif | |
|
(56,12) | (112,8) |
| train.tif | |
|
(40,8) | (86,28) |
| turkmen.tif | |
|
(56,22) | (116,28) |
| village.tif | |
|
(64,10) | (138,24) |
| milan_cathedral.tif | |
|
(48,-8) | (110,-16) |
| four_people.tif | |
|
(48,-16) | (90,-44) |
| railroad_bridge.tif | |
|
(52,2) | (154,4) |
Canny Edge Detection: In "emir.tif", the scoring method was inaccurate as there are a bunch of different colors and intensities in the color channels in a relatively confined space. To further improve the scoring, I used an edge detection filter, and ran the comparitive scoring image pyramid algorithm on these edge intensified image matrices. This improved the alignment of "emir.tif" and other images in which there were many distinct edges. Results for the improvement of "emir.tif" are shown below.
| Image | Unaligned | Aligned (no edge detection) | Offsets w/o edge detection | Aligned (edge detection) | Offsets w/ edge detection |
|---|---|---|---|---|---|
| emir.tif | |
 |
Green: (48,24); Red: (74, -718) | |
Green: (50,24); Red: (106,40) |