Amit Talreja Project 1 - Colorizing Pictures



Introduction

In the early 1900s Sergei Mikhailovich Prokudin-Gorskii traveled the Russian Empire and took pictures of the many things that he saw. As true color photography was not available he improvised and took multiple pictures with a black and white camera, each with a different filter over the lens (for the red, green and blue components of the image). As the images were taken at different times they are not exactly aligned, and simply stacking the pictures on top of one another does not produce a color photograph

In this project I will try multiple methods that aim to automatically align the different layers of the image so that we can view the image in color.



Part 1: Naive Alignment

The simplest method we tried to align the three images is an exhaustive search over a window of possible displacements to find one that seems to line up the best. In more detail, we take the red and green layers and try to align them to the blue layer by computing the sum of squared differences for the two images at all positions in a [-15, 15] window in both the x and y directions. In this way we find coordinate adjustments we can make to the red and green layers to make them line up with the blue layer. Although this method is naive and computationally inefficient it works well for the small JPG images. Results of this algorithms on the JPG images are included below, along with their corresponding offsets



Cathedral

Green to Blue offset: (2, 5)

Red to Blue offset: (3, 12)


Monastery

Green to Blue offset: (2, -3)

Red to Blue offset: (2, 3)


Nativity

Green to Blue offset: (1, 3)

Red to Blue offset: (0, 8)


Settler

Green to Blue offset: (2, 5)

Red to Blue offset: (3, 12)



Part 2: Image Pyramid

Although the naive method worked well for small JPG images, it takes a prohibitively long time to exhaustively search of a large window for the higher-resolution TIFF images. To solve this we implemented an image pyramid, runs the naive alignment algorithm on the image at multiple scales. In this particular implementation, we downscale the image by factors of 2 until its height is less than 200 pixels, at which point we run the naive alignment method from above (with a smaller window size) to get the displacement at this small scale. We then double the size of the image and use the displacement computed at the smaller scale as a starting point for a naive search at the larger scale. This procedure is repeated, doubling the size of the image at each step, until the original image size is reached and we have a displacement for the original image. Results from this method are included below:



Emir

Green to Blue offset: (24, 49)

Red to Blue offset: (-545, 449)


As you can see, this method did not work well for the Emir image, with the red channel shifting far from the blue channel. This is because the large amount of blue in the Emir's robe saturates the blue channel and leaves the red channel with low values. Computing the SSD on this alignment will produce large values even when the image is aligned, so the procedure gets thrown off. This can be solved by aligning the red and blue channels to the green channel instead of aligning to the blue channel, as shown in the following image.



Emir green

Harvesters

Green to Blue offset: (17, 59)

Red to Blue offset: (14, 123)


Icon

Green to Blue offset: (17, 41)

Red to Blue offset: (23, 89)


Lady

Green to Blue offset: (8, 55)

Red to Blue offset: (12, 119)


Self Portrait

Green to Blue offset: (29, 78)

Red to Blue offset: (37, 176)


Three Generations

Green to Blue offset: (14, 52)

Red to Blue offset: (12, 111)


Train

Green to Blue offset: (6, 43)

Red to Blue offset: (32, 87)


Turkmen

Green to Blue offset: (21, 56)

Red to Blue offset: (28, 116)


Village

Green to Blue offset: (12, 65)

Red to Blue offset: (22, 137)



Extra Credit: Edge Detection

I tried implementing a version of the image pyramid alignment procedure that operated on the images after they had been passed through a Sobel filter to detect the edges in the images. This did not produce images that looked significantly different than those produced from the regular image pyramid process so I have chosen not to include them here.



Additional Images


Statues

Green to Blue offset: (1, 6)

Red to Blue offset: (0, 13)


Rivers

Green to Blue offset: (0, -1)

Red to Blue offset: (-1, 7)