Project 1: RGB Glass Plate Negatives to Color Images

CS 194-26: Intro. to Computer Vision & Computational Photography, Fa21

Doyoung Kim

1. Overview

During the Russian empire, when colored photos did not exist, Sergei Mikhailovich Prokudin-Gorskii started to record all the scenes in Russia on a glass plate using red, green, and blue filter at 1907. Now, as the technology advanced, we can combine the three images to create a fully colored image.

In this project, I implement two functionalities, "Exhaustive Search" and "Image Pyramid", to restore the black and white images taken by Prokudin-Gorskii into color image.

2. Pre-Processing

Input images for this project have forms of the following:

Three photo scenes from above are each taken with blue, green, and red filters, and the goal of this project is to align those three images as perfect as possible to create a well aligned colored image. In order to have as much accuracy as possible for this project, I cropped out the border lines and divide the image into 3 sections having exactly identical width and height.

3. Exhaustive Search

As images taken from each filter is not well aligned, I pick the image taken with green filter to be the reference and displace images taken by blue and red filters to align well. And there are two key aspects on achieving this task:

1. numpy.roll() 2. SSD
numpy.roll() is used to move the entire image by the given number of pixels depending on the given axis. Using this function, we can move the entire image by 1 pixel to left, to the top, or in any directions. We displace the image exhaustively in many possible directions to find the best alignment. But how do we determine the given displacement gives the perfect alignment? We can use SSD.

Sum of Squared Difference(SSD) allows to compute the similarity of two images. Smaller the SSD between two images(matrices), the more similar they are. Hence, while displacing blue or red image in possible directions, we compute the SSD between the green image(reference) and displaced blue or red image for each displacements and update the smallest SSD val and the corresponding displacement until the exhaustive search is over.

Results of aligned .jpg images

Examples provided by the course:

Blue Offset: (-5, -5)
Red Offset: (0, 7)

Blue Offset: (-3, 3)
Red Offset: (0, 6)

Blue Offset: (-3, -3)
Red Offset: (0, 4)

Extra Examples:

Blue Offset: (2, -2)
Red Offset: (0, 5)

Blue Offset: (0, -5)
Red Offset: (0, 7)

4. Image Pyramid

Simple exhaustive search were good enough for the low resolution images like .jpg. However, If the image size gets much bigger, like .tif, it can take very long time to find the best displacement as the number of pixels it has to search through increases as well when the size of the image increases. And implementing Image Pyramid can reduce this time to find the best displacement for large image.

Image Pyramid basically reduces the scale of the image until its size becomes small enough to find the best displacement with exhaustive search in reasonable time. Say we rescale the image in 1/2 for attempt to rescale. When the scale of the image becomes reasonably small, we perform the exhaustive search on it to find the best displacement for the small image. Then, we multiply this displacements by 2, so it can correspond to the image before it was halved. With the replacement, given for the larger image, we start the exhaustive search from there and find the best displacement. And we repeat this process until we perform the exhaustive search on the original scale image. This can be easily implemented with recursive function.