CS 194-26 Project 1

Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

Rohan Narayan, ID 26650514

Step 1: Basic Alignment

The first step was basic alignment of the 3 photos. Without translation of the green and red filtered images, the compiled image would look like the image on the left. Alignment allows all images to line up on top of one another, leading to a clear image, such as the one on the right.

How

For Step 1, we assume that we work with small images. Try all possibilities of translation of the image over a [-15,15] grid. To do this, use np.roll() to shift the image by a specified (x,y) translation. For each translation, compute the Sum of Squared Differences between the shifted photo and the original. In our case, we would shift the green/red filtered photo and compare to the blue filtered image. We then choose the shift that yields the lowest Sum of Squared Differences.

Input: All three filters

No Alignment

Output: With Alignment (search over grid)

Step 2: Pyramid Structure

With larger images, the (x,y) the displacement needed to align images would be greater than 15–with these larger images (on the scale of 4000x3000), an exhaustive search would take too long. Therefore, we use a pyramid structure to downscale the image, then align the smaller-sized images. We then scale up the displacements, and use them to align the higer-resolution images.

How

For an image, continue to rescale until the image is small enough (250x250). At the smallest scale, do an exhaustive search over displacement possibilities in the range of [-15,15] using the approach described in Step 1. When a displacement is found, scale up the displacement, and apply it to the next higher-resolution image. Then do a exhaustive search over displacements in the range of [-3,3]. Continue until you have reached the highest-resolution image you have.

Smaller Photos

Below we can see the process executed on smaller .jpg images. Hovering over each photo will show the (x,y) displacements used for the Green and Red filtered images.

Larger Photos

Below we can see the process executed on larger .tif images. All executions lasted less than 30 seconds. Like above, hovering over each photo will show the (x,y) displacements used for the Green and Red filtered images. Below this is the execution time to complete the alignment.

Problems Encountered

Besides normal debugging issues, I ran into an issue where some images were aligning properly (images like icon.tif), but many others weren't. I fixed this issue by only computing the SSD on the "inner" part of the images. In this case, I discarded the outer fifth of the image.

Problems with emir.tif

Although the method worked for all other images, the emir.tif image wasn't working correctly. As the spec mentions, this is probably because the brightness values were different. To solve this issue, I tried to align on a different color. Because there wasn't as big of an issue with brightness values, the image aligned correctly.

Blue: (-24,-49), Red: (17,57), Time: 20.65s

emir.tif

emir.tif, aligned on blue

emir.tif, aligned on green

Example Photos

Below we can see the process executed on other images in the collection. To view original source, hover over the photo. Note: image names were made up by me

Bells and Whistles: Basic Cropping of White Borders

There is a noticeable white border on the left and right of almost all the images. I aimed to remove these borders. In order to do this, I added up all the columns for all three images, once aligned. Since the white columns would have much larger pixel values, the summed values would be noticeable higher than the rest of the image.

The results can be seen below. The black border around the image shows the cropped white borders.

self_portrait.tif

No Cropping

With Cropping

settlers.jpg

No Cropping

With Cropping