Overview

In this project, I manipulated images from the Prokudin-Gorskii collection, where each image was a collection of 3 photos taken of the same subject--one with red-exposure, one with green-exposure, and one with blue-exposure. I then reconstructed the original image.

Approach(es)

I tried two approaches, both of which involve aligning the red and green channels with the blue channel according to the lowest possible difference in pixel intensity using the Sum of Squared Differences (SSD). First, I tried the naive approach of just using the SSD on the photos. With this approach, I computed the SSD of various offsets in a [-15, 15] range for red-on-blue and green-on-blue image compositions, choosing the offsets that are smallest in their SSD values. I then aligned the images according to these offsets. However, while this approach worked well on the smaller .jpgs, it was too slow to feasibly work on the larger .tif files. Each SSD took a nontrivial amount of time to compute, and since there were more pixels in the larger files, one would have needed to search a much larger area than [-15, 15] to find the global minimum SSD value.

In the second approach, I used an image pyramid to combat the runtime issues. The image pyramid works by recursively downscaling the image down in terms of resolution until it was less than 500 pixels to a side. Then, I would run the above naive SSD algorithm described above on the downscaled image. Using the offsets derived from the downscaled image, I would recursively pass the information upwards in increasingly higher resolutions, using the offset information gained from smaller resolutions to narrow my search space. For example, if a lower resolution image got an SSD offset of (-2, 3), I know I would need to search in an area of approximately (-4, 6), assuming I halved the number of pixels in the smaller image. Thus, I would not need to search a [-15, 15] window and instead I can search a [-3, 3] window, speeding up the process. This method worked relatively well for the .tif images while maintaining a reasonable (< 1 minute) runtime per image.

Bells & Whistles

Better features: Emir.tif and some other images didn't align correctly because those images had very similar pixel intensity throughout the photos. I used the sobel edge detector function in scimage to get the edges in the images and then use those to align which made the image alignment more accurate. This makes the contrast in pixels much more clear, as the edge-detected images have just two colors, a dark one and a light one corresponding to the edge. Thus, by running the SSD function on the images, the lowest SSD will correspond to where the images line up exactly.

There were a couple images that weren't aligned perfectly (for example, Self Portrait). I think this is due to the fact that my algorithm used the previous recursive call's offset info in generating the next call's offset numbers, so any small imperfection in an earlier call will get compounded and exacerbated the more calls there are, eventually resulting in a noisy image.

Final Images