CS 194 Project 1

Overview

The main focus of the project was to align a set of three RGB filtered images and, through image processing, produce a colored image. The images originated from Prokudin-Gorskii Photo Collection which consisted of 3 snapshots of the same image each using a different filter. These images are then aligned and layer for each respective RGB color channel to create a colored version.

Section I: Aligning Algorithm

Part 1: Lower Quality Images (JPGs)

The JPG images inherently were lower resolution images, thus an Exhaustive Search Algorithm was sufficient to properly align the images together. I decided to find the best displacement that aligned the red and green channel with the blue channel as the center comparison which was purely a design decision. The Exhaustive Search Algorithm proceeds as follows: each of the red and green images would be rolled (displaced) in a 30x30 grid containing offsets between [-15, 15] in both the x and y direction. In order to determine which offset produced the best alignment, I used the normalized cross correlation (NCC) metric (the dot product between the two image vectors) to compare and chose the maximum score from all 900 possible offsets. Finally, after finding the best red and green offset, we would stack the images together in their respective color channels to produce the aligned image.

Part 2: Higher Quality Images (TIFs)

Because of higher resolution images (TIFs) having more pixels, ideal displacement offset was typically outside of the 30x30 offset grid I defined in the Exhaustive Search Algorithm. This cause a major issue as to find the most ideal displacement, Exhaustive Search Algorithm would simply take too long to calculate. This motivated the Image Pyramid Algorithm which decrease the number of offsets required to compare through the use of downsizing the image through the use of recursion. The motivation behind the Image Pyramid Algorithm is that by downsizing the image to a more reasonable size to align, we can initially find a general displacement of where to align which serves as a starting point to fine tune the exact alignment. For these high-res images, I still used NCC as the alignment decision metric and overlayed the red and green images on the blue.

The Image Pyramid Algorithm proceeds as follows:

Resize the images to a smaller size by a scale of 2. Repeat this for a depth of 4 times to hit the base case
Call the Aligning Exhaustive Search Algorithm on the smaller image to choose the best offset determined by the maximal NCC metric
Once the optimal was found, recursively pass up the ideal offsets to the subsequent larger-image function call.
The subsequent call can scale up the offset found in the previous downsized image by a factor of 2 and process to call the Exhaustive Search Algorithm to fine tune the alignment with a starting displacement from the prior provided offset
Repeat this until the initial function call which returns the ideal image with the ideal offsets

Section II: Problems

Cropping Optimizations

When running the Exhaustive Search Algorithm on the entire image, the produced color image would be improperly aligned. This is because of the edges of the images that Prokudin-Gorskii took. Since they originally were a real photo in the early 1900s, the edges of the photos because discolor and played a heavy contribution to NCC metric measurement. As a result, I decided prior to aligning the images, I would crop 5% of the image on all sides to remove all the discoloration which ended up working wonderfully.

Runtime Optimizations

Although the Image Pyramid Algorithm significantly improved the running time to find the best offset compared to Exhaustive Search, it still was not sufficient to find the best alignment with the 1 minute time constraint. So to improve my runtime, I noticed that the Exhaustive Search Algorithm scanned 900 different offsets in a [-15, 15] grid in both x and y. Because we are already aligning the image at a smaller scale, I believe that this displacement was not necessary and instead only scanned 256 different offsets in a [-8, 8] grid in both x and y. This allows my image alignment function run in approximately a minute.

Image: emir.tif

Using the alignment algorithm listed above, all the images produced beautiful and clear results other than emir.tif. The red filtered image was noticably shifted to the left where its should be. Upon analyzing, this is because the images to not have the same brightness values which caused improper alignment based on the NCC metric. To fix this, I actually decided to re-align the image based on the green filtered image instead and essentially align the red and blue images to the green. This actually turned out to produce proper alignment.