# CS 194-26 Fall 2020

## Overview

In this project we aim to use Sergey Prokudin-Gorsky's work to recreate color photos from the early 20th century. We do this by taking advantage of his ingeunity: he left us with photos on a glass plate where, for a given scene, he would take three seperate single-channel photographs of the same scene with a Blue, Red, and Green filter. With this information we have enough to reconstruct a colored depiction of the scenes by overlaying the three images.

The problem? The images are not laid out in perfect alignment, and naively splitting the glass panes into three and directly overlaying them results in terrible color images. And so our task is to write an algorithm that, given the raw pixel values for each channel, determine the best offset such that the final color reconstruction is aligned.

## Approach

The general idea for my approaches is to independently attempt to overlay two channels onto the remaining channel (e.g. red and green onto blue). I find the offset for each by making a pass over some range of offsets, and for each, calculating some image matching metric (where higher is better). I then simply return the offset that produced the best image matching metric.

For smaller, low-resolution imanges, this can be acheieved by simply iterating over some reasonable range of pixels ([-15, 15] in my case). This is computationally too expensive on larger images, so I've implemented a faster search with an image pyramid, where offsets are calculated on scaled-down versions of the image recursively. This worked reasonably well for most images.

## Single-scale

Exhaustively search over the range of [-15, 15] pixels, return offset that maximizes the normalized cross-correlation (NCC), which is the dot product between the two normalized images.

Offset (x, y) is defined as shifting x rows down, y columns to the right.

## Multi-scale

With high-resolution images, instead of starting a search over a large range, I downsize the image and start a search over a range on the downsized image to find an offset for the downsized image, then use this offset as the center for a smaller search range when computing offset image matching metrics on the original image. Of course, this process can be done multiple times, and represents an image pyramid.

Parameters:

• Image Pyramid Depth: 5
• Scaling Factor: 0.65
• Default Search Range: [-5, 5]
• Base Case Search Range: [-15, 15]

Initially I tried this with overlaying Red and Green onto Blue, but I found that for 'emir', this actually worked quite poorly. Based on the glass plates, I think the reason is that in terms of pixel values, Blue and Red are on opposite ends (especially for the clothes, where one is dark while other is bright). Meanwhile Green is somewhere between the two in terms of brightness, so finding the dot product between the other colors and Green would be the best compromise.

### Results

After some expiermentation I found that overlaying Red and Blue onto Green actually worked quite well, even with the standard image pyramid and NCC approach!