CS194-26: Image Manipulation and Computational Photography

Project 1: Images of the Russian Empire

by Christy Koh, September 2021

Overview

This project aims to align the series of color-filtered photographs taken by Prokudin-Gorskii in the early 1900's. He captured various subjects in 3 images, individually taken with red, blue, and green filters; here, an alignment algorithm is applied to combine these into a single colored photograph.

Single-Scale: Exhaustive Search

First, I define an shifting-window alignment algorithm which searches over a user-defined window of x- and y- displacements. The image is trimmed of noisy edges before alignment, with 40% of the pixels trimmed from each edge, in order to score using the center of the image, which has the most interesting features. For each candidate displacement, the trimmed image is then scored using normalized cross-correlation (scoring using sum of squared distance is also supported). The alignment that maximizes the NCC is retained.

The results below were obtained using a displacement window of [-15px, 15px].

Calculated offsets: shift_ncc.txt

Multi-Scale: Image Pyramid Search

In order to efficiently process larger images and reduce the size of the displacement window, a multi-scale application of the single-scale shifting-window algorithm is applied. To accomplish this, the image is recursively scaled down by a factor of 2 until the width of the coarse scaled image is less than a user-specified threshold. The single-scale alignment algorithm is then applied recursively to the scaled images, traversing a coarse-to-fine search pyramid.

To explain the rationale behind parameters selected:

• A window of (-2px, 2px) for both axes was selected to allow for fine correction of +/- 1px errors caused by image downscaling by a factor of 2.
• From the simpler images such as cathedral.jpeg, a shift of 15px was attainable by 4 recursive calls, which translated to a 'smallest-image' threshold approximately between 40px (~300/2^3) and 75px. This was tuned manually to optimize for all images.

Compared to the single-scale algorithm, the multi-scale implementation achieved a 12x speedup for "lady.tiff". The average runtime for a ~70MB .tiff file is under 20 seconds.

Calculated offsets: shift.txt

The alignment for "emir.tiff" was inaccurate; this is due to the poor correlation between red and blue channels on the emir's robe, exposing the weakness of correlation of raw RGB values as a metric for a good alignment. To address this, I adjusted the algorithm to align the blue and red channels to the green channel instead. Ideally, an edge-detection algorithm or some better scoring metric should be applied for more robust performance.

Additional Images

Below are the results of the algorithm on an additional set of pictures from the collection.

Calculated offsets: shift_additional.txt