overview

In this project we align the red, green, and blue channels of images taken by Sergei Mikhailovich Prokudin-Gorskii. Note: credit given to Kevin Chen's fall 2016 assignment for use of his website theme and scripts.

setup

Jupyter (for Python 3) is required to be installed to run the code. To run the included code, you need to run proj1.ipynb OR proj1.py in the same folder as the images, e.g. the /data folder. You also need to have the out_path folder defined, and inside out_path another folder called other_examples. Also, other_examples should be another directory in the root folder containing the other images I downloaded for testing

jupyter notebook proj1.ipynb
Or this command: 
jupyter notebook proj1.ipynb

Upon opening the notebook, just run through all the cells

approach

distance metric

To estimate the distance between images so to align them, I tried both SSD (sum of squared differences) and NCC on the images (normalized cross correlation)


	def calc_SSD(a, b):
	diff = a - b
	return np.sum(np.multiply(diff, diff))


	def calc_NCC(a, b):
	mean_a = np.mean(a)
	mean_b = np.mean(b)
	return np.sum(np.multiply(a-mean_a, b-mean_b))

image pyramid

I implemented the image pyramid algorithm recursively: upon taking an a large image, I shrink it by half on both x and y and check if the height is below 500. If it's smaller than 500 pixels in height, I stop shrinking and calculate the offsets to best align the red and green to the blue image using a generous start radius of search of offsets to test (e.g. a radius of 20 to 30 pixels). Then in each layer above, I scale that offset calculated by 2 both on x and y, apply it, and run it with a much smaller search window of radius 3 (as the first initial rolls should have mostly aligned the image). This makes it much less expensive for searching for the optimal alignment in the full resolution scale.

This allows most alignments to finish within 1 min (depending on the start radius) on my 2013 Macbook Air

bells and whistles

edge detection

To try to improve alignment, I tried the sobel edge detection algorithm to pick out features of the different images. This helped for some images, but for other images the edges were difficult to detect using this algorithm and didn't help much.

contrast

Since edge detection didn't fully help find features, I tried to increase the contrast of the images, by pushing the lightest pixels to white and the darkest pixels to black. I used the sigmoid function for this, and sometimes tweaked the threshold to better align images as some channels were much darker or much lighter, and I wanted to match them. I ended up using sigmoid contrast to improve alignment of most of my images.

cropping

Since the original images had white and black borders that interfered with alignment, I also enforced cropping off a fixed 250 pixels off each edge for the large TIF images (demonstrated in the melons above). This helped for better alignment of each channel.

example results

These are the results after using image pyramid for the high resolution images. In all instances, the offset is given for the red and green channels, from the bottom left corner of the blue channel. You can click to view any of these images in a higher resolution.