In this project, we develop an automated alignment algorithm using image pyramids to align the three color channels of images in the Library of Congress's Prokudin-Gorskii photo collection. Sergei Mikhailovich Prokudin-Gorskii was the first color photographer, and used an invention of his own genious design to snap three simultanous photographs, each of which used different color filters. When combined, these three apparently monochrome photographs produce stunning images of the world in color.
Image Pyramid¶
The challenge, however, lies in aligning the three photos, each of which would have been taken at a slightly different angle. Prokudin-Gorskii did so by hand, but today we can produce better results using automated image alignment algorithms. In this lab, we will demonstrate one such algorithm: the image pyramid.
1. Warming up¶
The Prokudin-Gorskii photo collection represents the first color photography
Import libraries¶
In [27]:
%pylab inline
%load_ext autoreload
%autoreload 2
import skimage as sk
import skimage.io as skio
from plotting import *
from align import *
from utils import *
Read in the Bukhara Emir and plot the three channels¶
In [12]:
emir = sk.img_as_float(read_pg_img('data/emir.tif'))
print(emir.shape)
In [13]:
plot_3_channels(emir)
Combine the three channels without alignment¶
In [29]:
imshow(emir)
show()
2. Single-scale alignment¶
In [47]:
im = sk.img_as_float(read_pg_img('data/monastery.jpg'))
print(im.shape)
In [58]:
plot_3_channels(im)
2.1 Manually adjust the red channel and compare with the unaligned version:¶
In [62]:
r_translated = translate_img(im[:,:,0], translation=(0,-10))
im_align = np.dstack([r_translated, im[:,:,1:]])
ssd_realign = ssd_loss(r_translated, im[:,:,1])
ncc_realign = 1 - ncc_loss(r_translated, im[:,:,1])
ssd_orig = ssd_loss(im[:,:,0], im[:,:,1])
ncc_orig = 1 - ncc_loss(im[:,:,0], im[:,:,1])
plot_comparison(
im, im_align,
title1 = f'Before: SSD={ssd_orig:0.3f}, NCC={ncc_orig:0.3f}',
title2 = f'After: SSD={ssd_realign:0.3f}, NCC={ncc_realign:0.3f}'
)
2.2 Use an automated alignment procedure to do the same¶
Using SSD as loss:
In [75]:
r_align = align_by_grid_search(im[:,:,0], im[:,:,1])
im_align = np.dstack([r_align, im[:,:,1:]])
ssd_realign = ssd_loss(r_align, im[:,:,1])
ncc_realign = 1 - 2*ncc_loss(r_align, im[:,:,1])
plot_comparison(
im, im_align,
title1 = f'Before: SSD={ssd_orig:0.3f}, NCC={ncc_orig:0.3f}',
title2 = f'After: SSD={ssd_realign:0.3f}, NCC={ncc_realign:0.3f}'
)
Using NCC in loss:
In [74]:
r_align_ncc = align_by_grid_search(im[:,:,0], im[:,:,1], loss_fn=ncc_loss)
im_align_ncc = np.dstack([r_align_ncc, im[:,:,1:]])
ssd_realign2 = ssd_loss(r_align_ncc, im[:,:,1])
ncc_realign2 = 1 - 2*ncc_loss(r_align_ncc, im[:,:,1])
plot_comparison(
im, im_align_ncc,
title1 = f'Before: SSD={ssd_orig:0.3f}, NCC={ncc_orig:0.3f}',
title2 = f'After: SSD={ssd_realign:0.3f}, NCC={ncc_realign2:0.3f}'
)
2.3 Align all three channels¶
In [102]:
%%time
im_align = align_img(im, ref=1)
In [103]:
plot_comparison(im, im_align)
Try a larger alignment grid
In [104]:
%%time
im_align = align_img(im, ref=1, window=(-30,30))
In [105]:
plot_comparison(im, im_align)
2.4 Crop edges¶
Uniform cropping
In [108]:
im_align = align_img(crop_img_edges(im, pixels=(20,20,20,20)), ref=1)
plot_comparison(im, im_align)
3. Image Pyramid¶
3.1 Try the image pyramid on emir.tif¶
In [14]:
emir_cropped = crop_img_edges(sk.img_as_float(emir), pixels=(200,200,200,200))
In [48]:
%%time
# TODO: try ncc_loss
emir_aligned = align_pyramid(emir_cropped, debug=True)
3.2 Try NCC loss¶
In [49]:
%%time
emir_ncc = align_pyramid(emir_cropped, loss_fn=ncc_loss)
In [24]:
plot_comparison(emir_ncc, emir_aligned, title1="Using SSD loss", title2="Using NCC loss")
To my eyes, the SSD and NCC losses produce nearly identical results.
4. Other images¶
Castle¶
In [50]:
%%time
castle = read_and_crop_pg_img('data/castle.tif')
castle_aligned = align_pyramid(castle)
plot_comparison(castle, castle_aligned)
Harvesters¶
In [51]:
%%time
harvesters = read_and_crop_pg_img('data/harvesters.tif')
harvesters_aligned = align_pyramid(harvesters)
plot_comparison(harvesters, harvesters_aligned)
Icon¶
In [52]:
%%time
icon = read_and_crop_pg_img('data/icon.tif')
icon_aligned = align_pyramid(icon)
plot_comparison(icon, icon_aligned)
Lady¶
In [53]:
%%time
lady = read_and_crop_pg_img('data/lady.tif')
lady_aligned = align_pyramid(lady)
plot_comparison(lady, lady_aligned)
Melons¶
In [54]:
%%time
melons = read_and_crop_pg_img('data/melons.tif')
melons_aligned = align_pyramid(melons)
plot_comparison(melons, melons_aligned)
Onion Church¶
In [55]:
%%time
onion_church = read_and_crop_pg_img('data/onion_church.tif')
onion_church_aligned = align_pyramid(onion_church)
plot_comparison(onion_church, onion_church_aligned)
Self-portrait¶
In [56]:
%%time
self_portrait = read_and_crop_pg_img('data/self_portrait.tif')
self_portrait_aligned = align_pyramid(self_portrait)
plot_comparison(self_portrait, self_portrait_aligned)
Three generations¶
In [57]:
%%time
three_generations = read_and_crop_pg_img('data/three_generations.tif')
three_generations_aligned = align_pyramid(three_generations)
plot_comparison(three_generations, three_generations_aligned)
Train¶
In [58]:
%%time
train = read_and_crop_pg_img('data/train.tif')
train_aligned = align_pyramid(train)
plot_comparison(train, train_aligned)
Workshop¶
In [59]:
%%time
workshop = read_and_crop_pg_img('data/workshop.tif')
workshop_aligned = align_pyramid(workshop)
plot_comparison(workshop, workshop_aligned)
Some images I chose¶
Stained glass¶
In [60]:
%%time
stained_glass = read_and_crop_pg_img('data/stained_glass.tif')
stained_glass_aligned = align_pyramid(stained_glass)
plot_comparison(stained_glass, stained_glass_aligned)
Cathedral roof¶
In [61]:
%%time
cathedral_roof = read_and_crop_pg_img('data/cathedral_roof.tif')
cathedral_roof_aligned = align_pyramid(cathedral_roof)
plot_comparison(cathedral_roof, cathedral_roof_aligned)
Cliff¶
In [62]:
%%time
cliff = read_and_crop_pg_img('data/cliff.tif')
cliff_aligned = align_pyramid(cliff)
plot_comparison(cliff, cliff_aligned)
In [ ]: