CS 194 Project 1: Images of the Russian Empire
Richard Chen
Russian photographer Sergei Mikhailovich Prokudin-Gorskii captured images of the Russian Empire in the 1900s by taking pictures of the same scene three times each, placing either a red, blue, and green filter in front of each. The goal of this project is to develop an algorithm that can process these images and recreate an rgb image of the original scene.
Approach
For the small images, a brute force search is satisfactory. I found the displacements of the green and red plates in relation to the blue plate by searching from a range of [-15,15) pixel shifts in both the x and y directions. I evaluated these shifts using a normalized cross correlation. The higher the normalized cross correlation is between two images, the more "similar" they should be to each other.
This aproach works fairly well when images are not that big and a [-15,15) range is satisfactory. But it doesn't work when the images are a lot bigger. Instead I implemented an image pyramid. I start with downscaled versions of the r,g,b channels and gradually work my way up to the pictures full resolution.
For the large images, I downscaled each dimension by a factor of 64 each. Then I found the pixel shift from these downscaled versions searching a range of [-3,3] in both x and y. When I found this value, I would store this information (keeping in mind that this was for a downscaled version). I would then shift the image according to my value, and recursively call the alignment method, except this time with a downscale factor of 32 from the original instead of 64. Every recursive call gets more and more precise to the true pixel shift, until I reach the full resolution.
I had a lot of problems with emir.tif. It seems the problem is the patterns on his clothes. Because there are so many similar patterns, the alignment method gets confused and gives incorrect values. The results improved slightly when I cropped different amounts of the image, but there was no real insight to playing with these margins, so I didn't try to play that game. Most likely to get a clear image, I would need to implement some sort of edge detection. Shaving off 5% from every margin seemed to give decent performance on all other pictures though.
Automatic Cropping
I attempted automatic croping by noting how the np.roll function behaves when I shifted the r,g, and b channels. Let's say I found the pixel shift of the red channel from the blue channel to be 10 pixels in the y direction and 7 pixels in the x direction. When I shifted red channel using np.roll, it would take the last 10 pixels in the y direction and send it to the very beginning of the picture, and similarly for the x direction. These "overflowed" pixels would obviously create unpleasant artifacts around the borders. To address this, I looked at both the red and green channel and chose the maximum values where I shifted these channels in relation to the blue channel. Then I cropped the images in accordance to how much I shifted the images. The results seemed to work decently.
The upload file size limit was 25 MB and my zipped file was too large for that. To account for that, I just screenshotted some of my original photos to make the size smaller instead. This results in very small black margins on some of the photos when my screenshot was not perfect. Shouldn't affect any of the visible results though for cropping.
Monastery
G offset:[-3,1]
Red offset:[3,2]
Crop
No Crop
Tobolsk
G offset:[3,2]
Red offset:[6,3]
Crop
No Crop
Cathedral
G offset:[7,-1]
Red offset:[12,-1]
Crop
No Crop
Emir
G offset:[49,24]
Red offset:[0,-333]
Crop
No Crop
Harvesters
G offset:[60,16]
Red offset:[124,13]
Crop
No Crop
Icon
G offset:[40,17]
Red offset:[89,23]
Crop
No Crop
Lady
G offset:[53,8]
Red offset:[117,10]
Crop
No Crop
Melons
G offset:[82,8]
Red offset:[178,12]
Crop
No Crop
Onion_church
G offset:[51,26]
Red offset:[108,36]
Crop
No Crop
Self_portrait
G offset:[78,28]
Red offset:[176,36]
Crop
No Crop
Three_generations
G offset:[54,11]
Red offset:[112,9]
Crop
No Crop
Train
G offset:[43,5]
Red offset:[87,31]
Crop
No Crop
Village
G offset:[64,11]
Red offset:[137,21]
Crop
No Crop
Workshop
G offset:[52,0]
Red offset:[105,-12]
Crop
No Crop
Example Images
Lake
G offset:[27,6]
Red offset:[60,5]
Crop
No Crop
Board
G offset:[50,12]
Red offset:[113,19]
Crop
No Crop
Door
G offset:[20,18]
Red offset:[77,28]
Crop
No Crop
Shutters
G offset:[55,46]
Red offset:[127,85]
Crop
No Crop
Shed
G offset:[60,6]
Red offset:[126,4]
Crop
No Crop
