CS194-26 Project 1: Colorizing the Prokudin-Gorskii photo collection
Overview:
Our objective is to align images by moving around the X and Y to find the position where the red, green, and blue plates "line up" to produce a proper RGB image.
Each color plates have their own intensities at various points which are offset by an unknown amount (since they were taken at different points but of the same image), each of which need to be combined to make a full image.
This involves stacking 3 plates (Red / Green / Blue) into a 3D numpy array which can be displayed by matplotlib or skimage.io.
Exhaustive search:
For small images, we simply traverse each possible X and Y for a range of -30 to 30 of the red and blue (compared to the green) to find the position with the greatest match.
Setting the green as relative worked out the best as it was both in between the two in the RGB stack as well as consistently the most similar to red and blue (as opposed to blue and red being similar to each other.)
We score the greatest match by as based on the smallest L2 norm, which is taken by subtracting one image from another and then squaring the difference. The minimum differential was used as the most aligned image.
red : ( -1 , -7 ) blue : ( 2 , 5 )
red : ( -1 , -6 ) blue : ( 2 , -3 )
red : ( -1 , -4 ) blue : ( 3 , 3 )
Pyramid search:
For larger images we break down the images into courser images via nd.imgzoom recursively for a given number of steps (I chose 4 steps, or 4 levers of exponential recursion) and then run exhaustive search to find the best fit, using the previous layer as a starting point.
Not all worked to a great effect.
I also changed from using padding the width by a given window to using a roll function.
Moreover, I scaled shift windows by powers of two for each level, making a simple yet nearly identical method of scaling for larger image sizes while maintaining the window size integrity all while reducing compute time.
red : (116, 9 ) blue : (-114, 4)
red : (65, -3) blue : (-64, -1) --
Seemed to have a misaligned Y for the blue and red images, possible due to the window sizing: red moved very high, while blue moved very low. This could be due to the variety between red and blue values in intensity in the image array.
red : (48, 5) blue : ( -46, 4)
red : (44, -10) blue : ( -42 , -9)
red : (96, 3) blue : (-94, 3)
red : (57, 10) blue : ( -57, 8)
red : (97, 5) blue : (-96, 3)
red : (57, 0) blue : ( -56, -1)
red : (111, 1) blue : ( -110, 3) --
Also has a varied Y axis for blue vs red. Similar to emir. It seems like the blue aligned very low compared to the red and green plates.
red : (51, -11 ) blue : ( -50, -11)
Bells and Whistles
Autocropping: We ideally want to remove the dead space, or the black/single color areas around the edges. The way I attempted to implement this is to progressively search each row if the variance or image sum was too extreme (all the same color, or too close to black or white for any plate).
We search inwards on each border until we find a line that seems normal, and stop cropping for that border.
A crude method with imperfect results: A few single color planes (like the bottom of cathedral) aren't being cropped because it seems like there is data for a red and blue in that area as well as green. Perhaps the green plate is just larger.
Some points (left of monastery) have data that seems like a dead zone to a human but has some data that is read by the computer (and on closer inspection, it seems like there is an attempted image there; just gibberish if you zoom in).
Selected Photos
House image, downloaded as a jpg and ran using exhaustive search. red : ( 0 , -6 ) blue : ( 0 , 3 )
Flowers in a meadow, downloaded as a tif and ran using pyramid search. red : ( 47 , -17 ) blue : ( -42 , -13 )
Roses, downloaded as a tif and ran using pyramid search. red : (54, -8) blue : ( -50, -5)