CS 194-26: proj1

IMAGES OF THE RUSSIAN EMPIRE: COLORIZING THE PROKUDIN-GORSKII PHOTO COLLECTION

Approach

• Alignment
  • RGB alignment (alignment as dictated by the project)
  • Aligning to blue (as dictated by the project)
  • Aligning to green [FINAL CHOICE]
  I found for almost all images, alignment to green worked better on my images. This may have been because the greens were more balanced in the images. For example, in emir.tif, the blues were very strong (in the costume).
  • Sobel edge alignment
  In this method, I ran a Sobel edge detector on the images. Initially, although it seemed to work better in some of my images, I found that RGB alignment to green (> blue) produced equal, if not superior results. Hence I used RGB alignment to G for my final project. Note that my offsets are thus calculated for alignment to the green filter (rather than the blue filter). Modification of the code (simply swap green, blue variables can be used to produce the offsets if we were to do the other way around).
• Cost metric
  • SSD [FINAL CHOICE]
  • NCC
  I didn't find a significant difference between the cost metrics and ended up using SSD because it ran faster.

Overview of the Process

1. (I eliminated this step later) Run Sobel edge detection on each R/G/B image to produce the edges of the image.
Here is a sample of what might be produced [R, G, B]:


To achieve this, I applied the filter as found on wikipedia:

Note that I used the function: from scipy.signal import convolve2d to optimize my code. Following this, to enhance the edges a little more, I apply my code later used to increase contrast (effectively just enhances values by a thresh and then remaps them to the range[0, 1]).
2. Run alignment (pyramid scheme) aligning the red and blue to green filter. For this, I first created an image pyramid which takes the image and repeatedly halves the height and width until its smallest dimension is ~4 pixels (done by an operation with the log2 to find out how many times I can halve). Then I apply alignment to each of the sets of images starting with the smallest image, calculating displacement at each step. When I get to the next image in the image pyramid, I apply double the previous displacement calculated (since the image is doubling in size). At the last image, I only apply the displacement calculated (not double), since we are no longer doubling in size.
   Note also that I apply alignment only to the middle 60% of the image (cropping out the edges during calculation of offsets) as when I looked at most of the images, the subjects were in the middle (what I would ideally like to align to, rather than overcomplicated backgrounds).
3. Following this, I rolled my matrices by their calculated displacements.
4. Then I begin image modifications (bells + whistles!)
   1. I first apply a border cropping function. This function is a bit janky; I first calculate the average R, G, and B values for each image, as well as the standard deviation of each. Then I scan through the image and find the first (and last) x and y values where the R, G, B values are all within 0.8 standard deviations of the mean OR cumulatively, their values are within 1 standard deviation of the overall mean, and each individual color is within 1.1 standard deviation of its individual mean. The logic behind this was just that extreme white values should then be cropped out (white borders), and (hopefully) the solely red/green/blue borders arising from the shifts should also be cropped out. This was moderately successful. You can see in the image chart below how images were cropped. My method never over cropped but also left a lot to be desired.
   2. Following this, I apply a color adjustment. I separate all the pixels into a histogram with 100 bins; if the first 25% of the values (white leaning) comprise over 50% of total value, then I modify the curve such that we shift our values over to the darker side (closer to 1.0). Conversely, the opposite applies if the last 25% of our values comprise of 50% of total value. If neither of the above are true, I simply stretch out the histogram curve a little by truncating both ends for up to 2% of all pixels.
   3. Following this, I increase contrast of the whole image by purposefully multiplying values to beyond a [0, 1] ratio (default set to -0.25, 1.25) and then truncating the results such that we are back in a [0,1] range, making colors a bit more vibrant.

Issues

Result On Sample Images

Summary Table of Offsets

Image	Red Offset	Blue Offset	Before Bells and Whistles	After Bells and Whistles
cathedral.jpg	[1,7]	[-2,-5]
monastery.jpg	[1,6]	[-2,3]
nativity.jpg	[-1,4]	[-1,-3]
settlers.jpg	[-1,8]	[0,7]
emir.tif	[17,51]	[-24,-47]
harvesters.tif	[-3,59]	[-17,-51]
icon.tif	[5,47]	[-17,-37]
lady.tif	[3,59]	[-8,-47]
self_portrait.tif	[6,73]	[-26,-65]
three_generations.tif	[-1,51]	[-14,-47]
train.tif	[26,41]	[-6,-39]
turkmen.tif	[7,53]	[-20,-53]
village.tif	[10,63]	[-11,-57]
berries.tif	[9,26]	[-10,15]
holy_mother.tif	[-3,53]	[-13,-39]
tsar_gifts.tif	[14,45]	[-19,-24]
resurrection.tif	[-15,51]	[2,-45]

Visualization After Bells and Whistles Progression

No Adj	After Color Adj	After Color and Contrast Adj

Resources Used