CS 194-26 Image Manipulation and Computational Photography

Overview:

The goal of this project was to align images from the Prokudin-Gorskii photo collection of Russia, which were taken with exposures on glass plates using a red, blue, and blue filter to colorize them into a single colored image. This was done using the maximum normalized cross correlation (NCC) as a heuristic following border cropping, and using additional bells and whistles to improve the alignment and ultimate colorization.

Naive Implementation:

I began with a naive implementation. Before calculating the optimal alignment, I would crop 10% from each side of the image to remove the border prior to alignment. For each picture, we then loop through shifts of range [-15, 15] for both x and y to find the optimal alignment, using the maximum NCC to measure simlarity between the channels as a heuristic for alignment. Though this worked well for smaller JPG images, considering all ranges of pixel shifts for larger images would have taken too long.

Image Pyramid Optimization:

Thus, for larger TIFF images, we require a more efficient image pyramid implementation. To align larger images, I rescaled the image down by a factor of 0.5 up until a constant threshold of less than 50 pixels. For the smallest scale, I found the shift in range [-5, 5] for x and y with the maximum NCC after cropping. Thus, for the next scale we can use the best shift from the previous scale (which is x0.5 smaller) multiplied by two as a starting point to search for the optimal shift for the current scale. For example, if the optimal shift at img size 32 x 32 were (-3, 2), we would begin our search for our optimal shift for img size 64 x 64 at (-6, 4) and consider all offsets within [-5, 5] of that offset for x and y.

Obstacles:

Alignment for emir.tif in particular looked odd, due to differences in brightness across the various channels. As a solution to this, rather than align against the blue channel as originally implemented in the skeleton code, I instead switched my implementation to align against the green channel which yielded much better results for some images such as emir, or produced largely similar outcomes for other images.

Emir - Aligned Against Blue: green: (49, 24) red: (128, -571).             Aligned Against Green: blue: (-49, -24) red: (57, 17)

Bells and Whistles:

Automatic Contrasting:

I implemented automatic contrasting using two methods. Firstly, I tried using the OpenCv's cv2.createCLAHE method to create a contrast limited adaptive histogram equalizer to automatically adjust the contrast of the image to improve image quality. This improved the appearance of some images that appeared washed out, however it was difficult to do this while cropping out the image borders so the borders may affect the contrast. I also implemented automatic contrasting using the 2nd and 98th percentile pixels to perform contrast stretching cropping out 10% of the image on each side, which gave a more noticable effect on some images. We show the effect of both on Lady below.

Lady - Before Automatic Contrasting:             After Automatic Contrasting (Percentile):

After Automatic Contrasting (Histogram):

Better Features (Edge detection)

I added edge detection for better feature of alignment by adding a sobel filter to each color channel to create contour images such as the one below. Overall, these alignments ended up largely the same as or slightly different by a couple of pixels than the original alignment strategy. See results on data set below for alignment difference.

Cathedral - Edge

Results on Data Set: (NOTE: Shifts are in (y, x))

These images were compressed due to the 25 MB upload limit, which may result in some graininess as a result of compression for images that were originally lower quality. Bells and Whistles shows the effect using edge detection and automatic contrasting using the percentile approach.

Cathedral - blue: (-5, -2) red: (7, 1).             Bells and Whistles: blue: (-5, -2) red: (7, 1).

Monastery - blue: (3, -2) red: (6, 1).             Bells and Whistles: blue(3, -2) red: (6, 1).

Nativity - blue: (-3, -1) red: (4, -1).              Bells and Whistles: blue: (-3, -1) red: (4, -1)

Settlers - blue: (-7, 0) red: (8, -1).             Bells and Whistles: blue: (-7, 0) red: (8, -1).

Emir - blue: (-49, -24) red: (57, 17).             Bells and Whistles: Emir - blue: (-49, -24) red: (58, 17)

Harvesters - blue: (-59, -16) red: (65, -3).         Bells and Whistles: blue: (-60, -17) red: (64, -3)

Icon - blue: (-41, -17) red: (48, 5).             Bells and Whistles: blue: (-42, -17) red: (48, 5)

Lady - blue: (-51, -9) red: (61, 3).             Bells and Whistles: (-56, -9) red: (63, 3)

Self Portrait - blue: (-78, -29) red: (98, 8).             Bells and Whistles: blue: (-78, -29) red: (98, 8)

Three Generations - blue: (-53, -14) red: (58, -3).       Bells and Whistles: blue: (-54, -12) red: (58, -4)

Train - blue: (-42, -5) red: (43, 27).             Bells and Whistles: blue: (-41, -2) red: (43, 26)

Turkmen - blue: (-56, -21) red: (60, 7).             Bells and Whistles: blue: (-57, -22) red: (60, 7)

Village - blue: (64, -12) red: (73, 10).             Bells and Whistles: blue: (64, -10) red: (73, 10).

Extras:

Village Naziya - blue: (-16, -2) red: (26, 1).             Bells and Whistles: blue: (-16, -2) red: (27, 1).

Pinkhus Karlinskii - blue: (-52, 22) red: (55, -33).             Bells and Whistles: blue: (-54, 23) red: (55, -34).

Church - blue: (-18, -18) red: (29, 11).             Bells and Whistles: blue: (-16, -19). red: (29, 11)