Approach
Simply stacking the three images for any scene shows the work needed to be done.
Alignment
In order to align the red and green images to the blue, I first exhaustively searched for the correct displacements. Over a range of [-15, 15], I would circularly shift one image and calculate the sum of squared differences (SSDs) between the shifted image and the blue image at each step. I did this both horizontally and vertically and recorded the shift along each axis that resulted in the minimum SSD on that axis.
It is important to note that I actually did not compute the SSD over the whole image, but instead I only calculated it over the center of the image by slicing it as [height//4: -height//4, width//4: -width//4]. This was done in order to not take into account the noisy borders that would be made even more noisy upon circularly shifting at each step.
I would then stack the images shifted appropriately and save the resulting image. This method worked well on smaller images of around 400 x 350 pixels, but it did not run fast enough on the bigger images of around 3700 x 3200 pixels because the search range would need to be much wider than [-15, 15].
Aligned Small Images
Unaligned
|
Aligned
|
Unaligned
|
Aligned
|
Image Pyramid
I therefore enhanced the algorithm to downsize each image to under 400 pixels in width and height by factors of 2. Using this image pyramid, I started at the smallest image at the top of the pyramid and used my original algorithm therein to search a range of [-30, 30] for the optimal shift. I would then go down one level and calculate the correct alignment using the previous level's shift doubled and the same range. I repeated this until the top level, and the algorithm would only have to make slight adjustments at each level, thereby keeping the search range small and improving the speed.
Middle-of-Photo Image Pyramid for Blue Channel
Level 5
|
Level 4
|
Level 3
|
Level 2
|
Level 1
|
This algorithm performed well, as can be seen in the following aligned images.
However, there were still improvements to be made, as the simple metric of SSD using the raw pixel values could not be guaranteed to produce optimal results due to brightness differences in the different channels' images. This only showed to be a problem in "emir", and I was able to rectify it by aligning to the green channel rather than the blue one.
Aligning by a Different Channel
Unaligned
|
Aligned by Blue Channel
|
Aligned by Green Channel
|
Edge Detection
In order to ensure near-perfect alignment, I utilized the Roberts edge detector implemented in scikit-image's filters.roberts function on images when going through pyramid alignment since the resulting edge definition made the SSD metric more accurate.
Middle-of-Photo Image Pyramid for Blue Channel with Roberts Edge Transformation
Level 5
|
Level 4
|
Level 3
|
Level 2
|
Level 1
|
Auto-Cropping
I went further to try to automatically remove borders based on the variance between the three channels, and it worked well for the colored borders but not as well for the black and white ones. I found that just looking at raw pixel value for those worked well, but that heuristic did not hold across all images, so I restricted the crop to about six percent of the axis length to prevent overcropping. I found a reasonable threshold for the variances and values and also took those over a mean of a group of columns or rows to try to make it more robust.
Uncropped
|
|
Auto-Cropped
|
Auto-Contrast
I went even further and performed auto-contrasting on the images using scikit-image's exposure.equalize_adapthist function, the result of which is shown below.
Auto-Contrasted Photo
|
|
Results with Channel Shifts [y,x]
cathedral G: [5,2] R: [12,3]
|
emir G: [49,24] R: [107,40]
|
harvesters G: [60,18] R: [124,14]
|
icon G: [42,17] R: [90,23]
|
lady G: [56,9] R: [115,13]
|
monastery G: [-3,2] R: [3,2]
|
nativity G: [3,1] R: [8,0]
|
self_portrait G: [78,29] R: [175,37]
|
settlers G: [7,0] R: [15,-1]
|
three_generations G: [53,13] R: [111,9]
|
train G: [43,8] R: [86,33]
|
turkmen G: [56,22] R: [117,29]
|
|
village G: [65,11] R: [137,21]
|
cherez G: [19,17] R: [43,22]
|
rabota G: [39,26] R: [92,41]
|
pastushok G: [58,13] R: [119,19]
|
