Overview

This project fundamentally relied on making small adjustments to the color overlays in order to cleanly overlap 3 separate images as an RGB color spectrum. For smaller images we use exhaustive search over an interval defined by a range of -50 to 50 and for larger images we use pyramid search to try to determine the optimal interval using a scaling factor of 1/2 with each level of the pyramid search. Our loss function to evaluate the standard loss was simply SSD Loss given in vectorized form by

((img1 - img2) ** 2).sum()

Processing

Image Processing fundamentally relied on the following steps

1. Crop the borders of the image using pixel equalities between the three color spectrums
2. Use either standard or pyramid alignment in order to determine the optimal shift for each image
3. Rescale color intensities and contrasts for final image

Single Scale Processing

Single Scale processing was completed by aligning the green color spectrum against the blue and then aligning the red against the blue. Here alignment means using a range of -50 to 50 pixels as well as

np.roll

to compute possible shifts of the first spectrum. This shift was the compared with the second spectrum (e.g.

align(r, b)

would shift r and compare against the blue color spectrum) using the loss function defined above. For single shift scaling we observed there was often substantial noise around the periphery of the images, thus we only aligned the center 1/2 of the image (1/4 of the pixels were not considered from each edge during alignment). Single shift finally outputs the optimal shift that would most closely align the center of the image with the center of the secondary color spectrum.

Multi-Scale Pyramid Processing

Multi-scale pyramid processing fundamentally relied on the same logic as single-scale processing with substantial optimizations necessary for large images. This algorithm is defined by the following steps:

1. We again crop to focus on aligning only the middle 1/2 of the images.
2. Define a resize factor to scale down the image (Here it is set to 2)
3. Define a number of levels for processing (Defined here as 5)
4. Rescale the image to

   1 / RESIZE_FACTOR ^ LEVELS 

5. Compute the optimal horizontal and vertical shift using exhaustive search across some window (Here window = 5)
6. Magnify the image, horizontal, and vertical shifts by RESIZE_FACTOR and decrement levels
7. Repeat while levels > 0, updating the search region to only search within a border of the scaled horizontal and vertical shifts from the previous level

Because this algorithm relies on heavily downscaling the image, it was not a feasible option for small jpg images (exhaustive search performed much more efficiently).

Post-Processing

After the alignment of the images had completed we used

exp.rescale_intensity

from skimage to rescale color intensities and then wrote to the outfile.

Running the Scripts

The following are the required python packages

            numpy==1.15.1
            opencv-python==3.4.2.17
            scikit-image==0.14.0
            scipy==1.1.0
            stsci.image==2.3.0
            stsci.tools==3.4.13
        

To regenerate all the images simply run

        python3 main.py
    

This will take each image in photos and write an output to "out_path/"

Final Results

Image Cropping