My approach

The ultimate task of this project is to take 3 images representing a photo taken through 3 different filters (Red, Green, Blue) and to realign them to reproduce the colored image. My approach was to first come up with a naive approach to align the image. My first approach was to try to lay images ontop of each other trying out every position in a 15 pixel radius displacement window. I used the SSD of the two overlayed images as my metric and saved the shift with the best correlation as my "best shift". This however did not work for bigger images in which a displacement window will either be too small to be effective or to large to be efficient.

Image Pyramid

To fix this problem, I implemented the image pyramid into my alignment. My approach was to resursivley scale down the image to a size in which a 15 pixel displacement window was both efficient and effective. Then I recusively went back down the pyramid taking that "best shift" and dynamically fine-tuning it by checking an exponentially smaller displacement window as I went down the pyramid and always keeping the shift that had the best correlation.

Cropping and Edge Detection (Canny Filtering)

These two together still produced unhappy results on certain images such as monastery.jpg and harvesters.tif

The problem: There is often a black border around the images which can give innaccurate score calculations.

Solution: Crop the image and run scoring on the middle 90% of the image, cutting out the 10% of the border.

While this solution improved some images, it did not solve all, such as emir.tif another solution was needed ...

Solution: using another metric such as the canny filter (I tried the sobel filter as well and found that the canny filter was ultimately more successful at aligning images). The filter essentially detects edges in images and produces a new image in which only the edges show up as pixels. This is great for edge heavy images such as emir and armor and worked great for those images. This method was ultimately successful because we could focus on scoring the borders instead of trying to score every single pixel in the image. Since the edge is a good factor of alignment, it generally produced good alignments on most images.

Overall, I found that there was easy way to simply align a picture. Different pictures responded better to different approaches. Sometimes the naive approach without cropping or filtering worked best, while other times cropping was the solution, and for certain images the only thing that was needed was to align based on the canny filter (such as emir.tif).

Offsets for cathedral.jpg

Naive: Green: (1, -1) Red: (7, -1)

Cropped: Green: (5, 2) Red: (0, 1)

Canny: Green: (8, 0) Red: (8, -1)

Offsets for nativity.jpg

Naive:
Green: (3, 1) Red: (7, 1)

Cropped
Green: (3, 1) Red: (7, 1)

Canny: Green: (3, 1) Red: (7, 1)

Offsets for monastery.jpg

Naive:
Green: (-6, 0) Red: (9, 1)

Cropped
Green: (-3, 2) Red: (3, 2)

Canny: Green: (-3, 1) Red: (3, 1)

Offsets for settlers.jpg

Naive:
Green: (7, 0) Red: (14, -1)

Cropped
Green: (7, 0) Red: (14, -1)

Canny: Green: (7, 0) Red: (15, -1)

Offsets for harvesters.tif

Naive:
Green: (118, -3) Red: (120, 7)

Cropped
Green: (60, 16) Red: (124, 13)

Canny
Green: (60, 18) Red: (123, 10)

Offsets for lady.tif

Naive:
Green: (57, -6) Red: (123, -17)

Cropped
Green: (53, 8) Red: (117, 10)

Canny
Green: (57, 9) Red: (123, -24)

Offsets for self_portrait.tif

Naive:
Green: (50, -2) Red: (130, -5)

Cropped
Green: (78, 28) Red: (24, 0)

Canny
Green: (77, 29) Red: (117, -8)

I found these image to be the only one that I felt was insufficient. Neither cropping nor canny filtering improved the slightly blurrly naive alignment. I believe this is because the images are more seperated between mostly blue colors, mostly green colors, and mostly red colors as well as the lack of defined edges and objects in the photo.