Overview

The main goal of ths project is to align and restore the color image from three separate images of red, green and blue channels. There were two different approaches to the alignment, one approach was to use recursive window sliding with image pyramid and the other method was to detect features in the image and estimating the best affine transformation relating two images. Cropping on the other hand is achieved by detecting straight lines around the edge of the image and finding the largest inscribed rectangle.

How it works

1. Sliding Image

Image was aligned by translating images with various offsets and selecting an offset that minimizes the chosen image similarity metric. SSE was chosen as the image metric for this project.

However, the search space for the offset is too big (potentially (-100, -100) to (100, 100) or bigger) to search exhaustively. To reduce the search space, we can get the approximate offset using resized version of the images and recursively find more accurate offset. For example, moving 1 pixel with 8x reduced images have an effect of moving 8 pixels on the original image. In this project, images were resized by 2x until the height of the image gets smaller than 128 pixels and the offset of (-4, -4) to (4, 4) were searched for minimum SSE.

Also, Canny Edge Detection is used before searching for better results. Comparing the edges yielded better alignment as different channels have different values which caused some misalignment with unprocessed images.

Left: Alignment with unprocessed image
Right: Alignment with Canny

2. Feature Detection

The problem with method 1 is that it only searches the offset space even when the image requires scaling and shearing for perfect alignment. By detecting key points in both images and aligning them resulted in the best result, and this method was able to align all of them with no misalignment in any part of the image.

First, the keypoints of the images are detected with Accelerated-KAZE algorithm. Then, the best affine transformation matrix is calculated by least-squares approximation. To eliminate the error due to outliers (for example, keypoints on the border), outlier keypoints are eliminated by measuring the residual error and checking if the error is larger than 1 standard deviation. This outlier elmination is repeated until the standard deviation is smaller than an arbitrary value, 2 for this project.

Red: outliers, Green: transformed keypoints, lines: residual error of the outliers

3. Cropping

Cropping is done by detecting straight lines from the edge of the image. First, Canny Edge detection is performed, and among the detected edges only the edges that are straight enough and on the edge of the image are filtered. Then, the interior area is determined by extending the detected straight lines to the edge of the image. Finally, the largest interior rectangle is calculated and the image is cropped based on that rectangle.