The goal of this project is to align images from the Produkin-Gorskii collection that are split into red, green, and blue channels. I used exhaustive search and pyramid search to find the best displacements for each channel. I also experimented with using edge masks to find best displacements and to perform automated cropping.
We perform a grid search over different x and y offset combinations and compute a SSD metric between the shifted layer and the base layer. We select the offset that results in the lowest SSD.
Recursively downsampling the image and computing alignments using exhaustive search over small grids at each level.
I calculated edge masks for each image and use them to calculate SSDs of displaced channels, as well as for cropping after alignment.
In exhaustive search, I searched over a grid of x and y displacements ranging from -15 to 15 pixels. For each displacement, I calculated the SSD between the displaced channel and the base channel. I selected the final displacement as the one that results in the lowest SSD metric.
In pyramid search, we call exhaustive search recursively on downscaled images. I downsample the image 5 times, scaling by a factor of 0.5 each time, to generate 6 levels. Then, I start at the smallest level and run exhaustive search to find the best displacement using a grid from -6 to 6 across both x and y. I use twice the displacement of the previous smaller level in order to shift the current level at each recursive step. At the very end, my displacement returned is a sum of the scaled displacements at each level. Originally, I was not getting good alignment with a recursion depth of 4 and a grid size of -5 to 5, but after I increased to a depth of 5 and grid size of -6 to 6 I got good alignment. For Emir specifically, I had to set the green channel as the base instead of blue to get a good alignment.
For extra credit, I used Canny Edge Detection to compute edge masks for images. I then use these edge masks instead of the original images when computing SSDs between displaced images and the base. I also use these edge masks to perform automatic cropping of the final aligned images.
For masks, I perform cropping by checking the outer 5% of rows and columns of the mask. For each row and column, I check how many white pixels there are in that row. If the number of white pixels exceeds a threshold value (num_rows/5 for each column and num_cols/8 for each row), then I mark that row. At the end, I set the innermost marked row/col of left, right, top, and bottom as the cropping border. Here are a few examples of images generated using edge masks to calculate SSD, and for automated cropping.
As you can see, the automated cropping does a good job of cropping out the bands at the edge of the image. Also, using edge masks to compute SSD outputs displacement values that are very close to what pyramid search outputs. Therefore, using edge masks does not necessarily improve performance significantly, but it may make the edges slightly sharper on some images.