To start the project, I implemented an inefficient search, which searches over a predefined displacement window.
I eventually ran into problems with runtime, after I implemented the code using nested for loops. To improve runtime, I focused on using vectorized numpy functions, but I eventually settled on using a list comprehension instead. The exhaustive search computes the best match within several displacement windows.
After trying both SSD and NCC, I found that SSD produced better results, so I used it in my implementation.
After a lot of testing, I found that the optimal bound was 30 pixels when rolling the image. I came to the conclusion through trial and error. Hence, when using only the exhaustive search, I try all shifts in [-30, 30].
To improve efficiency and results, I implemented cropping during the exhaustive. I implemented a function that crops all but a specified percentage of the image. The cropped images are used when measuring alignment.
I tried many percentages, from 40% to 95%, and determined that measuring with the middle 50% worked best on most images, as it preserved quality, while improving speed.
As recommended in the project specification, I implemented a recursive pyramid search to improve efficiency for larger images. The function calls the exhaustive search function for each level. After tuning the variables, I found that scaling by a factor of 3 at each level, with 3 levels, was optimal. By using a larger factor, I was able to reduce the number of levels, and therefore, improve the runtime. The runtime issues that I mentioned in part 1 occurred while I was testing my pyramid search. The maximum search window size increases with each level, with a maximum of 15, to improve runtime. When using this algorithm, the Emir image did not align well, presumably due to the color scheme. This motivated the change of base color that I added as an extra feature. The key upgrade to this algorithm is that it crops a, with enough pixels to the left and at the top such that when we look at the middle 50% of the original image, the rolling occurs in the same way it would on the original image. Hence, we can crop first, and roll a much smaller image, thereby saving a lot of time. This method significantly improves runtime, and the additional pixels ensure that the alignment quality is the same as it would be if we rolled the entire image then compared the middle 50%. I used the same Matching Metric (SSD) and Cropping percentage (50%) as in part 1.
After experimenting with a variety of combinations, I decided to align the color layers to the green layer, rather than the blue one. It especially improved the results for the emir image, and made slight improvements on the others. See the before and after below.
While I attempted to make a function that found borders based on their grayscale value, removing lines of pixels on the edge that were either mostly black, or mostly white, I decided to settle with cropping a constant portion of the images, as it tended to work on most images. Unfortunately, after cropping the jpg images, they became more pixelated. This is because they are small to begin with. Overall, cropping improved the look of the images by getting rid of unecessary bordrs, and getting rid of the odd bars of color produced by the shifts I made. Cropping also made it easier to apply normalization later.
I used a skimage function to normalize the colors. The results were mixed, but the normalization was definitely impactful, and on average, it improved the looks of the images. When I cropped after normalizing, I noticed that the Church image looked better, but most images turned out better when I cropped first. This feature had a negative effect on the church image, as a result. I love how it turned out for the Emir and the Onion Church, as it made both images more vibrant.
Below is each output image, with the color offsets found by the core algorithm. I chose the last three photos myself. I downloaded .tif files for each of them, and used pyramid search. All .tif files were processed using the pyramid search, while .jpg files were processed with the exhaustive search from part 1. Some images are shown side by side to show the before and after of the bells and whistles for select images.
tobolsk: Green offset: (3, 3), Red offset: (7, 3)
cathedral: Green offset: (5, 2), Red offset: (12, 3)
monastery: Green offset: (-3, 2), Red offset: (3, 2)
workshop: Green offset: (54, -1), Red offset: (107, -12)
church: Green offset: (25, 4), Red offset: (58, -4)
emir: Green offset: (48, 24), Red offset: (71, 42)
three_generations: Green offset: (53, 15), Red offset: (111, 12)
melons: Green offset: (81, 10), Red offset: (178, 14)
onion_church: Green offset: (51, 27), Red offset: (108, 37)
train: Green offset: (42, 6), Red offset: (87, 33)
icon: Green offset: (41, 17), Red offset: (90, 23)
self_portrait: Green offset: (78, 29), Red offset: (175, 37)
harvesters: Green offset: (59, 17), Red offset: (123, 14)
lady: Green offset: (55, 8), Red offset: (117, 11)
lasnaia: Green offset: (41, -1), Red offset: (96, 9)
flowers: Green offset: (50, -5), Red offset: (97, -24)
sunset: Green offset: (43, 23), Red offset: (0, -27)
