Wenbo_Wang_Proj1

Align R_G_B images using Exhaustive SSD Search and Pyramid SSD Search

I. Introduction

This is a python program for recovering color images from the digitized Prokudin-Gorskii glass plate images, which are R, G, B channel images.

The Sum of Squared Differences (SSD) is used here for scoring how well the images match: SSD = sum((image1 - image2)^2).

Two align/search methods, exhaustive SSD search and pyramid SSD search, are implemented here, which can be tested accordingly in files exhaustive_search_general.py and pyramid_search_general.py

A file named main.py is created for test convenience.

II. Exhaustive SSD Search

1. Introduction

The first search method that we implement is the exhaustive ssd search, which will go through all of the possible x, y shift in the search range and record the best shift decided by the SSD score.

2. Choice of search range, max shift and search step

Instead of comparing entire images and making 1 pixel step for each time, we can set different search_window_rate , search_range_rate , and search_step_rate according to different image size.

* search_window_rate = 0.25 # window range length/ max image range length
* search_range_rate = 0.05 # max movement length / max image range length
* search_step_rate = 0.05 # min search step / max movement length

In this project, the 2-channel images that are smaller than 500 by 500 (i.e., three jpg images) will be processed with its 90% image range length to cut off the noisy fringe. The max shift of the image is set as 4% of the image range length, and the x, y searching step is designed as 1 pixel. This means for a 400 by 400 image, the valid image range will be 360 by 360, the max shift will be 16 pixels, and the searching step/resolution will be 1 pixel.

While the images that are larger than 500 by 500 (i.e., all of the tiff images) will be processed with its 25% image range length. The max shift was set as 5% of the image range length and the search step was designed as 5% of the max shift. For a 3200 by 3200 image, the valid image range will be 800 by 800 pixels, the max shift will be 160 pixels, and the searching step/resolution will be 8 pixels.

3. Searching result

The exhaustive searching result for all of the example images are shown below.

Exhaustive_Search_Result

In this search, the average searching time for the small images is 4s, while for the larger images is over 50s. The search quality for these two kinds of images can be seen as below.

Exhaustive_Search_cathedral.png
Exhaustive_Search_tobolsk.png
Exhaustive_Search_train.png
Exhaustive_Search_lady.png
Exhaustive_Search_three_generations.png
The extra image downloaded from the Prokudin-Gorskii collection is gruppa.tiff. The search result for it is shown below.

Exhaustive_Search_gruppa.png
The search for images emir.tiff and melons.tiff failed in the exhaustive search, which are shown below.

Exhaustive_Search_emir.png
Exhaustive_Search_melons.png

This failure happened due to the different brightness of the original R_G_B images. The x, y shift calculated here indeed produces a smaller SSD value compared with the right x, y shift.

However, this failure disappeared in the pyramid ssd search, thanks to its iterative range.

III. Pyramid SSD Search

1. Introduction

The second search method is the pyramid search, which starts from the coarsest scale (i.e., 1/16 of the original image length) and goes down the pyramid (i.e., 1/8, 1/4, 1/2 and 1 of the original image length) to update the estimate shift. The recursive function was used here to make the iterative search.

2. Choice of pyramid level, search range, max shift and search step

In this project, the pyramid search starts from the 4th pyramid level (i.e., 1/16 of the original image length) and ends after the 2nd pyramid level (i.e., 1/4 of the original image length).

The 1st and zero pyramid search were omitted after finding that their results were much similar to the twice and four times of the 2nd search result. Hence, the final x, y shift can be obtained by multiplying 4 to the 2nd pyramid search result.

The search range here is set as 90% of the original image, which will cut off the noisy fringe image.

The max shift will change between different pyramid levels(i.e., 6 for 4th, 8 for 3rd, 10 for 2nd level search). This will reduce the searching error and increase the searching resolution.

The search step here was assigned as 1 pixel.

3. Searching result

The pyramid searching result for all of the example images are shown below.

Pyramid_Search_Result

The average searching time for the small images is 0.8s, while for the larger images is 12s. The search quality for these two kinds of images can be seen as below.

Pyramid_Search_cathedral.png
Pyramid_Search_harvesters.png
Pyramid_Search_icon.png
Pyramid_Search_onion_church.png
Pyramid_Search_village.png
The extra image downloaded from the Prokudin-Gorskii collection is krest.tiff. The search result for it is shown below.

Pyramid_Search_krest.png

The search for images emir.tiff and melons.tiff succeeded in the pyramid search, which are shown below.

Pyramid_Search_emir.png
Pyramid_Search_melons.png

IV. Search Comparision

The Comparision of exhaustive search and pyramid search can be seen as below.

Search_Comparision.png

While comparing, we assume the exhaustive search result is the "right answer" of the problem, since the x, y shift indeed reaches the smallest SSD value. In this case, we can find that the difference between the pyramid search result and exhaustive search result can be controlled within 4 pixels, which is the search resolution of the 2nd pyramid level.

In pyramid search, the average searching time was shorten by analyzing the higher pyramid level image and restricting the searching range. The search for images emir.tiff and melons.tiff were succeeded here thanks to this search range restriction.

In conclusion, the pyramid search is much more time efficient than the exhaustive search, but with more complex structure.