Images of The Russian Empire
Hongzhan (Jeff) Yu, cs194-26-acc, cs194-26 project 1
OVERVIEW
In this project, we are asked to reconstruct color images from digitized Prokudin-Gorskrii glass plate images with as few artifacts as possible. In brief, we separate the original image into three color channels corresponding to B, G, and R in the beginning; then the displacement vector that best describe the relative position of G or R with B can be calculated based on Normalized Cross-Correlation to decrease the misalignment. Finally, color images can be produced by stacking up these three channels in the order of R, G, and B.
Regarding of the setup, simply run the script with
python3 main.py target.file --flag1 --flag2
As long as the entered input file exists in the same folder with script file's, the output file would be produced and can be found under the same folder. Note that the name of the output file varies with the change of flags. And here's the specification of all working flags:
| --edge | Edge Detection |
| --contrast | Automatic Contrasting |
| --crop | Automatic Cropping |
|
|
APPROACH
Based on the introduction described above, this project can be separated into two parts: one is to find the displacement vectors by applying Normalized Cross-Correlation, and another is to create a Image Pyramid; The concept of Image Pyramid is introduced to improve the performance efficiency-wise.
• Normalized Cross-Correlation (NCC)
Instead of explaining the arithmetic procedures, I prefer to explain how the adoption of NCC helps: in order to make 3 separated channels more stackable and the alignment more precise, G and R channels should be modified to have as much similarity with B channel as possible to avoid the artifacts due to misalignment. NCC helps to quantify the amount of overlapping between two images. Within certain window interval (16 by default), the displacement vector that maximizes NCC is the one by which the image should be translated.
• Image Pyramid
When we are dealing with large images (.tif files in our cases), the running cost would rise sharply if exhaustive search is still being used, due to the poor scalability of matrix multiplication. Thus we are suggested to use Image Pyramid. Downscale the original image to an acceptable level first, and consider it as our starting level. Then we derive the best displacement vector at the lowest resolution level, and keep upscaling the image and updating the displacement vector in the meanwhile until the original resolution has been reached.
To emphasize the benefits of Image Pyramid, the time cost for the reproduction of each images would be listed below in the GALLERY section. In addition, I've tried to run the exhaustive search to reproduce emir.tif without the use of Image Pyramid, and it took nearly 5 minutes.
BELLS & WHISTLES
• Auto Contrasting
The basic idea of my Auto Contrasting implementation is to set the color of the brightest pixel to white and the darkest pixel to black, and then rescale the colors of all other pixels based on linearity. After following the instructions from Jun-Yan's info-session, my images are produced in a more vivid way!
|
|
• Edge Detection
With the assistance of Canny Engine, I'm able to detect the edges in each images so that the misalignment due to brightness differences would be avoided. To be more specific, I pre-processed each color channels with Canny edge detector before applying any aligning procedures, and it turned out that the color image does look much clearer now.
|
|
• Auto Cropping
When I just started the implementation of Auto Cropping, I chose to find the patterns hiding in the colors of those images that contain wired edges from misalignment. I noticed that those edges are usually composed of the colors in which there exist obvious gaps among channels. Therefore, the criteria I set is whether, in the average color of row/column, one channel's value is greater than another's by 0.25. If this criteria is met, that row/column would be eliminated.
|
|
However, my implementation is not working quite well in certain cases. For example, turkmen.tif got totally ruined by it, and, thus, the one shown in GALLERY was produced with Auto Cropping disabled.
|
GALLERY
• .JPG Files
g: (2, -3) r: (2, 3) time: 0.804 sec |
g: (1, 3) r: (0, 7) time: 0.829 sec |
g: (0, 7) r: (-1, 14) time: 0.803 sec |
g: (2, 5) r: (3, 12) time: 0.821 sec |
• .TIF Files
g: (17, 60) r: (14, 123) time: 27.9 sec |
g: (23, 49) r: (40, 105) time: 28.3 sec |
|
g: (17, 41) r: (23, 89) time: 25.2 sec |
g: (12, 64) r: (23, 137) time: 29.8 sec |
g: (8, 55) r: (12, 117) time: 28.6 sec |
g: (13, 53) r: (11, 110) time: 25.5 sec |
g: (5, 43) r: (32, 87) time: 29.8 sec |
g: (21, 56) r: (28, 115) time: 26.6 sec |