Images of the Russian Empire

Single-scale Alignment

After separating the 3 plate negatives into 3 separate images, naively overlapping the images directly on top of each other will result in a blurry image (as seen below). To recreate a clearer image, we will find the (x, y) displacement vectors necessary for aligning the red and green plates over the blue plate.

For smaller images, we can just exhaustively search over the potential ranges that we want to displace the image by and check if that particular displacement is a "good alignment" or not. We determine a "good alignment" with a high normalized cross-correlation: the inner product between the normalized vector of the red/green and the blue image. Since we want to align the subject and not so much the black and white edges, we crop the edges so that the NCC is more representative of a "good alignment".

Single-scale alignment works great on the small images. There are some digital artifacts near the border because the plates don't align exactly; however, this can easily be fixed by cropping.

Unaligned	Aligned	Offset
		G (5, 2) R (12, 3)
		G (-3, 2) R (3, 2)
		G (3, 2) R (6, 3)

Multi-scale Alignment

Although an exhaustive search across all possible displacements worked well for smaller images, we will need a much more efficient method to search for the displacement vectors for larger images. Using an image pyramid decreases the amount of search necessary (especially compared to the exhaustive naive approach) and heavily improves the runtime of our algorithm.

According to OpenCV, image pyramids are a "a collection of images - all arising from a single original image - that are successively downsampled until some desired stopping point is reached." This is incredibly useful for search, since it costs significantly less to search on a downsized image. Our algorithm first recurses down 5 times, reducing the image by half each time, and solves for the best displacement for the smallest image. Then we search within a small range of the so-far best displacement with the second smallest image, and so on.

While running the algorithm on the larger images, we discovered that NCC did not work well for aligning Emir and the lake. This is probably because the plate "brightness" can differ depending on the color captured via the camera. For example, Emir's clothes on the blue plate is much brighter comapred to Emir's clothes on the red plate.

The lake plates also suffers from the same problem. In addition to that, we think because there are not many defining features in the image, it can be hard for NCC to determine exactly how to overlap. For both of these images, NCC was not be a good estimator for "good alignment".
(Open image in a new tab for full resolution)

Unaligned	Aligned	Offset
		G (49, 24) R (93, -305)
		G (59, 16) R (124, 13)
		G (40, 17) R (89, 23)
		G (48, 9) R (112, 11)
		G (82, 10) R (179, 13)
		G (51, 26) R (108, 36)
		G (78, 29) R (176, 37)
		G (53, 14) R (112, 11)
		G (42, 5) R (87, 32)
		G (65, 12) R (137, 22)
		G (52, 0) R (104, -12)
		G (14, 19) R (39, 19) Village of Vilizhno on the Chusovaia River
		G (7, 20) R (8, 41) View of Ostashkovsky stretch of Seliger Lake from the embankment of Nilov Monastery
		G (74, 34) R (156, 53) Fat sheep
		G (69, 25) R (146, 36) Kush-Beggi

---

Extra Credit

---

Better Alignment

We found that NCC was not necessarily the best feature for edge alignment, especially for images like Emir that have a large difference in image intensity across plates. To properly align Emir and the lake, it would probably be a better idea to find the similarity of images in terms of edges instead of raw pixel values. Our preprocessing now involves converting our image to edges with Canny edge detection (found through some quick Googling).

Automatic Cropping

The images are now nicely aligned, but it would be even nicer if we can automate the cropping process. Ideally, we would want an algorithm that can detect the ugly border lines and remove them. Luckily, with a quick Google search we discovered a very relevant algorithm for this job: Hough Transform. Hough Transform attempts to identify straight lines in an image. Using Hough Transform directly on the aligned image did not work great-- there were too many other straight lines that the algorithm was picking up on, and it would often not pick up the lines of colored borders.

We found it more effective to convert the aligned image to just edges with Canny edge detection, before finally putting it through Hough Transform. There were more parameters to play around with-- we made the images show more edges (specifically enough for the colored border to show) and we picked lines that were long enough to potentially be a border. We still found many lines for some images, but we were able to identify the borders by keeping lines that were exactly vertical and horizontal close enough to the edge of the image.

This method worked reasonably well for most images, but probably needs some more tuning.