Overview
In this project, we were given RGB glass plate negatives of photos taken by Sergei Mikhailovich Prokudin-Gorskii
and had to align each color to reconstruct a color image. In order to accomplish this, I implemented two possible
approaches: in either case, you must search across a small range of possible x and y offsets (I used offsets in
[-15, 15] as suggested since I didn't see much improvement with a larger range) and select the offset that produced
the images with the greatest amount of overlap (as measured by the smallest SSD). The plates that were actually offset
were the red and green plates.
In order to effectively deal with larger images, I implemented an image pyramid: by scaling the image down by a factor
of 1/2 until it was sufficiently small (I chose 500 pixels as a max but in practice it's probably best to take a
log of the maximum dimension to account for negatives with different sizes). At this point, I calculated the offset
as before, and used this to offset each scaled image (multiplying the offsets by 2 at each level).
To get the best results, the images that I aligned were the result of a sobel filter. This meant that the edges of
the image were given greater weight when aligning. Additionally, I did not take into account the outer 10% of the images
when aligning - this is so the colored borders at the edge did not affect my alignment.
Below are the results: in addition to the image, the offsets for the green and red plates are listed. The offsets
are given in (x, y) tuples. All images were aligned using the image pyramid approach, although the smaller images
ended up, in effect, being produced by exhastive search across [-15, 15].
Results
Observations and improvements
-
It seems that in many cases, the sobel filter had little effect:
However, there were two cases where it seems to have significantly helped.
For each following pair, the image aligned using the sobel filter is first.
In the first set, since each negative is at a wildly different brightness, it is hard
to align just off of hue, so it makes sense to align based on the edges. The second
set has a similar issue on the right side of the image. This is the only image for which
the alginment algorithm was significantly off. However, by changing the fixed color from
blue to green, the alignment is significantly better:
This makes sense, as the image is "dominated" by the blue channel on the right side: it would be
possible to improve it by comparing the relative intensities of the three channels and choosing
which one to have as the "base" channel accordingly.
It is also interesting that in the first pair, the sobel filter is enough to fix this issue but it
is not enough in the second pair. This might be due to the relative position of the saturation: in the
second pair, the saturated area is to the right, but in the first pair, it is centered in the figure.
- Other than that one situation, the alignments were mostly good. There are a few images that are noticably
(albeit slightly) off: one such example is
Notice the cyan "shadow" around the building. It might be possible to tweak this with a more exhaustive search or
different heuristic of measuring alignment, but it would probably be most helpful to attempt to align these by also
rotating and resizing the images slightly (notice how the mis-alignment is more prevelant towards the top).
One other way the image could be improved is to apply an additional filter after reconstructing the image: many images
are quite bright, including
This could probably be as simple the intensities to make the darkest pixel 0, but a non-linear mapping to get greater range
at low intensities would help.
-
There was at least one case where the sobel filter led to a worse alignment:
(Sobel on the left)
It might be the case that the central object here is "too complicated" to easily align using edges.