Arjun's CS294-26 HW1
Below I describe the approach I took creating color images from the RGB screens.

1. Split the image vertically into 3 equally sized sub images. The hope is that the Blue Screen is fully in the first sub image and so on.
2. Apply heuristic algorithms to find dirty borders and edge intensities
   • The dirty border algorithm first looks left to right. It examines the absolute value difference between neighboring columns. The middle region in between the spikes is assumed the valid area.
   • Edge intensities are defined as the max between horizontal absolute pixel change and vertical.
3. Allign Green to Blue then Red to Blue attemping to minimize the difference between edge intensities for the screen pairs. Edge intensities provided a more grounded signal that should actually match as opposed to raw color values from different channels.
   • Allignment is only computed for valid pixels as defined by the dirty border algorithm
   • Allignment is done in a hierarchial fashion where we start with a coarse granularity and large maximum delta, then iteratively divide granularity and maximum delta by 2.
   • To avoid setting image size specific parameters, the finest granularity, FINE, is defined to be 0.1% of the image dimension (or 1 if the image dim < 1000px). The subsequent granularities are FINE * 2, FINE * 4... The maximum delta is 10 or enough to allow a 10% shift whichever is smaller.
4. Touch up images by rerunning dirty border removal algorithm. with all the channels.


To summarize the 3 additional items are:
• Dirty Border Algorithm
  • Using it to mask out border pixels for alignment
  • Using it to eliminate the border for final display
• Using edge feature similarity rather than raw value for alignment
Some ablations are included at the end of this webpage

Here are the outputs for the example small images. Deltas are shown with respect to the blue screen. The title format is Image_Name (Green dX, Green dY) (Red dX, Red dY)

monastery (2, 3) (2, -3) (small)


cathedral (2, -5) (3, -12) (small)


tobolsk (2, -3) (3, -6) (small)


icon (18, -42) (24, -90)


train (3, -42) (30, -84)


emir (24, -48) (39, -108)


melons (12, -81) (12, -177)


harvesters (18, -60) (12, -123)


workshop (0, -51) (-12, -102)


three_generations (12, -54) (9, -111)


church (3, -24) (-3, -57)


onion_church (24, -51) (36, -108)


self_portrait (30, -78) (36, -174)


lady (9, -57) (12, -120)


field (30, -24) (42, -60) (personal selection)


castle (9, -45) (9, -96) (personal selection)


And a couple ablations. Photo on the left with basic implementation, those on the right include the few additions listed above.


Here the issue remedied is the remaining border.


For smaller images in addition to the border, the shift is often more impacted by including invalid pixels.