For each channel in (r, g, b):
For each side in (top, bottom, left, right):
Average the values of a 1xN strip of pixels along that edge
If the average is > .9 or < .1, it is part of a white or black border, and can be cropped
I scanned each side up to a depth equal to 1/12th of the overall image. This was because for lighter background images, such as the nativity, my algorithm would happily crop as much as it could of the sky! If I had more time, I would investigate using edge detection to do the cropping instead. The results of my cropping on the train image are shown below. For the most part I am very happy with how it turned out.