Project 1 – Gary Yang

1. Intro

Goal: take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible.


Method: I divide the raw input glass plate images from top-to-down into B,G,R channels, align them and stack them in RGB order to generate the appropriate output. For the alignment method, I search exhaustively over a window of possible displacements with [-15,15] on the x-axis and y-axis respectively. The metric I used is the L2 norm, whichever displacement had the smallest L2 norm is chosen as the result. For the image pyramid, I first resized the image to begin the first coarse alignment at either when the height or width of the image is smaller than 512 or the pyramid is 4 layers deep. For each layer I decrease the size by 2x2=4. After getting the initial alignment, I fine tune it with the center 100x100 region of the bigger images. For Bells and Whistles, I approached the goal in two different ways: 1)using edges[1] as the input features in lieu of the raw pixel values; 2)using the ORB detector[2] to find matching keypoints and then find the homography between the image that needs to be aligned and the target one. It is worth noting that the ORB detector is a double-edged sword: 1)good: it does not need image pyramids to process big images and 2)bad: if the brightness between channels change too much, it would have a high failure rate.


[1]https://docs.opencv.org/trunk/da/d22/tutorial_py_canny.html
[2]https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html


2. Results

Here is the table summarizing the outputs:

Not aligned

Aligned with raw pixel values

Aligned with edge features (Bells & Whistles)

Aligned with ORB keypoints (Bells & Whistles)

Reason for failure


Displacement of green channel: dx: 2 dy: 32

Displacement of red channel: dx: 5 dy: 98

Displacement of green channel: dx: 1 dy: 33

Displacement of red channel: dx: 5 dy: 97

The homography matrix between green channel and blue channel is: [[ 1.01243800e+00 2.88669284e-04 -1.34366645e+01]

[ 4.46747793e-03 1.00893696e+00 1.66294698e+01]

[ 1.76816879e-06 5.81088759e-07 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 9.83233785e-01 -6.18014345e-03 3.20364896e+01]

[-2.20593997e-03 9.86229016e-01 1.10286553e+02]

[-2.91438583e-06 -1.60806960e-06 1.00000000e+00]]


Displacement of green channel: dx: -1 dy: 1

Displacement of red channel: dx: -1 dy: 7

Displacement of green channel: dx: 2 dy: 5

Displacement of red channel: dx: 3 dy: 12

The homography matrix between green channel and blue channel is: [[ 1.03872637e+00 -2.07050395e-03 -2.02050710e+00]

[ 1.53911445e-02 1.01507069e+00 1.52900937e+00]

[ 1.02457946e-04 -4.29641678e-05 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 1.02931468e+00 4.68330955e-02 -1.53466078e+00]

[-5.26364542e-03 1.08156886e+00 6.95242499e+00]

[-7.57452681e-05 3.84860695e-04 1.00000000e+00]]

ORB: brightness changes too much for successful paring of keypoints


Displacement of green channel: dx: 23 dy: 48

Displacement of red channel: dx: 56 dy: 120

Displacement of green channel: dx: 23 dy: 48

Displacement of red channel: dx: 40 dy: 105

The homography matrix between green channel and blue channel is: [[-9.93432628e+00 4.22298267e+00 6.88269030e+03]

[-6.58954294e+00 -9.69919043e-01 9.84387154e+03]

[-3.41782958e-03 1.74502087e-03 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 9.55390107e-01 -1.61518854e+00 1.89109014e+03]

[ 9.85803236e-02 -5.88858329e-01 1.48573081e+03]

[ 3.28740558e-04 -6.75024675e-04 1.00000000e+00]]

ORB: brightness changes too much for successful paring of keypoints

Displacement of green channel: dx: 19 dy: 60

Displacement of red channel: dx: 17 dy: 124

Displacement of green channel: dx: 22 dy: 59

Displacement of red channel: dx: 20 dy: 122

The homography matrix between green channel and blue channel is: [[ 1.02881193e+00 -1.65544736e-02 9.52864635e+00]

[ 1.61166470e-02 1.01306459e+00 1.80323811e+01]

[ 5.72760340e-06 -1.70030304e-06 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 1.40584671e+00 1.98244823e-01 -7.12398998e+02]

[ 1.31074206e-01 1.43806870e+00 -4.98924678e+02]

[ 5.12325503e-05 7.07528358e-05 1.00000000e+00]]

ORB: brightness changes too much for successful paring of keypoints


Displacement of green channel: dx: 18 dy: 40

Displacement of red channel: dx: 24 dy: 88

Displacement of green channel: dx: 14 dy: 27

Displacement of red channel: dx: 38 dy: 102

The homography matrix between green channel and blue channel is: [[ 1.02318409e+00 -2.38083927e-02 3.12619409e+01]

[ 3.30422279e-02 9.86224727e-01 1.58672702e+01]

[ 1.14954119e-05 -8.73967485e-06 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 9.76104736e-01 8.77730525e-03 3.34979667e+01]

[-2.35066570e-02 1.01088573e+00 9.43646582e+01]

[-8.18478412e-06 3.88138789e-06 1.00000000e+00]]



Displacement of green channel: dx: 4 dy: 56

Displacement of red channel: dx: 8 dy: 113

Displacement of green channel: dx: 50 dy: 94

Displacement of red channel: dx: 114 dy: 210

The homography matrix between green channel and blue channel is: [[-7.46604820e+00 1.07574541e+01 3.89535512e+03]

[-5.58353224e+00 8.01452775e+00 2.97758386e+03]

[-2.02917166e-03 2.95073867e-03 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[-1.00238007e+00 -5.78833346e-02 3.64463750e+03]

[-5.13288144e-01 -3.95819684e-02 1.86045919e+03]

[-2.75029218e-04 -1.58653649e-05 1.00000000e+00]]

Edge: not enough edges for feature rich alignment



ORB: brightness changes too much for successful paring of keypoints


Displacement of green channel: dx: 26 dy: 93

Displacement of red channel: dx: 10 dy: 153

Displacement of green channel: dx: 50 dy: 126

Displacement of red channel: dx: 54 dy: 210

The homography matrix between green channel and blue channel is: [[1.00646912e+00 3.13310498e-04 9.83850768e-01]

[3.02587366e-03 1.00881504e+00 6.92424214e+01]

[4.86372597e-07 1.22350597e-06 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 7.85115822e-01 3.22795354e+00 -2.49198667e+03]

[ 6.08281121e-02 -4.25032613e-01 1.58076060e+03]

[-1.21373806e-04 3.54131575e-04 1.00000000e+00]]

ORB: brightness changes too much for successful paring of keypoints


Displacement of green channel: dx: 0 dy: -6

Displacement of red channel: dx: 1 dy: 9

Displacement of green channel: dx: 2 dy: -3

Displacement of red channel: dx: 2 dy: 3

The homography matrix between green channel and blue channel is: [[ 1.00051982e+00 1.19185681e-02 1.51606517e+00]

[ 2.14685821e-03 1.01275770e+00 -4.61407027e+00]

[-4.19944306e-05 8.22534520e-05 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 9.78744248e-01 -8.95373689e-02 1.62198494e+01]

[ 1.06080036e-02 9.04067919e-01 1.13194784e+01]

[ 9.88664159e-05 -3.16349818e-04 1.00000000e+00]]



Displacement of green channel: dx: 26 dy: 50

Displacement of red channel: dx: 36 dy: 108

Displacement of green channel: dx: 26 dy: 49

Displacement of red channel: dx: 37 dy: 107

The homography matrix between green channel and blue channel is: [[1.00901896e+00 5.84432446e-04 1.43405467e+01]

[2.98045186e-03 1.01134201e+00 3.17128501e+01]

[2.26292647e-07 1.69800844e-06 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[1.03293859e+00 7.17991702e-04 1.49134260e+01]

[2.10022620e-02 1.02293037e+00 7.16022298e+01]

[8.13240526e-06 3.34781068e-06 1.00000000e+00]]



Displacement of green channel: dx: 29 dy: 77

Displacement of red channel: dx: 37 dy: 175

Displacement of green channel: dx: 29 dy: 77

Displacement of red channel: dx: 37 dy: 175

The homography matrix between green channel and blue channel is: [[ 1.00473297e+00 2.25608954e-03 2.00064179e+01]

[ 3.03920154e-03 1.01055835e+00 5.82791833e+01]

[-3.27199432e-07 1.70354585e-06 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 9.92900749e-01 -8.22425069e-03 5.06703803e+01]

[ 1.79493000e-03 9.92173739e-01 1.74352192e+02]

[-1.70788149e-06 -1.49653093e-06 1.00000000e+00]]



Displacement of green channel: dx: 16 dy: 50

Displacement of red channel: dx: 12 dy: 108

Displacement of green channel: dx: 58 dy: 90

Displacement of red channel: dx: 58 dy: 154

The homography matrix between green channel and blue channel is: [[ 9.92841164e-01 -9.95208064e-03 2.80134101e+01]

[ 9.09099854e-04 9.83147658e-01 6.45128791e+01]

[-6.49017215e-07 -4.29052894e-06 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 9.82473507e-01 -1.18324919e-02 3.65102785e+01]

[-1.87593967e-03 9.69656368e-01 1.43117285e+02]

[-1.96575702e-06 -5.79982575e-06 1.00000000e+00]]



Displacement of green channel: dx: 2 dy: 3

Displacement of red channel: dx: 3 dy: 6

Displacement of green channel: dx: 3 dy: 3

Displacement of red channel: dx: 3 dy: 6

The homography matrix between green channel and blue channel is: [[ 9.74684229e-01 -9.19511256e-03 4.41149927e+00]

[-5.86565093e-03 9.61338337e-01 6.85918025e+00]

[-4.18886070e-05 -7.77127525e-05 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 1.00541285e+00 4.21400686e-03 2.42392866e+00]

[-5.67604933e-05 1.01568451e+00 5.26429835e+00]

[-1.07944254e-05 4.94024046e-05 1.00000000e+00]]



Displacement of green channel: dx: 6 dy: 42

Displacement of red channel: dx: 32 dy: 85

Displacement of green channel: dx: 6 dy: 42

Displacement of red channel: dx: 32 dy: 85

The homography matrix between green channel and blue channel is: [[ 1.00063821e+00 -6.62601976e-03 1.08503874e+01]

[ 2.38188989e-03 1.00283319e+00 2.86352313e+01]

[-8.97780977e-07 -5.06959367e-07 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 1.03611948e+00 1.06955943e-02 -1.52097616e+01]

[ 2.52171220e-02 1.02593136e+00 3.61732253e+01]

[ 7.81191595e-06 3.07579965e-06 1.00000000e+00]]



Displacement of green channel: dx: 0 dy: 52

Displacement of red channel: dx: 2 dy: 59

Displacement of green channel: dx: 0 dy: 53

Displacement of red channel: dx: 2 dy: 120

The homography matrix between green channel and blue channel is: [[ 1.00304922e+00 2.09752319e-04 -4.18920641e+00]

[ 1.85457818e-03 1.00616464e+00 4.19956304e+01]

[-8.31107037e-07 1.70368966e-06 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 1.00620563e+00 -2.33868101e-03 -1.34017987e+01]

[ 7.11685938e-03 1.00730274e+00 8.64916980e+01]

[ 1.26254972e-06 7.81850825e-07 1.00000000e+00]]




Self-chosen ones


Displacement of green channel: dx: 0 dy: 2

Displacement of red channel: dx: -1 dy: 3

Displacement of green channel: dx: 1 dy: 2

Displacement of red channel: dx: 2 dy: 6

The homography matrix between green channel and blue channel is: [[ 8.11764632e-01 3.80923818e-02 1.50974702e+01]

[-7.14044364e-02 9.45782701e-01 1.10049451e+01]

[-5.47661810e-04 1.63560916e-04 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 9.98149724e-01 -2.31477247e-03 2.11213211e+00]

[-1.68416261e-03 1.00216219e+00 5.79502048e+00]

[-2.75904945e-05 1.88189240e-05 1.00000000e+00]]



Displacement of green channel: dx: -1 dy: 0

Displacement of red channel: dx: -2 dy: 7

Displacement of green channel: dx: 0 dy: 3

Displacement of red channel: dx: -2 dy: 7

The homography matrix between green channel and blue channel is: [[ 9.88833930e-01 4.27175587e-02 -2.26174015e+00]

[-3.78526510e-02 1.08791771e+00 -2.39256437e+00]

[-1.99454145e-04 3.11337855e-04 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 9.86252202e-01 -8.36440991e-02 6.11839754e+00]

[ 2.40259300e-02 8.90537953e-01 1.25031991e+01]

[ 1.43113225e-04 -4.02355962e-04 1.00000000e+00]]



Displacement of green channel: dx: 1 dy: -1

Displacement of red channel: dx: 5 dy: 2

Displacement of green channel: dx: 1 dy: 3

Displacement of red channel: dx: 1 dy: 6

The homography matrix between green channel and blue channel is: [[ 1.01008813e+00 1.67454197e-02 -1.78658044e+00]

[-3.38395853e-03 1.04513304e+00 -2.50574389e+00]

[-2.99369811e-05 9.40384487e-05 1.00000000e+00]]

The homography matrix between red channel and blue channel is: [[ 1.14689450e+00 3.72387765e-02 -8.50346709e+00]

[ 3.26222513e-02 1.21387311e+00 -1.53515954e+01]

[ 1.92527900e-04 4.88413458e-04 1.00000000e+00]]