Project 1 – Gary Yang

1. Intro

Goal: take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible.

Method: I divide the raw input glass plate images from top-to-down into B,G,R channels, align them and stack them in RGB order to generate the appropriate output. For the alignment method, I search exhaustively over a window of possible displacements with [-15,15] on the x-axis and y-axis respectively. The metric I used is the L2 norm, whichever displacement had the smallest L2 norm is chosen as the result. For the image pyramid, I first resized the image to begin the first coarse alignment at either when the height or width of the image is smaller than 512 or the pyramid is 4 layers deep. For each layer I decrease the size by 2x2=4. After getting the initial alignment, I fine tune it with the center 100x100 region of the bigger images. For Bells and Whistles, I approached the goal in two different ways: 1)using edges[1] as the input features in lieu of the raw pixel values; 2)using the ORB detector[2] to find matching keypoints and then find the homography between the image that needs to be aligned and the target one. It is worth noting that the ORB detector is a double-edged sword: 1)good: it does not need image pyramids to process big images and 2)bad: if the brightness between channels change too much, it would have a high failure rate.

2. Results

Here is the table summarizing the outputs:

 Not aligned Aligned with raw pixel values Aligned with edge features (Bells & Whistles) Aligned with ORB keypoints (Bells & Whistles) Reason for failure Displacement of green channel: dx: 2 dy: 32 Displacement of red channel: dx: 5 dy: 98 Displacement of green channel: dx: 1 dy: 33 Displacement of red channel: dx: 5 dy: 97 The homography matrix between green channel and blue channel is: [[ 1.01243800e+00 2.88669284e-04 -1.34366645e+01] [ 4.46747793e-03 1.00893696e+00 1.66294698e+01] [ 1.76816879e-06 5.81088759e-07 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 9.83233785e-01 -6.18014345e-03 3.20364896e+01] [-2.20593997e-03 9.86229016e-01 1.10286553e+02] [-2.91438583e-06 -1.60806960e-06 1.00000000e+00]] Displacement of green channel: dx: -1 dy: 1 Displacement of red channel: dx: -1 dy: 7 Displacement of green channel: dx: 2 dy: 5 Displacement of red channel: dx: 3 dy: 12 The homography matrix between green channel and blue channel is: [[ 1.03872637e+00 -2.07050395e-03 -2.02050710e+00] [ 1.53911445e-02 1.01507069e+00 1.52900937e+00] [ 1.02457946e-04 -4.29641678e-05 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 1.02931468e+00 4.68330955e-02 -1.53466078e+00] [-5.26364542e-03 1.08156886e+00 6.95242499e+00] [-7.57452681e-05 3.84860695e-04 1.00000000e+00]] ORB: brightness changes too much for successful paring of keypoints Displacement of green channel: dx: 23 dy: 48 Displacement of red channel: dx: 56 dy: 120 Displacement of green channel: dx: 23 dy: 48 Displacement of red channel: dx: 40 dy: 105 The homography matrix between green channel and blue channel is: [[-9.93432628e+00 4.22298267e+00 6.88269030e+03] [-6.58954294e+00 -9.69919043e-01 9.84387154e+03] [-3.41782958e-03 1.74502087e-03 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 9.55390107e-01 -1.61518854e+00 1.89109014e+03] [ 9.85803236e-02 -5.88858329e-01 1.48573081e+03] [ 3.28740558e-04 -6.75024675e-04 1.00000000e+00]] ORB: brightness changes too much for successful paring of keypoints Displacement of green channel: dx: 19 dy: 60 Displacement of red channel: dx: 17 dy: 124 Displacement of green channel: dx: 22 dy: 59 Displacement of red channel: dx: 20 dy: 122 The homography matrix between green channel and blue channel is: [[ 1.02881193e+00 -1.65544736e-02 9.52864635e+00] [ 1.61166470e-02 1.01306459e+00 1.80323811e+01] [ 5.72760340e-06 -1.70030304e-06 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 1.40584671e+00 1.98244823e-01 -7.12398998e+02] [ 1.31074206e-01 1.43806870e+00 -4.98924678e+02] [ 5.12325503e-05 7.07528358e-05 1.00000000e+00]] ORB: brightness changes too much for successful paring of keypoints Displacement of green channel: dx: 18 dy: 40 Displacement of red channel: dx: 24 dy: 88 Displacement of green channel: dx: 14 dy: 27 Displacement of red channel: dx: 38 dy: 102 The homography matrix between green channel and blue channel is: [[ 1.02318409e+00 -2.38083927e-02 3.12619409e+01] [ 3.30422279e-02 9.86224727e-01 1.58672702e+01] [ 1.14954119e-05 -8.73967485e-06 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 9.76104736e-01 8.77730525e-03 3.34979667e+01] [-2.35066570e-02 1.01088573e+00 9.43646582e+01] [-8.18478412e-06 3.88138789e-06 1.00000000e+00]] Displacement of green channel: dx: 4 dy: 56 Displacement of red channel: dx: 8 dy: 113 Displacement of green channel: dx: 50 dy: 94 Displacement of red channel: dx: 114 dy: 210 The homography matrix between green channel and blue channel is: [[-7.46604820e+00 1.07574541e+01 3.89535512e+03] [-5.58353224e+00 8.01452775e+00 2.97758386e+03] [-2.02917166e-03 2.95073867e-03 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[-1.00238007e+00 -5.78833346e-02 3.64463750e+03] [-5.13288144e-01 -3.95819684e-02 1.86045919e+03] [-2.75029218e-04 -1.58653649e-05 1.00000000e+00]] Edge: not enough edges for feature rich alignment ORB: brightness changes too much for successful paring of keypoints Displacement of green channel: dx: 26 dy: 93 Displacement of red channel: dx: 10 dy: 153 Displacement of green channel: dx: 50 dy: 126 Displacement of red channel: dx: 54 dy: 210 The homography matrix between green channel and blue channel is: [[1.00646912e+00 3.13310498e-04 9.83850768e-01] [3.02587366e-03 1.00881504e+00 6.92424214e+01] [4.86372597e-07 1.22350597e-06 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 7.85115822e-01 3.22795354e+00 -2.49198667e+03] [ 6.08281121e-02 -4.25032613e-01 1.58076060e+03] [-1.21373806e-04 3.54131575e-04 1.00000000e+00]] ORB: brightness changes too much for successful paring of keypoints Displacement of green channel: dx: 0 dy: -6 Displacement of red channel: dx: 1 dy: 9 Displacement of green channel: dx: 2 dy: -3 Displacement of red channel: dx: 2 dy: 3 The homography matrix between green channel and blue channel is: [[ 1.00051982e+00 1.19185681e-02 1.51606517e+00] [ 2.14685821e-03 1.01275770e+00 -4.61407027e+00] [-4.19944306e-05 8.22534520e-05 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 9.78744248e-01 -8.95373689e-02 1.62198494e+01] [ 1.06080036e-02 9.04067919e-01 1.13194784e+01] [ 9.88664159e-05 -3.16349818e-04 1.00000000e+00]] Displacement of green channel: dx: 26 dy: 50 Displacement of red channel: dx: 36 dy: 108 Displacement of green channel: dx: 26 dy: 49 Displacement of red channel: dx: 37 dy: 107 The homography matrix between green channel and blue channel is: [[1.00901896e+00 5.84432446e-04 1.43405467e+01] [2.98045186e-03 1.01134201e+00 3.17128501e+01] [2.26292647e-07 1.69800844e-06 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[1.03293859e+00 7.17991702e-04 1.49134260e+01] [2.10022620e-02 1.02293037e+00 7.16022298e+01] [8.13240526e-06 3.34781068e-06 1.00000000e+00]] Displacement of green channel: dx: 29 dy: 77 Displacement of red channel: dx: 37 dy: 175 Displacement of green channel: dx: 29 dy: 77 Displacement of red channel: dx: 37 dy: 175 The homography matrix between green channel and blue channel is: [[ 1.00473297e+00 2.25608954e-03 2.00064179e+01] [ 3.03920154e-03 1.01055835e+00 5.82791833e+01] [-3.27199432e-07 1.70354585e-06 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 9.92900749e-01 -8.22425069e-03 5.06703803e+01] [ 1.79493000e-03 9.92173739e-01 1.74352192e+02] [-1.70788149e-06 -1.49653093e-06 1.00000000e+00]] Displacement of green channel: dx: 16 dy: 50 Displacement of red channel: dx: 12 dy: 108 Displacement of green channel: dx: 58 dy: 90 Displacement of red channel: dx: 58 dy: 154 The homography matrix between green channel and blue channel is: [[ 9.92841164e-01 -9.95208064e-03 2.80134101e+01] [ 9.09099854e-04 9.83147658e-01 6.45128791e+01] [-6.49017215e-07 -4.29052894e-06 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 9.82473507e-01 -1.18324919e-02 3.65102785e+01] [-1.87593967e-03 9.69656368e-01 1.43117285e+02] [-1.96575702e-06 -5.79982575e-06 1.00000000e+00]] Displacement of green channel: dx: 2 dy: 3 Displacement of red channel: dx: 3 dy: 6 Displacement of green channel: dx: 3 dy: 3 Displacement of red channel: dx: 3 dy: 6 The homography matrix between green channel and blue channel is: [[ 9.74684229e-01 -9.19511256e-03 4.41149927e+00] [-5.86565093e-03 9.61338337e-01 6.85918025e+00] [-4.18886070e-05 -7.77127525e-05 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 1.00541285e+00 4.21400686e-03 2.42392866e+00] [-5.67604933e-05 1.01568451e+00 5.26429835e+00] [-1.07944254e-05 4.94024046e-05 1.00000000e+00]] Displacement of green channel: dx: 6 dy: 42 Displacement of red channel: dx: 32 dy: 85 Displacement of green channel: dx: 6 dy: 42 Displacement of red channel: dx: 32 dy: 85 The homography matrix between green channel and blue channel is: [[ 1.00063821e+00 -6.62601976e-03 1.08503874e+01] [ 2.38188989e-03 1.00283319e+00 2.86352313e+01] [-8.97780977e-07 -5.06959367e-07 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 1.03611948e+00 1.06955943e-02 -1.52097616e+01] [ 2.52171220e-02 1.02593136e+00 3.61732253e+01] [ 7.81191595e-06 3.07579965e-06 1.00000000e+00]] Displacement of green channel: dx: 0 dy: 52 Displacement of red channel: dx: 2 dy: 59 Displacement of green channel: dx: 0 dy: 53 Displacement of red channel: dx: 2 dy: 120 The homography matrix between green channel and blue channel is: [[ 1.00304922e+00 2.09752319e-04 -4.18920641e+00] [ 1.85457818e-03 1.00616464e+00 4.19956304e+01] [-8.31107037e-07 1.70368966e-06 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 1.00620563e+00 -2.33868101e-03 -1.34017987e+01] [ 7.11685938e-03 1.00730274e+00 8.64916980e+01] [ 1.26254972e-06 7.81850825e-07 1.00000000e+00]] Self-chosen ones Displacement of green channel: dx: 0 dy: 2 Displacement of red channel: dx: -1 dy: 3 Displacement of green channel: dx: 1 dy: 2 Displacement of red channel: dx: 2 dy: 6 The homography matrix between green channel and blue channel is: [[ 8.11764632e-01 3.80923818e-02 1.50974702e+01] [-7.14044364e-02 9.45782701e-01 1.10049451e+01] [-5.47661810e-04 1.63560916e-04 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 9.98149724e-01 -2.31477247e-03 2.11213211e+00] [-1.68416261e-03 1.00216219e+00 5.79502048e+00] [-2.75904945e-05 1.88189240e-05 1.00000000e+00]] Displacement of green channel: dx: -1 dy: 0 Displacement of red channel: dx: -2 dy: 7 Displacement of green channel: dx: 0 dy: 3 Displacement of red channel: dx: -2 dy: 7 The homography matrix between green channel and blue channel is: [[ 9.88833930e-01 4.27175587e-02 -2.26174015e+00] [-3.78526510e-02 1.08791771e+00 -2.39256437e+00] [-1.99454145e-04 3.11337855e-04 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 9.86252202e-01 -8.36440991e-02 6.11839754e+00] [ 2.40259300e-02 8.90537953e-01 1.25031991e+01] [ 1.43113225e-04 -4.02355962e-04 1.00000000e+00]] Displacement of green channel: dx: 1 dy: -1 Displacement of red channel: dx: 5 dy: 2 Displacement of green channel: dx: 1 dy: 3 Displacement of red channel: dx: 1 dy: 6 The homography matrix between green channel and blue channel is: [[ 1.01008813e+00 1.67454197e-02 -1.78658044e+00] [-3.38395853e-03 1.04513304e+00 -2.50574389e+00] [-2.99369811e-05 9.40384487e-05 1.00000000e+00]] The homography matrix between red channel and blue channel is: [[ 1.14689450e+00 3.72387765e-02 -8.50346709e+00] [ 3.26222513e-02 1.21387311e+00 -1.53515954e+01] [ 1.92527900e-04 4.88413458e-04 1.00000000e+00]]