Preface

Wow, this entire project was a disaster for me. I had too many non-academic external forces happening during the past month for me to really be able to focus on this project, nor did I even really have the patience to reimplement this problem from scratch given it's a solved problem in many programs and python libraries. I was seriously shocked at how challenging this project was for me, considering I have quite a bit of experience in imaging, graphics, computer vision, and computational photography related courses (7 total, 5 in the CS department, 4 at the graduate level); I just was really not vibing with this project. I'm submitting what I have in the hopes of some type of partial credit at this point so I can shift my focus to trying to get some credit for my final project, which I think my skills will fit much nicer into. If you're wondering why this is so late, please check with Professor Efros for relevant extension requests in my email correspondence with him (I have had a variety of extenuating circumstances and it has obviously affected my ability to complete this project). I still have not gotten a clear answer to whether comments in my code are considered for the rubric. I have copied the most salient info regarding my approach here, but if you are wondering if I actually know how to do something or want more details on my approach, I believe my code is very well documented and commented and you will likely find any answer to any questions about my approach there.

Part A - Image Warping and Mosaicing

A1) Image Collection

The pictures I chose to use were a collection of images that I shot from an art installation I did last semester. I had trouble panorama stiching them automatically, so I thought they would be perfect for this project.

A2) Recovering Homographies

I tried two approaches to try to make this work. Neither worked as well as a ground-truth implementation such as that given in OpenCV. You can see a comparison of the results below. Strangely, even the ground truth of OpenCV doesn't seem to match up with the actual ground truth points, but it works in the later image warping.

Basic Linear Systems

Below is my function description code that delineates the mathematical approach of my first attempt at computing the homography. I then solved the final system using least squares.


def computeH_1(im1_pts, im2_pts):
    """
    Computes Homography given the input corresponding points
    :param im1_pts: Points to compute homography from
    :param im2_pts: Points to compute homography to
    :return: Homography Matrix

    Using input points p_i=(x_i, y_i) from im1 and p_i'=(x_i', y_i') from im2 the homography is:
    p'.T = H @ p.T
    |x_1' x_2' x_3'     |   |h00 h01 h02|   |x_1 x_2 x_3     |
    |y_1' y_2' y_3' ... | = |h10 h11 h12| @ |y_1 y_2 y_3 ... |
    |1    1    1        |   |h20 h21 h22|   |1   1   1       |

    We can reformulate this as b = A @ x to recover H with least squares as follows:
    |-x_1'|    |x_1 y_1 1   0   0   0   0   0   0   |
    |-y_1'|    |0   0   0   x_1 y_1 1   0   0   0   |
    |1    |    |0   0   0   0   0   0   x_1 y_1 1   |
    |-x_2'|    |x_2 y_2 1   0   0   0   0   0   0   |
    |-y_2'| =  |0   0   0   x_2 y_2 1   0   0   0   |  @ |h00 h01 h02 h10 h11 h12 h20 h21 h22|.T
    |1    |    |0   0   0   0   0   0   x_2 y_2 1   |
    |-x_3'|    |x_3 y_3 1   0   0   0   0   0   0   |
    |-y_3'|    |0   0   0   x_3 y_3 1   0   0   0   |
    |1    |    |0   0   0   0   0   0   x_3 y_3 1   |
    | ... |    |                ....                |
    """

SVD Composition

Below is my function description code that delineates the mathematical approach of my second attempt at computing the homography, based on a mathematical result I found online.


def computeH_2(im1_pts, im2_pts):
    """
    Computes Homography given the input corresponding points
    :param im1_pts: Points to compute homography from
    :param im2_pts: Points to compute homography to
    :return: Homography Matrix

    After this problem gaslighting me to hell, I tried a different approach given on StackOverflow here:
    https://math.stackexchange.com/questions/494238/how-to-compute-homography-matrix-h-from-corresponding-points-2d-2d-planar-homog

    Basically, we want to solve this, but use SVD to avoid the trivial answer:

    | 0 |    |x_1 y_1 1   0   0   0   -x_1*x_1'   -y_1*x_1'  -x_1'|
    | 0 |    |0   0   0   x_1 y_1 1   -x_1*y_1'   -y_1*y_1'  -y_1'|
    | 0 |    |x_2 y_2 1   0   0   0   -x_2*x_2'   -y_2*x_2'  -x_2'|
    | 0 | =  |0   0   0   x_2 y_2 1   -x_2*y_2'   -y_2*y_2'  -y_2'|  @ |h00 h01 h02 h10 h11 h12 h20 h21 h22|.T
    | 0 |    |x_3 y_3 1   0   0   0   -x_3*x_3'   -y_3*x_3'  -x_3'|
    | 0 |    |0   0   0   x_3 y_3 1   -x_3*y_3'   -y_3*y_3'  -y_3'|
    |...|    |                          ....                     |

    """

A3) Image Warping

In order to warp, I tried to invert our homography to transform every pixel coordinate in the transformed image to a sample in the original image. To do this, I tried to estimate the dimensions of the transformed image by transforming the bounding box. I then made a grid of the coordinates in this estimated output image, and sampled from the initial image accordingly. However, something didn't work with this approach, and I don't know why. You can see my warp (left) and the output of a ground truth implementation (OpenCV, right) below.

A4) Rectification

As you might guess, this part of the project wasn't looking too promising.

A5) Blending Mosaic

I tried to complete this portion of the project as well, but wasn't even able to even when using the cv2 ground truth implementations, because I was losing my mind with my transformed images going out of image bounds. No matter what I did, I couldn't seem to account for these bounds by offsetting. You can see my attempt at a final mosaic using my approach (left) and with the help of OpenCV below (right). Welp, ¯\_(ツ)_/¯. No hope for part B...

A6) Conclusion

Through this project, I learned that if you find it frustrating to reimplement functions already implemented in other libraries, maybe find a different solution (in this case, take the L on this project) before starting down a fruitless rabbithole.