CS 194-26 Project 1: Coloring the Prokudin-Gorskii Collection

Nick Titterton 26608602, cs194-26-ago, 2018 November 10

Overview

If you have multiple images from the same XYZ location, but different rotations, you can combine the images using a projective transform.

Calculating Homographies

For a projective transform, given initial coordinates \(x\) and \(y\), we want to calculate matrix \(H\) such that $$ \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} wx' \\ wy' \\ w \end{bmatrix}$$ That gives us the equations $$ax + by + c = wx'$$ $$dx + ey + f = wy'$$ $$hx + gy + 1 = w$$ \(w\) is not a variable we want to solve for (since it is not an element of the homography), so we convert the above 3 equations into 2 using \(w = hx + gy + 1\) $$ax + by + c - hxx' - gyx' - x' = 0$$ $$dx + ey + f - hxy' - gyy' - y' = 0$$ We want the bottom right parameter of the matrix to equal 1, so we add a \(1000i = 1000\) constraint (1000 so that it is sufficiently weighted vs the other pixel-valued constraints), and then use least squares on a sufficient number of points (at least 4, I used 6 points each time).

Performing the Warp

Once we have a homography \(H\) converting points from image 1 to image 2, we can perform the warp. The pseudocode goes like this:

  1. Find bounding box by looking at transformed corners of image (after normalizing out depth \(w\))
  2. Compute interpolation of original image (sk.RectBivariateSpline)
  3. Compute \(H^{-1}\)
  4. For each pixel in the warped image, use \(H^{-1}\) and the interpolation to compute the value
This is very computationally expensive without vectorization. I had a very nicely vectorized Project 4, but haven't vectorized my code for this yet.

Creating Mosaics

Once you have a warped image, you can combine the warped image and the image it was warped to to create a mosaic! Here are some examples

There are some issues with the borders here: I used a simple take-the-pixel-by-pixel-max blend, which worked pretty well except for the chairs in the Soda 6th floor image and teh portrait in the pool table image.

What I learned

I learned a lot about homographies and transforms: esp. how to solve for a homography using least squares.