Image Warping and Mosaicing

Steven Cao / cs194-26-adx / Project 5 (Part 1)

In this project, we'll see how to stitch images together into mosaics. This first part involves image stitching using manually picked point correspondences.


Methods


Image stitching involves three steps:

  1. Take the pictures. The pictures should come from the same position but with the view rotated. Therefore, you should try to keep the camera lense fixed and only rotate it.
  2. Pick points of correspondence. While 4 points are enough in theory, I collected 8 points to reduce noise.
  3. Using these point correspondences, warp one image to the other such that the pairs of points lie on top of one another. The warp, which is a \(3 \times 3\) matrix with \(8\) degrees of freedom, can be computed with least squares as follows: $$ \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} w'x' \\ w'y' \\ w' \end{bmatrix} $$ Given this equation, we can first write \(w'\) in terms of the other parameters. Then, we can solve for \(a\) through \(h\) by rearranging the equation as follows: $$ \begin{bmatrix} x & y & 1 & 0 & 0 & 0 & -xx' & -yx' \\ 0 & 0 & 0 &x & y & 1 & -xy' & -yy' \\ \end{bmatrix} \begin{bmatrix} a \\b\\c\\d\\e\\f\\g\\h \end{bmatrix} = \begin{bmatrix} x'\\y' \end{bmatrix} $$
  4. Finally, once the first image is warped to the geometry of the second image, we can blend them together.


Results: Rectification


First, let's use warping for a different application, rectification. Rectification means transforming the image so that one of the planes is directly parallel with the image plane. We can perform rectification by (1) picking four points forming a square on the plane of interest, and then (2) warping these points into an actual square, or the points \((0,0), (a,0), (0,a), (a,a)\). Below are some examples. In the first case, we rectify to the right face of the building, and in the second case, we rectify to the face of the building.


Results: Mosaicing


Next, let's look at examples of mosaicing. I took these pictures in my home. Unfortunately I couldn't get any nice pictures of scenery due to the quarantine. In each example, we show the two original images, the mosaic with simple average blending, and the mosaic with linear blending.


Conclusion


I enjoyed making mosaics and I am glad that I now have this technique in my toolkit. It was interesting to learn that you actually shouldn't move the camera while taking the panoramic photo. It seems that since most people take pictures of scenery, which is faraway, the parallax effect is less noticable. But during the project, I realized that it matters a lot if you take pictures of nearby things!