CS 194 Proj5 Part A + B

Kehan Wang

Part A

Shoot the Pictures

📸 📸 📸

Recover Homographies

To recover the Homography transformation matrix, I take apart the p’ = Hp equation and rearranged it into Ax = b format, of which x is a vector of the 8 degrees of freedom. Then I was able to call np.linalg.lstsq to solve for the 8 unknowns.

Warp the Images

My warping pipeline is as the following:

  1. compute target shape of the warped image using H and forward homography
  2. generate a list of all target warping points
  3. inverse warp target pts to im pts
  4. filter all out-of-bound points in all the im points
  5. interpolate to find pixel values of in_im pts
  6. fill in target from the interpolation values

Example image warp result:

Image Rectification

By warping towards a rectangle, we are able to rectify a given image.

Example:

Blend images into mosaic

Given two images with corresponding points, I can create an empty canvas in the back that is big enough to fit both after blending, and put them in two canvases.
By using laplacian stacks from project 2, I was able to blend these two canvases together with a smooth transition. Here are some of my blending results:

Bells and Whistles:

With the help of our image mosaic blending and a few special “weapons” (a back massage mallet and a pan), I was able to recreate the fight between Kakashi and Obito, two characters in Naruto with 1 single actor - me.

Kakashi and Obito had hated and loved each other, and it was after this fight that all their complications dissolved into friendship. Just like them, I have liked myself and hated myself, and hopefully by morphing into them I can find a way to become my own friend.

Part B

Harris Interest Point Detector

Using corner_harris and corner_peaks from scikit-image, I was able to detect pixels with high probability of being corners using a sigma of 1.

Adaptive Non-Maximal Suppression

By first ranking all corners by their H values, then finding the maximum radius of suppression, we can list all corner in order of their radius of suppression. In this way, corners of most significance locally are extracted(I extracted 50 corners in this image) and we reduce the amount of work needed for feature matching.

Feature Descriptor extraction

For feature extraction, I first guassian blurred the picture to reduce aliasing, and then extracted a patch of 40 by 40 with the corner in the middle, but only keep an 8 by 8 image(a downsample by scale of 5). These 8 by 8 patches around my corners are then normalized to 0 mean, 1 std, and flattened to be my feature descriptors.

Examples of my feature descriptors:

Feature Matching

Given two images and their feature descriptors, I used SSD to calculate the similarity scores between two feature descriptors. For each feature in image 1, I calculate SSD for every feature in image 2, and find the most similar and second similar feature. Using Lowe’s threshold, if the SSD of the most similar over SSD of the second similar feature is smaller than 0.1, we’ll consider these two features a good matching.

Images with matching features:

RANSAC - Robust Homography method

Given that we have extracted these matching feature, there maybe one or two features that are not matched and it can completely mess up our homography. Therefore, we use RANSAC to randomly select subsets of matching features, and find the largest set of matching features that consistently fit onto one homography. I used 1000 iterations of RANSAC to calculate the best homography, and used it to warp my images.

Results

Manual vs Auto stitching

Manual stitching introduces error when selecting keypoints. It is also difficult to remember the point sequence in order to create matching features.

It is apparent that auto-stitching performs better and saves time and effort.

What have I learned?

I think the coolest thing I have learned in this project is how to read a paper, understand it, and re-implement it.