CS 294 026 Project 4 IMAGE WARPING and MOSAICING

Author: Tiancheng Sun

Shoot the Pictures

Since I don't have a digital camera and it is hard to maintain the same point of view on cell phone, I only shot two pictures for image rectification and two image for mosaic. The rest of the images are collected from the internet. Here are the images shot from my cell phone:

dx dy
Below are the images I collected from internet (credits to Lisa Chan):
dx dy
dx dy

Recover Homographies

Here comes the most complex part of this project. The target is reshape the original pH = p' equation into a form that the 3 by 3 matrix H becomes a 8 by 1 vector h so that we could use the least square equatoin to solve it; So first, here is the original equation:

dx
By expand it, we will got ax + by + c = wx' || dx + ey + f = wy' || gx + hy + i = w
Since all the quations shares a w, we could combine the three equations above into two equation (we can set the scale factor i as 1):
ax + by + c - gxx' - hyx' = x'
dx + ey + f - gxy' - hyy' = y'
And it could be transfer into a matrix when we have multiple points:
dx
The h11 ~ h32 in the above pic represent a ~ h accordingly
And since the equation above exist in the forme of Ah = b, we could construct A and b then let the lstsq funciton solve the correct answer for us. ## Warp the Images Here are the two example of the rectified img as required:
dx dx
rectify target is that sky blue book
dx dx
rectify target is the Children of the Sea book

Blend the images into a mosaic

Although I only demostrated two image blending here, the function itself could blend any number of imgs together, all the user need to do is take the output of previous round and treat it as a new image and select control point between it and the next image.

The first mosaic:

dx dx
dx

The second mosaic:

dx dx
dx

Tell us what you've learned

There are mainly two parts. First of all, I learned how to convert the pH = p' homography equation into the form that could be solved by the least square equation. I believe the method of re-organizing the structure of that linear algebra equation would be useful in the future. The second part I learned is how to develop a big function efficiently, during the implementation of warpImage() function, it come to my notice that write pseudo code first would be very helpful.

Part B, [Auto]

Detecting corner features in an image

In this part, I utilized the harris corner detection algorithm to find out all possible "corners" in the incoming images, to avoid too many noisy corners get reported, I set the min_distance to 5 to surpress the non-local max values, below are the results I got:

dx dx
It might looks a little bit not comfortable if you have Trypophobia, but if you inspect these imgs closely you shall notice that the corners are identified fairly nicely.

Implement Adaptive Non-Maximal Suppression

Just as professor mentationed in the lecture, this part really cost me sometime to fully understand... Focus on the "practical" part mentioned in the paper, what we have to do is find some number of pts that has the best suppersion radius. The pratical way in the paper says for each interest pts xi in the image, we can compute its "suppersion radius" and sort it to get the best pts. The problem is how to define the suppersion radius -> The so called radius is defined by a 'hull', and the hull is make up by all the pts xj that make xi not qualify as a local suppressor. After find all these xj, the minum distance of (xj - xi) is the supperison radius of pt xi.
In my implementation, I used c_robust = 0.9 as the paper suggested. Below is the reuslt I obtained when I suppressed the total points amount to 500:

dx dx

Extracting a Feature Descriptor for each feature point

After finishs find out all the possible corner points, the next step is to find a way to "discribe" these points. As the paper suggested, I captured a 40 40 img block and downsampled it to a 8 8 patch, here are the two results (I just randomly choose two, they are not paired):

dx dx

Matching these feature descriptors between two images

Now, we got the corners and coresponding deescriptors, it's times to pair them together base on their similarity between each other. I applied the ssd function to compute the similarity score and checked the ratio of the best match and second best match, the point pair will be consider valid only if (best match / second best match) < 0.3, below are the outcome paired points for this step:

dx dx
Look them closely, you shall notice that the relationship found by the algorithm is very rational

Use a robust method (RANSAC) to compute a homography

To rule out outliers and compute a stable homography, I used the random 4 point selection RANSAC algorithm we discussed on class. Under 10000 loops, the best homography could include 30 pts as inliers, below is the result after RANSAC outlier rejection:

dx dx

Proceed as in the first part to produce a mosaic

Showtime!

Mosaics 1:

dx dx
the img on the left comes from part A, the pts is hand selected, and as we can see, due to "human error", the grass is little bit blurry. But the grass in the auto generated pic (on the right) is quite clear

Mosaics 2:

dx dx
left image come from part A, the right img is auto generated

Mosaics 3:

dx
Auto generated, I didn't get enough time in part A to manually define the points for it

Also, as a proof of that the algorithm could take any number of input, here is a pano generated from three pics:

dx

What have you learned?

I believe the most important thing I learned from this thing is how to implemented a poorly documented algorithm. The core technique that helps me the most is to "make sense" of the algorithm, that is, not only looking at what the author says they deisgned the algorithm, we also have to understand why that step will work, what they are trying to do this first etc.. This would be very helpful since it makes sure that we won't run into a wrong path.