For shooting the pictures to use for panorams, I made sure to hold my hand very steady and only rotate the camera about its sensor. I also made sure to find areas that had static scenary and easily identifiable corner (such as doors, windows, etc) to make marking correspondences easier.
To recover the homographies, I used a least squares solution to solve for an H matrix that would map im1 points to im2 points and reduce the MSE of im2 points against im2-pts-recovered-from-im1-points.
For these operations I used inverse mapping to determine what to do. More specifically I determined the desired size of the final image, be it a rectification or mosaic, and then use the inverse of the homography matrix to determine what point in would correspond to from im1 and the use an interpolation to determine the proper color. Then I overlayed im2 using a similar strategy, but without the homography inverse bit (since the inverse is trivial). Then I used a left to right alpha blending to get rid of image seams, though some seams are still visible.
