I used the following matrix to calculate the Homographies between two pictures. Since there may not be a perfect solution, so I used the least square solution
I used the night street as an example to demonstrate the rectified images
I used another street corner
Here are three sets of merge
After wraping one picture onto another, I used another alpha channel to do the weighted blending so the transition are as smooth as possible, which works best if the alignment is on the same horizontal level or vertical level. Otherwise, the pictures will have some sharp edge on some degree
The image rectification is pretty interesting and useful. It's hard to believe that we can the different sides of the building just from one picture