We have used Laplacian pyramids to blend images in a ealier project. Another way to approach this problem is using image gradient to blend images. We constrain the gradient between the source region and the target region such that the blending boundary is smooth. Solve for a least square problem we can recover the best gradient blend.
To test our implementation logic, a toy problem is solved first. We have a grayscale input image and want to construct a image that matches it. Using the gradient constraints on the image itself, the reconstructed image should have the same gradients with the original image. To set the correct intensity level, we set an additional constraint on a single pixel value, the top left pixel should match.
I turned the equations into matrix format and solved with sparse least square method in Scipy. The result successfully recovers the original image. Original on the left, recon on the right.
We now blend an image to another image. The constraints are similar to the toy problem, with the difference at the boundary. The gradient between the variable pixel and the target pixel, which is constant, is constrained with the gradient between the two pixel at the source image coordinates.
I write the equations into matrix format. The four equations for a pixel are added together. So we have H*W variables and H*W equations and in the matrix format we have a square matrix. This way the sparse linear system is easy to solve using the spsolve method in Scipy.
Here are some results:
The first two cases works pretty well. I consider the last one a failure case. The dolphin's color is weird; it turned pinkish. The reason behind this the blue water surrounding the dolphin in the source image. The blue color is very bright and saturated in contrast to the dark blue water in the target image. In order to keep the boundary smooth, the gradient error was speaded to the whole region and turned out to be pink for the dolphin pixels.
The above implementation only constrain the gradient to that of the source image. Sometimes, the target image have strong texture and thus strong gradient. To keep these texture smooth, we can modify the algorithm to constrain the gradients for each pixel to the strongest gradient in either the source or the target. With this modification, smooth source blend into a texture heavy target image looks good.
Below is a great demonstration:
We can see the mixed gradient method, shown on the right, works a lot better.
I implemented the algorithm in the paper 'A Neural Algorithm of Artistic Style'. The VGG19 network structure is down bellow, with maxpooling changed to avgpooling.
Same as the paper, I used the Conv4_2 layer for content loss, and Conv1_1, Conv2_1, Conv3_1, Conv4_1, and Conv5_1 layers for style loss. Trained 200 iterations with lr=0.3. The content weight is 5e-7.
Here are the results:
