Image quilting and neural style transfer!
Michael Jayasuriya
Image quilting
We want to take a texture and convincingly synthesize more texture using the image as a reference. We may try just taking random patches and stitching them together in this fashion.
Random patch texture synthesis
Unfortunately, this result isn't very convincing. Let's instead try something more intelligent. We can allow the patches to slightly overlap with one another. By doing this, instead of sampling random patches, we can sample from the patches that are within some tolerated total square distance from the patch we are overlapping with. We will call this algorithm quilt_simple
Overlapping patch texture synthesis
This one is more convincing! We can see that there are some seams still visible, but the result is pretty convincing. I'm not mad! However, we can make this even better by allowing a nonlinear seam, one that goes along the path of least difference. Let's take the first step of our algorithm, and illustrate this process.
Output image Best neighbor patch
Mask on neighbor patch Mask applied to neighbor patch
Neighbor patch put in output after min_cut
Now let's try it on some images.
Seam finding texture synthesis
Bells and whistles 1: I also wrote my own cut code, pictured below:
Min cut code
This uses dynamic programming to find a min cost path across the image. We start from the left side of the image, and at any arbitrary point along the path, we can find the step behind that path by taking the min cost of the neighbors of that node. That results in a 2d dynamic programming matrix that will keep track of cost and the path.
Now let's try doing texture transfer, which takes a target image and transfer texture onto that target image. Let's do it for barack obama.
Texture transfer example 1
I think what was gotten here was mostly Obama's suit. In addition, because we don't do iterative texture transfer, the transfer isn't very good. The only way that we can change this is by making the setting of the weight on correspondence between the output patch and the target image weighted much higher. For the below, I tried 2 iterations of the iterative texture transfer algorithm. This is an example of that.
Texture transfer example 2
Neural style transfer
Architecture
First we will take a look at the model that we are looking at, the VGG19 model. We use this model pretrained, and we extract a few of the convolutional layers for our use. What we discover in this project is that the earlier layers correspond to style, whereas the later layers in the convolutional part of the network correspond to content.
The point here is that we want to freeze the models weights, and not optimize them at all. Instead, we want to optimize an input image, which may start as random noise. We want to set up two losses for this image. This image needs to be similar in content to the content image, which can be done by taking the mean squared error with the content of the image defined by the features given by the convolutional layers. We also need to find a representation of style, so we take a matrix of feature correlations, which we can do using the Gram matrix of the feature maps.
I also use the LBFGS optimizer given by the analysis of this blog: https://blog.slavv.com/picking-an-optimizer-for-style-transfer-86e7b8cba84b
I use around 400 steps for the optimisation, which seems to be reasonable for the input image to stop changing and converge. Here we can try our neural style transfer algorithm out on a few inputs. First we need some images for style. Here is one image from Picasso which to me looks like some sort of robot, and a picture of a starry night by an artist named Vincent Van Gogh. Apparently he's pretty famous.
Results
But now I become starry night. Let's try it out on me.
Starry night style transfered onto a picture of me
The meat didn't work that well with this, but my face looks awesome. Let me zoom in, the people in the background look great.
Let's try the image that was used in the paper.
This reproduces the results featured in the paper in Figure 2, page 5, just trained for less iterations due to time constraints and I had to crop the image because the images must be the same size. Let's try this again with my another style image, a monet. Let's picture my dog as a monet.