Neural Style Transfer and Image Quilting
Allan Yu
Neural Style Transfer
We start with a content image and a style image and through a neural network, output an image that maintains the content of the content image but with the style of the style image.
Network
The follow image is my model. Just like the paper, I use a pretrained VGG19, adding content loss layers after the first five convolutional layers and a style loss layer after the fourth content loss layer. I also replace maxpools with avgpools; I found that this resulted in better output images, just like the paper. As the input image, I start with the content image (so I start with a content loss of 0). I use the LBFGS optimizer, which I found to work better than Adam. In terms of hyperparameters, I found that in the general case, weighing the style image by 100000000 times more than the content image worked well. (This may be a bit biased to personal preference though!) I also normalize the images before processing.
Results
Here are the results of my network ran on three styles for Neckarfront. Compared to the paper, my images are a bit lighter in general. I definitely weigh content higher (this can be clearly seen in the starry night blend: my output has clear buildings in the center of the output, while the paper's does not). If I were to weigh style more, then my results would look more similar to the paper's results.
Neckarfront
Styles
My Results
Paper Results (For Comparison)
More Results
Here are more results on other content and style images. The last example is a failure case. I believe that the failure occurs for a number of reasons. First, the style image is not well defined: the right side of the image is a bit too plain. The content image is a bit plain as well: not nearly as noisy as Neckarfront. Furthermore, the horizontal trees in the style image clash with the vertical towers of the content image. I believe these factors caused the model to work less effectively.
Content
Style
Result
Content
Style
Result
Content
Style
Result
Image Quilting
Here, I randomly sample patches from a texture image to fill a size. Here is an example.
Original Image
Sampled