CS 194: Image Manipulation and Computational Photography, Fall 2020

Final Project

Rami Mostafa: cs194-26-abo

Neural Transfer

For this project, I reimplemented the paper to transfer art styles onto various photos. This was implemented using a modified vgg19 network with layers representing the Content and Style losses. Both layers compute the MSE loss between the original and generated images, although the Style loss is computed utilizing the gram matrix of the input and target images. The final loss is computed by weighting the content and style losses by alpha and beta respectively. Additionally, and LBGFS optimizer was used and every output was computed across 5000 iterations. I also downsampled the images to make processing faster (which may result in some lower quality images but with much better results).

There are some points in which I found better results by straying away from the details of the paper. For example, the paper found ratios of alpha/beta equal to 1e-3 and 1e-4 to provide the best results, however I found the best results using a ration of 1e-7. Additionally, removing the factor of 1/4 in the computation of the Style Loss helped as well.

Model Architecture

As stated before, I started with a vgg19 model, and added ContentLoss and StyleLoss layers at the points designated by the paper.

Neckarfront Outputs

Here are some outputs of applying various art on the provided Neckarfront image

Custom Outputs

Here are some image and art combinations I tried aside from neckarfront

And below I have a failure case. I believe this case fails because it is harder to apply a painting style to a specific subject as opposed to a landscape photo. More importantly, there aren't a whole lot of details in the art or the image of the dog. The straight lines making the branches in the art are only found in the outlines of the subject of the photo. The lack of variety of color in each input also makes applying neural transfer even more difficult.