CS 194 Final Project Part 2

Mounted at /content/drive
/content/drive/MyDrive/neuralArt

A neural algorithm of artistic style reimplementation:

This algorithm can be summarized as taking a content image and a style iamge, passing them through a pretrained classification neural net (VGG 19), storing the intermediate results.

Then with the canvas image pass it through the same convnet, and calculate the difference between the intermediate layers as the loss function (for the style image we use the gram matrix difference because this has no notion of spatial dimension). To take the Gram matrix of matrix M do $M * M^t$. For the content image we do simple MSE loss.


I tyically take 1e6 as the weight for the style and 1 for the content, but my implementation allows us to vary this. Also adam is not the optimizer recommended by the authors, instead lbgfs.



Below are the results!

Visualize the Algorithm

Downloading: "https://download.pytorch.org/models/vgg19-dcbb9e9d.pth" to /root/.cache/torch/hub/checkpoints/vgg19-dcbb9e9d.pth





















Above is the implementation that provided me the best results. My architecture differs from the paper in that the paper decides to use the first conv layer after each max pooling whereas I simply used the first 4 conv layers for style and the last of them as the style constraint.















Neckarfront:





















 
 









































































 
 









































































 
 


































































Berkeley:





















 
 









































































 
 


































































One interesting feature is this algorithm is non deterministic. We can start with different random images and get different "paintings" out.






















 
 









































































 
 


































































Friends!





















 
 









































































 
 









































































 
 









































































 
 









































































 
 


































































tour Eiffel





















 
 









































































 
 









































































 
 









































































 
 


































































Failure Case





















 
 


































































This is quite the failure case. It seems the content image has very little detail and style to actually transfer to as it's just the beach and sand. For this algorithm to produce appealing results we must carefully choose a content image that can exude the style well. There is quite a bit of noise above too for whatever reason.















Reflection














This project was very cool, easily my favorite one of the course! It's so cool how neural nets can in a sense be used to produce artwork. Maybe I can open up an art gallery!








    

  



 











 
 Gradient Domain Fusion 













Toy Problem
Original Image:


























































Recreation of image:











































Part 2



















Penguins


























































Naive solution above.
































































































































































Failure Case













































































This is an interesting failure case, we notice that the background of my snow versus the snow the grizzly is in is quite different. This is around sunrise and I am in the shade of a big mountain whereas this grizzly is shot in a high exposure setting. The algorithm cannot fully make the grizzlys surroundings white as there are still some artifacts but they are very subtle.















Bells and Whistles
Mixed Gradients








































































Mixed gradients is interesting as it increases the visibility of the transition region while maintaining the gradients better and thus the colors of the original source image. The octopus looks much lighter here than in the traditional poisson blending process which is how it looks in the target image.









































































Mixed gradients seamlessly overlays the word smile above the waterfalls. Very cool! Below is the traditional method's output for comparison. Note it produces artifacts.