Final Project: Texture Transfer
Image Quilting
Image quilting was implemented three ways in this project. First, patches were randomly selected and built into the output image. Second, patches were allowed to overlap and patches after the first one were selected to have small amount of error within a certain tolerance of the minimum error. Lastly, the algorithm was improved by finding a minimal error path seam when overlapping images.
Once image quilting was implemented a simple addition was calculated in order to implement texture transfer. While building the image during the image quilting algorithm, we modify the considered error by making a weighted average of the overlap error and a correspondence map. The correspondence map that I used was the luminance values of the target image.
Original Brick texture
Randomly quilted texture
Simple quilt algorithm texture
Quilt algorithm with seam finding.
Quilt algorithm with seam finding.
Quilt algorithm with seam finding.
Quilt algorithm with seam finding.
Quilt algorithm with seam finding.
One difficulty I encountered was the runtime of the algorithm. It was difficult to efficiently run the algorithm.
Texture transfer of feynman similar to paper
Neural Style Transfer
I implemented neural style artistic transfer by first using a pre-trained VGG16 model in pytorch. I used this as a feature extractor, so I used only the feature portion of the model. For specific convolutional layers of the model, I added a layer to calculate the content and style loss of inputted images. We create the new image by essentially training an image to minimize the loss between the content features of one image and the style features of another image. Content features loss were simply calculated by mean square error, while style features required the gram matrix to be calculated first. The gram matrix was then weighted by the size of the feature space and the MSE of the gram matrixes was calculated for the loss function. In accordance with the paper, Max_pool layers were replaced with average pool for better results. Furthermore, since I was having issues with vanishing gradients, I changed the ReLU layers to LeakyReLU layers.
The above 3 images (Starry Night, The Scream, and The Shipwreck) will be combined with the picture of Neckarfront.
The above 3 images are the artistically transferred images onto the image Neckarfront. A beta/alpha ratio of 1x10^7 was used and ran for 100 iterations of gradient descent using the LFGBS optimizer.
Monet's Impression, Sunrise photo combined with my own photo of the Shanghai Bund. A beta/alpha ratio of 1x10^7 was used and ran for 100 iterations of gradient descent using the LFGBS optimizer.
Original images
Dali's The Persistence of Memory artistically combined with a photo of my friend at Joshua Tree National Park.
Original Images
Failure Cases
Attempted texture transfer from toast to my friend's face.
Attempted artistic transfer from Rembrandt's lion resting to my cat.
I believe that these cases failed due to having realistic features. Artistic styles don't really transfer well, and in the case of toast the texture does not transfer well under these settings. Perhaps choosing different convolutional layers may have better results. The first 5 convolutional layers were also attempted, however, with results still lacking.
Bells and Whistles
For the image quilting project, I implemented cut.m by myself in python. I used dynamic programming to store the error values up to a pixel. Then, once I filled my dp array, I found the minimum value of the last column and traced back to find the minimum cost path. Lastly, I created a mask to have ones filled in each column under the path.
I also attempted to make a toast image of my friend.
