Image quilting was implemented three ways in this project. First, patches were randomly selected and built into the output image. Second, patches were allowed to overlap and patches after the first one were selected to have small amount of error within a certain tolerance of the minimum error. Lastly, the algorithm was improved by finding a minimal error path seam when overlapping images.
Once image quilting was implemented a simple addition was calculated in order to implement texture transfer. While building the image during the image quilting algorithm, we modify the considered error by making a weighted average of the overlap error and a correspondence map. The correspondence map that I used was the luminance values of the target image.
Original Brick texture Randomly quilted texture Simple quilt algorithm texture Quilt algorithm with seam finding. Quilt algorithm with seam finding. Quilt algorithm with seam finding. Quilt algorithm with seam finding. Quilt algorithm with seam finding. One difficulty I encountered was the runtime of the algorithm. It was difficult to efficiently run the algorithm. Texture transfer of feynman similar to paperI implemented neural style artistic transfer by first using a pre-trained VGG16 model in pytorch. I used this as a feature extractor, so I used only the feature portion of the model. For specific convolutional layers of the model, I added a layer to calculate the content and style loss of inputted images. We create the new image by essentially training an image to minimize the loss between the content features of one image and the style features of another image. Content features loss were simply calculated by mean square error, while style features required the gram matrix to be calculated first. The gram matrix was then weighted by the size of the feature space and the MSE of the gram matrixes was calculated for the loss function. In accordance with the paper, Max_pool layers were replaced with average pool for better results. Furthermore, since I was having issues with vanishing gradients, I changed the ReLU layers to LeakyReLU layers.
The above 3 images (Starry Night, The Scream, and The Shipwreck) will be combined with the picture of Neckarfront. The above 3 images are the artistically transferred images onto the image Neckarfront. A beta/alpha ratio of 1x10^7 was used and ran for 100 iterations of gradient descent using the LFGBS optimizer. Monet's Impression, Sunrise photo combined with my own photo of the Shanghai Bund. A beta/alpha ratio of 1x10^7 was used and ran for 100 iterations of gradient descent using the LFGBS optimizer. Original images Dali's The Persistence of Memory artistically combined with a photo of my friend at Joshua Tree National Park. Original Images