The project implements the NN style transfer algorithm from this paper by Gatys, Ecker, and Bethge. In the paper, the authors discovered that in neural nets, higher layers usually capture the high-level content while lower layers capture the details in pictures. Therefore, we can combine the content of one image with the style of another image using different layers of neural networks.
Following the methods in the paper, I used VGG19 and used conv1_1, conv2_1, conv3_1, conv4_1, and conv5_1 for style representation, and they used conv4_2 for content representation. We try to minimize the weighted sum of content and style loss.
I used some of my photos from Yosemite and the Golden Gate Bridge.
Here are some results!! They're really beautiful and I had so much fun doing this project. Imagine these artists were alive today and got to visit SF & Yosemite with me :)
Van Gogh
Picasso
Edvard Munch
Joseph Mallord William Turner
The project implements the image quilting and texture transfer algorithm from this paper by Efros and Freeman. For texture synthesis, we sample a small patch from a bigger image and find other patches that have similar overlapping regions and "quilt" them together to get the resulting image. For texture transfer, we recreate the look of the target image with the texture of a different texture image.
1. Randomly Sampled Texture: The most naive way of texture synthesis is to randomly sample some patches out of the image, then tile sampled patches until the output image is full.
2. Overlapping Patches: A better approach would be to find patches that have low SSD in the overlapping region (top, left, or both) with the current output image, and overlay them by using the middle of the overlap region as the cutting path.
3. Seam Finding The second approach allows us to find patches that relatively match well with each other at the overlapping regions, and this approach further improves it by finding the minimum-cost path between the patches.
Original Image:
(from left to right: Original image, Randomly Sampled Texture; Overlapping Patches; Seam Finding)
Here's an outline of seam finding algorithm:
We first initialize a blank image with the target output size.
We randomly sample 1 patch from the texture image and use it as the top-left patch
We randomly sample patches from the texture image under the constraint that the overlap region (top, left, or both) of the new patch and the existing output image has an SSD lower than a specific threshold
We find the minimum cut on the overlap region using dynamic programming.
Let's say we want to seam these two 85px x 85px patches horizontally, and their lap region has a width of 20 px.
We calculate the SSD of the overlap region (SSD of the right 10px of the first patch and left 10px of the second patch), and then find the min-cost contiguous path from top to bottom (as it's a horizontal seam)
Then we seam them together
More Results
(this one didn't work that well bc of the complicated texture in the original image)
Texture Transfer works quite similarly to texture synthesis. The main difference is that instead of using SSD, we use a weighted sum of overlap error & correspondence error to sample patches
total_error = alpha x overlap_error + (1 - alpha) x correspondence_error
Overlap error: SSD of the overlap between previously synthesized output & newly sampled patch
Correspondence Error: in my case, I use the luminance image intensities for the correspondence map. And Correspondence Error is the SSD of correspondence map of the target image current location & the correspondence map of the sampled patch from the texture image.
Here are some results of Feyman's face with different textures
Thanks for a great semester this class is def one of my fav classes at Berkeley!!