Background
In this project, we attempt to use an image quilting algorithm to generate patterns and do texture transfer. This is outlined in this paper: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/papers/efros-siggraph01.pdf by Efros (our professor!) et al.
The first goal of the project is to do texture generation and synthesis. Given a source image, we hope to generate a larger image using textures and patterns from that image.
The second goal of the project is to do texture transfer, in which we maintain the basic shape of an image, but apply the textures from another source image to it.
Below we will outline the steps on how we achieve these goals.
The first goal of the project is to do texture generation and synthesis. Given a source image, we hope to generate a larger image using textures and patterns from that image.
The second goal of the project is to do texture transfer, in which we maintain the basic shape of an image, but apply the textures from another source image to it.
Below we will outline the steps on how we achieve these goals.
Randomly Sampled Texture
The first very basic approach we will use is to randomly sample patches of the image to generate a pattern. This doesn't work very well, but is a good start to create a larger image that somewhat resembles the source image, as it is made up with pieces of the source image.
On the left below is the original source image, and the right side is the randomly generated image.
On the left below is the original source image, and the right side is the randomly generated image.
Overlapping Patches
We can do a better than just picking random patches though. To smartly pick patches that work well next to each other, we will allow the patches to overlap. We will then calculate an the ssd in the overlap region between possible patches that we are considering adding. We do this for all of the patches we could pull from the original image.
Then using a tolerance threshold, we pick a patch that is within some tolerance of the best possible match (lowest ssd difference in the overlapping region). With this result, we will get patches that have more similarity with each other, resulting in better texture generation.
Below are the results of using overlapping regions. On the left is using a very low tolerance (1e3) , and on the right is using a tolerance of 0.3. We notice that using a low tolerance results in a essentially the original image.
We notice that the results are a lot better than random generation. The bricks and text are way better aligned, resulting in a clean picture. We also note that the image on the right with a higher tolerance is worse, as there are some patches that are not properly aligned.
Then using a tolerance threshold, we pick a patch that is within some tolerance of the best possible match (lowest ssd difference in the overlapping region). With this result, we will get patches that have more similarity with each other, resulting in better texture generation.
Below are the results of using overlapping regions. On the left is using a very low tolerance (1e3) , and on the right is using a tolerance of 0.3. We notice that using a low tolerance results in a essentially the original image.
We notice that the results are a lot better than random generation. The bricks and text are way better aligned, resulting in a clean picture. We also note that the image on the right with a higher tolerance is worse, as there are some patches that are not properly aligned.
Seam Finding
The last improvement we will make is to implement seam craving on the overlap region. Rather than each new patch overwriting the old patch completely.
Below we illustrate an example of how this will work. There are two overlapping patches being considered that correspond to adjacent patches. We calculate a path of least cost between them, and create a mask that splits it along that seam. This way, the overlapping region will hopefully have a smooth transition.
Below we illustrate an example of how this will work. There are two overlapping patches being considered that correspond to adjacent patches. We calculate a path of least cost between them, and create a mask that splits it along that seam. This way, the overlapping region will hopefully have a smooth transition.
Below are the results of texture generation using seam carving
More example
Below is one more example with another pattern, and the results of random, overlap, and seam carving
We notice for this pattern that seam carving makes a really big improvement on the pattern, as it eliminates the clear lines caused by selecting grids by choosing seams that follow the threads.
Texture Transfer
With some small modifications, we can also do texture transfer. By adding a target image, we can add an additional loss for how closely a particular patch matches that target image. By combining this with the pattern generation from above, we can create a new image in the shape of the target image, but with new textures.
Below is the texture image, target image, and the result image.
Below is the texture image, target image, and the result image.
Works pretty well! It applies the sketch style to the old image, while maintaining the face of feynman.
Conclusion and Takeaways
This project really explores image textures and repeating patterns, and ways to generate and apply them. We explored a naive way to generate more of the same pattern, and then improved upon it with overlap errors and seam carving to get better alignment. While simple, these actually worked surprisingly well. Then we applied these ideas to do texture transfer.
One aspect I did struggle with and had to design around was the dimension of the images. Unfortunately, I was unable to have my code work with any and all dimensions, and had to maintain square images. This shouldn't be an issue for the algorithm, but is something to keep in mind as you calculate and take square patches from images.
One aspect I did struggle with and had to design around was the dimension of the images. Unfortunately, I was unable to have my code work with any and all dimensions, and had to maintain square images. This shouldn't be an issue for the algorithm, but is something to keep in mind as you calculate and take square patches from images.