Overview

For my final project, I worked on two pre-canned projects:

• Image Quilting
• Neural Algorithm of Artistic Style

Image Quilting

In this project, I worked on implementing the image quilting algorithm for texture synthesis and texture transfer. The algorithm is based on this paper. In texture synthesis, a larger texture image is created by combining many samples of a smaller texture image. We implement this with 3 different algorithms. In texture transfer, we use our methods for texture synthesis and add additional constraints to make the samples look similar to a target image.

Algorithms for Texture Synthesis

We perform texture synthesis using 3 different algorithms, starting off with simply combining randomly sampled patches without checking for compatability, adding overlap between patches to ensure that adjacent patches match decently, and using seam finding so to minimize the seams in the overlapping regions.

Random Patch Texture

Our first algorithm works by first creating an empty image of the desired size, then iteratively filling in the empty image with a randomly sampled patch (with size based on the user) from the source texture image until the image has been filled. In the regions of the image in which the entire sampled patch would not fit, i.e. the corners and edges, we simply truncate the patch so that it would fit.

Overlapping Patches

From our first algorithm results, we noticed that there were very noticable "borders" or visual artifacts between adjacent patches. Instead of simply selecting any random patch from the set of possible patches from our source texture image, we first find a filtered set of patches based on how well the overlapping regions matches with the current image. Each of the patches will overlap by a predefined margin and we use the SSD cost function to measure how well the overlapping regions match. Our algorithm is now as follows:

1. Compute a set of all possible candidate patches and their centers.
2. Find the region of overlap of where the patch would be placed, and the one's that have already been placed.
3. Compute the SSD of all candidate patches and the overlapping patches.
4. Filter out any patches that have ssd that do not fall within our desired tolerance range: (ssd <= (1.0+tolerance)*minE)
5. Select a random sample from the remaining patches.
6. Paste the sample into the corresponding region in the texture result.
7. Repeat steps 2-6 until image is filled to the desired output size.

Seam Finding

The overlapping patches approach definitely improved our result, but there were still edge artifcats. We implement seam finding to find the min-cost cut in the overlapping region, and form a mask that determines which pixels we will use in the new patch that we found from step 5 of the previous algorithm. I created my own implementation of Seam Finding for Bell's and Whistles, which I describe in the Bell's and Whistles section.

Texture Transfer

Texture Transfer followed nearly the same approach as texture synthesis, with the exceptions that we now also take in as input a target image which we used to structure/model our output off of, and changed our cost function to account for similarity to the target image. Additionally, we create a new parameter alpha that determines the weight applied to the cost associated with the overlap region and the new cost for similarity to the target region. Our algorithm is now as follows:

1. Compute a set of all possible candidate patches and their centers.
2. Find the region of overlap of where the patch would be placed, and the one's that have already been placed.
3. Compute the SSD of all candidate patches and the overlapping patches.
4. Blur the target image (intensity image) and the candidate patches.
5. Compute the SSD between the candidate patches and the corresponding region in the intesity imagae.
6. Compute the resulting cost value: alpha*SSD_overlap + (1-alpha)*SSD_intensity
7. Filter out any patches that have ssd that do not fall within our desired tolerance range: (ssd <= (1.0+tolerance)*minE)
8. Select a random sample from the remaining patches.
9. Paste the sample into the corresponding region in the texture result.
10. Repeat steps 2-9 until image is filled to the desired output size, which is now the size of the target image.

Bell's and Whistles: Seam Finding Implementation

Given a randomly sampled patch, we compute an error surface which is the ssd over channels of the sampled patch and the region for placement. There were 3 scenarios which we had to account for: vertical overlaps, horizontal overlaps, vertical and horizontal overlaps. We implemented a dynammic programming algorithm which takes in the error surface and finds the minimum cost path along the surface. We then used the minimum cost path to generate a mask containing the boundary in which we would use the pixels of the newly sampled patch. Thankfully, the process of seam finding was symmetric for vertical and horizontal overlaps, so we simply just implemented for the vertical direction and for the horizontal path, just used the vertical path method but transposed the input and the output. For the patches that overlapped in both, we individually computed the mask for the vertical and horizontal path, then combined them simply as follows: mask_h @ mask_v. Once we've found the masks, we can easily apply them to our sampled patch and paste it onto our texture result. The following images show the seam finding process for a pair of patches. In the mask, the white pixels denote pixels of value 1, meaning that we will use the corresponding pixel on the right patch in the overlapping region. As one can see, the algorithm works like a cookie cutter through the overlapping region to make the split.

Texture Synthesis Results

Texture Transfer Results