Comparing the Different Methods

In this project, we implemented three methods for sampling patches for texture synthesis.

In random sampling, the patches are chosen at random from the source image and tiled together. This results in mismatches between neighboring patches and can lead to weird-looking results.

To reduce these artifects, we take into account of previously places patches by introducing overlaps between patches. When placing a new patch, we take into account of the patches that have already been placed into the block and search across the source image for matches. We sort by the error for all sampling positions in the source image and randomly choose from the positions where the error is smallest. This improved the results but there can still be visible seam-like artifects in the output texture.

Finally, we used a seam-finding algorithm on the squared differences in the overlapping patches to mask the new patch before placing it. This removed most artifects, resulting in a natural-looking texture on the right.

Seam-finding Example

An illustration of the seam-finding algorithm is given below. The first two images are the overlapping patches with the overlapping parts marked. The third image shows the squared error between them and the fourth shows the min-cost path and the mask produced from the path for blending the new patch.

Texture Synthesis

To synthesize a new texture given an input image, we first place a randomly selected patch from the source image to the top left of the output image. We then iteratively “grow” the output texture row by row going to the right using the procedure described above to find suitable patches to place into the output image based on overlaps. When placing new patches, we mask them along the min-cost path using the seam-finding algorithm. Some other textures synthesized by the algorithm are shown below.

Texture Transfer

The texture synthesis algorithm can also be used for transferring textures to a guidance image while largely keeping the latter’s overall shape. To achieve this, we can add another error term comparing the differences between the sampled patches to the qualities of the target block in the guidance image when choosing patches based on overlaps. Some outcomes of texture transfer are given below.

Bells and Whistles

Custom version of cut

We built our own version of the cut helper function as customized_cut in the Jupyter notebook.