CS194 Final Project: Image Quilting

The goal of this project is the implement the image quilting algorithm for texture synthesis and transfer, first described in the SIGGRAPH 2001 paper by Efros and Freeman. exture synthesis is the creation of a larger texture image from a small sample. Texture transfer is giving an object the appearance of having the same texture as a sample while preserving its basic shape.

First I created a function that randomly sample square patches from the source image to fill the output image.

As shown above, the random sample method was fast but gave really poor result.

In order to solve the issue from the random sample method, a overlap region between the filled part of the output image and the patch was introduced. For every possible patch from the source image, the SSD of the overlapped region was calculated to produce a cost image. The cost image was then used to find the patch with the minimum cost. Then a patch was randomly selected from a set of the patches with a set tolerance from the minimum cost patch. If the tolerance is set to be very low and the first top left patch to be the top left of the source image, then he original source image or part of the source image was reproduced.

Although the tolerance of the above two output images are difference, the result output images are the same. They both produce the first 100 x 100 pixels region of the top left of the source image. This might be due to the patch size was too big such that it captures too much information about a particular brick. I have tried with various patch size ranging from 20 down to 5, but the result stays the same.

Although implementing overlapping patches helps with finding the right patch to minimize the SSD of the overlapping region, it is fairly common that the overlapping patches produce many edge artifacts. Seam finding was introduced to minimize the edge artifacts between two overlapping patches. The error surface of the overlapping region was first calculated using SSD, then the minimum continuous seam was found for both horizontal seam and vertical seam using dynamic programming. The patch found using overlapping patches algorithm can then be cut based on the seams found before paste to the output image to minimize the edge artifacts.

The minimum error surface calculated from the error surface used dynamic programming to find the optimal continuous seam. Then the optimal seam can be found by traversing backward from the end of the minimum error surface starting with the minimum value. A mask was then created to apply to the patch and output image to apply the seam correctly to the overlapped region.

The complete image quilting algorithm is as follow:

1. Go through the image to be synthesized in raster scan order in steps of one block (minus the overlap).

2. For every location, search the input texture for a set of blocks that satisfy the overlap constraints (above and left) within some error tolerance. Randomly pick one such block.

3. Compute the error surface between the newly chosen block and the old blocks at the overlap region. Find the minimum cost path along this surface and make that the boundary of the new block. Paste the block onto the texture. Repeat.

The same image quilting algorithm can also be used for texture transfer with some modification to the error term. For texture transfer, image being synthesized must respect two independent constraints: (a) the output are legitimate, synthesized examples of the source texture, and (b) that the correspondence image mapping is respected. We modify the error term of the image quilting algorithm to be the weighted sum, alpha times the block overlap matching error plus (1 - alpha) times the squared error between the correspondence map pixels within the source texture block and those at the current target image position. The parameter  determines the tradeoff between the texture synthesis and the fidelity to the target image correspondence map.

However, I have trouble to produce good result with the implemented algorithm described in the paper without the iterated method. I believe with the correct parameter through tuning the result would be much better.

I have created my own version of the cut.m. Please refer to the code for detailed implementation in python 3.