Image Quilting

Andrew Vo


UC Berkeley COMPSCI 194-26, Fall 2020

Learn more

Introduction

The goal of this project is to implement an image quilting algorithm for texture synthesis and transfer. Texture synthesis is the process of expanding a small input texture sample into a larger one. Texture transfer is giving a target image the texture properties of a source image while maintaining its shape. Many of the methods in this project are inspired by this SIGGRAPH 2001 paper by Alexei Efros and William Freeman.

Randomly Sampled Texture

Let's start with texture synthesis. The simplest way to expand a texture sample is to randomly sample small patches from the texture and stitch them together to create a larger texture. Several examples are shown below.

Overlapping Patches

As you can see, randomly sampling patches creates a result that has a grid-like structure to it with many edge artifacts in the image. We want to create a smoother texture image that looks like it has not been sampled at all. In this section, we will improve our quilting algorithm by choosing samples with a high degree of similarity in overlapping regions between them.

Let's define two patches $B_1$ and $B_2$ with overlapping regions $B_{o_{1}}$ and $B_{o_{2}}$ respectively. Since we want the difference between the overlapping regions to be small, I took the sum of squared differences ($SSD$) of the overlapping regions and added the patches to a set $S$, ordered by their SSD value. This cost formula can be expressed in the equation below.

$$SSD = \sum \left(B_{o_{1}} - B_{o_{2}}\right)^2$$

After creating $S$, I filtered out the patches that were not within a constant factor of the minimum differing patch and finally randomly sampled a patch from the set to be put in our final texture. The filtering process is shown below. $B_{SSD}$ is the $SSD$ value of patch $B$ and $t$ is a small tolerance constant.

$$S = \{B\} \hspace{0.1in} s.t. \hspace{0.1in} B_{SSD} < B_{min_{SSD}} * (1 + t)$$

After implementing this algorithm, we can see that our results look significantly better than randomly choosing samples. Note that some of the images have black borders at the edges because the patches did not fit evenly in the resulting image shape.

Seam Finding

Although our results from the previous section look excellent, we can still improve our textures by cutting the overlapping regions of patches along its minimum error boundary instead of cutting it in half. A diagram of this from the research paper is shown below.

To find the minimum error boundary, we will need to use dynamic programming to find the minimum cost contiguous path from the top to the bottom of the overlap. We can utilize the same $SSD$ metric used in the previous section to guide our dynamic programming algorithm modeled below. $E$ represents the minimum cumulative error up to some position $(i, j)$ in our overlap area and $e$ represents the $SSD$ cost at some position $(i, j)$.

$$E(i, j) = e_{i, j} + min\{E(i - 1, j - 1), E(i - 1, j), E(i - 1, j + 1)\}$$

This process is illustrated with some example pictures.

Patch 1
Patch 2
Overlap 1
Overlap 2
SSD Cost
Minimum Error Cut

After applying seam finding to the images in the previous section and to several of my own images, these are the results:

Comparing Methods

After going through the sections above, we can see how our results improved with each method.

Texture Transfer

We can do more than just creating larger texture samples with our image quilting algorithm. Using our texture synthesis code, we can slightly modify it so that we can transfer textures from a source image to a target image. This modification involves adjusting our $SSD$ cost metric such that it takes in an additional term for a correspondence map $\overrightarrow{C}$ between the source and target images. This map allows us to pair patches between the images that have a similar corresponding quantity such as illuminance, orientation angles, etc. The redefined cost metric is shown below. $\alpha$ is a constant value between 0 and 1 that will determine the weight of our SSD terms.

$$SSD = \alpha * \sum \left(B_{o_{1}} - B_{o_{2}}\right)^2 + (1 - \alpha) * SSD_{\overrightarrow{C}}$$

The results of texture transfer on some example images are shown below. For the correspondence map, I used the blurred illuminance values between images to define the pair relationships.

Bells & Whistles

For Bells & Whistles, I coded my own cut function in Python. In my implementation, my function takes in the overlapping cost image as input and performs the dynamic programming algorithm described in the seam section. In addition to keeping track of the lowest cost error through the overlap, I stored the coordinate positions so that I could reconstruct the min-cost path working backward. Although my function was designed for vertical overlaps, it also works for horizontal overlaps by rotating the horizontal overlap to the right by 90 degrees. The path returned by my implementation would then simply have their $x$ and $y$ values swapped so that the seam travels from the left to the right of the overlap.

Final Thoughts

Overall, this project was very interesting to work on! I had an enjoyable time coding the texture synthesis algorithms and creating funny texture transfer images. I had a lot of difficulty with image alignment in the overlapping and seam finding sections of the project but I was able to overcome these challenges by breaking down the texture creation into smaller subproblems. I am excited to see how else texture synthesis can be applied to images in unique ways!

Click here to see my Light Field Camera project!