CS194 Final Projects

By: Scott Shao

Image Quilting

Overview

The goal of this project is the implement the image quilting algorithm for texture synthesis and transfer, first described in the SIGGRAPH 2001 paper by Efros and Freeman. Texture synthesis is the creation of a larger texture image from a small sample. Texture transfer is giving an object the appearance of having the same texture as a sample while preserving its basic shape.

Example of texture synthesis
Example of texture transfer

Randomly Sampled Texture

First I created a function that randomly sample square patches from the source image to fill the output image.

Sample source image
Randomly sampled texture output

As shown above, the random sample method was fast but gave really poor result.



Overlapping Patches

In order to solve the issue from the random sample method, a overlap region between the filled part of the output image and the patch was introduced. For every possible patch from the source image, the SSD of the overlapped region was calculated to produce a cost image. The cost image was then used to find the patch with the minimum cost. Then a patch was randomly selected from a set of the patches with a set tolerance from the minimum cost patch. If the tolerance is set to be very low and the first top left patch to be the top left of the source image, then he original source image or part of the source image was reproduced.

Overlapping patches output with tolerance = 0.00001
Cost image
Ovverlapping patches output with tolerance = o.3

Although the tolerance of the above two output images are difference, the result output images are the same. They both produce the first 100 x 100 pixels region of the top left of the source image. This might be due to the patch size was too big such that it captures too much information about a particular brick. I have tried with various patch size ranging from 20 down to 5, but the result stays the same.



Seam Finding

Although implementing overlapping patches helps with finding the right patch to minimize the SSD of the overlapping region, it is fairly common that the overlapping patches produce many edge artifacts. Seam finding was introduced to minimize the edge artifacts between two overlapping patches. The error surface of the overlapping region was first calculated using SSD, then the minimum continuous seam was found for both horizontal seam and vertical seam using dynamic programming. The patch found using overlapping patches algorithm can then be cut based on the seams found before paste to the output image to minimize the edge artifacts.

Example of overlapping patches edge artifacts
Example of the overlapped region 1
Example of the overlapped region 2
Error surface of the overlapping region 1 and 2
Minimum error surface calculated from the error surface using dynamic programming
seam found from the minimum error surface
Overlapped region 1 with calculated seam
Overlapped region 2 with calculated seam
Output of the overlapped region with seam finding

The minimum error surface calculated from the error surface used dynamic programming to find the optimal continuous seam. Then the optimal seam can be found by traversing backward from the end of the minimum error surface starting with the minimum value. A mask was then created to apply to the patch and output image to apply the seam correctly to the overlapped region.

Source image
Seam finding output using 5 patch size, 3 overlap, and 0.1 tolerance
Source image
Seam finding output using 33 patch size, 5 overlap, and 0.3 tolerance
Source image
Seam finding output using 33 patch size, 5 overlap, and 0.3 tolerance
Source image
Seam finding output using 33 patch size, 5 overlap, and 0.3 tolerance

The complete image quilting algorithm is as follow:

1. Go through the image to be synthesized in raster scan order in steps of one block (minus the overlap).

2. For every location, search the input texture for a set of blocks that satisfy the overlap constraints (above and left) within some error tolerance. Randomly pick one such block.

3. Compute the error surface between the newly chosen block and the old blocks at the overlap region. Find the minimum cost path along this surface and make that the boundary of the new block. Paste the block onto the texture. Repeat.

Texture Transfer

The same image quilting algorithm can also be used for texture transfer with some modification to the error term. For texture transfer, image being synthesized must respect two independent constraints: (a) the output are legitimate, synthesized examples of the source texture, and (b) that the correspondence image mapping is respected. We modify the error term of the image quilting algorithm to be the weighted sum, alpha times the block overlap matching error plus (1 - alpha) times the squared error between the correspondence map pixels within the source texture block and those at the current target image position. The parameter determines the tradeoff between the texture synthesis and the fidelity to the target image correspondence map.

Texture transfer source
Texture transfer target
Texture transfer output partial image
alpha = 0.1, patch size = 9, overlap = 3, tolerance = 0.1
Texutre transfer output image
alpha = 0.8, patch size = 33, overlap = 5, tolerance = 0.01
Texture transfer source
Texutre transfer target
Texture transfer output image
alpha = 0.1, patch size = 9, overlap = 3, tolerance = 0.1

However, I have trouble to produce good result with the implemented algorithm described in the paper without the iterated method. I believe with the correct parameter through tuning the result would be much better.

Texture transfer source
Texture transfer target

Bells and Whistles

I have created my own version of the cut.m. Please refer to the code for detailed implementation in python 3.

Seam Carving

Overview

The goal of this project is the implement the seam carving algorithm for content-aware image resizing, first described in the SIGGRAPH paper by Avidan and Shamir. The algorithm will shrink the source image to a given dimension while try to preserve the most important contents in the image.

algorithm

The basic algorithm is as the follows:

1. Use a energy function to determine the importance of each pixel in a given source image.

2. Find the seam with the lowest importance in the source image.

3. Keep removing the seam with the lowest importance in the source image until the desired dimension is reached.

The algorithm was written with dynamic programming and the energy function used is cv2.Scharr. The energy function calculates the first x- or y- image derivatives using Scharr operator.

Source image
Energy map produced by Scharr function
Minimum energy map
Energy map produced by Scharr function
First horizontal seam with minimum energy
First vertical seam with minimum energy

Results

Source image: Surfer
Surfer output image
Source image: Cliff
Cliff output image
Source image: Feeding
Feeding output image
Source image: House
House output image
Source image: Last supper
Last supper output image
Source image: Tower
Tower output image
Source image: Girl
Girl output image
Source image: Road
Road output image

Fails:

Source image: Border collie
Border collie output image

Since the border collie is mostly black and white, the algorithm thinks that the dog has less energy.

Source image: Building
Building output image

The building is mostly horizontal and vertical lines, thus the energy map thinks it has less energy than the surrounding trees.

Source image: Fat person
Fat person output image

The result is not very bad, but since the whole image is a person when deleting vertical seams the edge of the person will be distorted, makes the output image looks bad.

Source image: Portrait
Portrait output image

One side of the face of the person is just shadow, thus the energy map thinks that portion of the image has less energy, makes the shrinked output image looks like the head of the person is cut in half.

Source image: Girl running on trail
Girl running on trail output image

This result is actually quite suprising, my guess is the background where the girl is was dark, similar to the clothing the girl was wearing, thus creating a less energy vertical seam in the energy map, such that the girl was being deleted when the width of the image was shrinking.

The most importance thing I learned from this project is the utilization of dynamic programming in the image processing pipeline. I have tested several different way of approaching the production of the minimum energy map without ddynamic programming and compare the speed with the dynamic programming method. I found out that none of the other method would yield the similar speed as the dynamic programming. This project is also a very helpful preparation for the image quilting project.