CS194 Final Projects
Image Quilting
Overview
The goal of this project is the implement the image quilting algorithm for texture synthesis and transfer, first described in the SIGGRAPH 2001 paper by Efros and Freeman. Texture synthesis is the creation of a larger texture image from a small sample. Texture transfer is giving an object the appearance of having the same texture as a sample while preserving its basic shape.
Randomly Sampled Texture
First I created a function that randomly sample square patches from the source image to fill the output image.
As shown above, the random sample method was fast but gave really poor result.
Overlapping Patches
In order to solve the issue from the random sample method, a overlap region between the filled part of the output image and the patch was introduced. For every possible patch from the source image, the SSD of the overlapped region was calculated to produce a cost image. The cost image was then used to find the patch with the minimum cost. Then a patch was randomly selected from a set of the patches with a set tolerance from the minimum cost patch. If the tolerance is set to be very low and the first top left patch to be the top left of the source image, then he original source image or part of the source image was reproduced.
Although the tolerance of the above two output images are difference, the result output images are the same. They both produce the first 100 x 100 pixels region of the top left of the source image. This might be due to the patch size was too big such that it captures too much information about a particular brick. I have tried with various patch size ranging from 20 down to 5, but the result stays the same.
Seam Finding
Although implementing overlapping patches helps with finding the right patch to minimize the SSD of the overlapping region, it is fairly common that the overlapping patches produce many edge artifacts. Seam finding was introduced to minimize the edge artifacts between two overlapping patches. The error surface of the overlapping region was first calculated using SSD, then the minimum continuous seam was found for both horizontal seam and vertical seam using dynamic programming. The patch found using overlapping patches algorithm can then be cut based on the seams found before paste to the output image to minimize the edge artifacts.
The minimum error surface calculated from the error surface used dynamic programming to find the optimal continuous seam. Then the optimal seam can be found by traversing backward from the end of the minimum error surface starting with the minimum value. A mask was then created to apply to the patch and output image to apply the seam correctly to the overlapped region.
The complete image quilting algorithm is as follow:
1. Go through the image to be synthesized in raster scan order in steps of one block (minus the overlap).
2. For every location, search the input texture for a set of blocks that satisfy the overlap constraints (above and left) within some error tolerance. Randomly pick one such block.
3. Compute the error surface between the newly chosen block and the old blocks at the overlap region. Find the minimum cost path along this surface and make that the boundary of the new block. Paste the block onto the texture. Repeat.
Texture Transfer
The same image quilting algorithm can also be used for texture transfer with some modification to the error term. For texture transfer, image being synthesized must respect two independent constraints: (a) the output are legitimate, synthesized examples of the source texture, and (b) that the correspondence image mapping is respected. We modify the error term of the image quilting algorithm to be the weighted sum, alpha times the block overlap matching error plus (1 - alpha) times the squared error between the correspondence map pixels within the source texture block and those at the current target image position. The parameter determines the tradeoff between the texture synthesis and the fidelity to the target image correspondence map.
alpha = 0.1, patch size = 9, overlap = 3, tolerance = 0.1
alpha = 0.8, patch size = 33, overlap = 5, tolerance = 0.01
alpha = 0.1, patch size = 9, overlap = 3, tolerance = 0.1
However, I have trouble to produce good result with the implemented algorithm described in the paper without the iterated method. I believe with the correct parameter through tuning the result would be much better.
Bells and Whistles
I have created my own version of the cut.m. Please refer to the code for detailed implementation in python 3.
Seam Carving
Overview
The goal of this project is the implement the seam carving algorithm for content-aware image resizing, first described in the SIGGRAPH paper by Avidan and Shamir. The algorithm will shrink the source image to a given dimension while try to preserve the most important contents in the image.
algorithm
The basic algorithm is as the follows:
1. Use a energy function to determine the importance of each pixel in a given source image.
2. Find the seam with the lowest importance in the source image.
3. Keep removing the seam with the lowest importance in the source image until the desired dimension is reached.
The algorithm was written with dynamic programming and the energy function used is cv2.Scharr. The energy function calculates the first x- or y- image derivatives using Scharr operator.
Results
Fails:
Since the border collie is mostly black and white, the algorithm thinks that the dog has less energy.
The building is mostly horizontal and vertical lines, thus the energy map thinks it has less energy than the surrounding trees.
The result is not very bad, but since the whole image is a person when deleting vertical seams the edge of the person will be distorted, makes the output image looks bad.
One side of the face of the person is just shadow, thus the energy map thinks that portion of the image has less energy, makes the shrinked output image looks like the head of the person is cut in half.
This result is actually quite suprising, my guess is the background where the girl is was dark, similar to the clothing the girl was wearing, thus creating a less energy vertical seam in the energy map, such that the girl was being deleted when the width of the image was shrinking.
The most importance thing I learned from this project is the utilization of dynamic programming in the image processing pipeline. I have tested several different way of approaching the production of the minimum energy map without ddynamic programming and compare the speed with the dynamic programming method. I found out that none of the other method would yield the similar speed as the dynamic programming. This project is also a very helpful preparation for the image quilting project.