CS194-26 Final Project
Kevin Lin, klinime@berkeley.edu
Image Quilting
Introduction
The goal of this project is to synthesize images with arbitrary size that contain the same texture as a reference image, mainly by autoregressively sampling image patches that minimize their overlap with the generated samples.
Randomly Sampled Texture
The most naive way to perform quilting is to randomly sample a patch and and tile the patch to fit the target dimension. Outputs:
You can see that the border between patches are distinct and the repetition is somewhat dull.
Overlapping Patches
This is the autoregressive sampling that compute a sample’s cost as the sum of squared difference (SSD) between the overlapping area of the sample and the generated image. A tolerance perameter tol is used to filter the samples with cost > min_cost * (1 + tol), and a sample is then randomly chosen. Outputs:
Sometimes and sometimes bad. The border is still distinct but now we have more variation.
Seam Finding
Seam carving is a dynamic programming algorithm that computes the seam across an image (vertically or horizontally) with the lowest total cost. Although the MATLAB code is given to us, since I don’t know MATLAB, I wrote my own seam carving in Python for bells & whistles. 🤣
In our case, the image to carve is the overlapping region and the cost is the SSD. Thus, we find the seam with the minimal difference between our new sample and generated samples, and merge them accordingly. Outputs:
Significantly better! Although you can see some hiccups if you look close enough.
(Bells & Whistles) Seam Carving
The algorithm for seam carving is as following:
- Suppose the objective is to find a vertical seam (transpose image yields the horizontal).
- Create a DP array the same size as the image, with each entry representing the cost of the minimum seam ending at the pixel.
- Copy the first row of the cost array to the first row of the DP array.
- For each subsequent row, the seam cost is the minimum of the neighboring pixels from the previous row plus the cost of the current row.
- The pixel with minimum seam cost at the last row is part of the minimal seam.
- Backtrack by appending the neighboring pixel with minimum seam cost.
Visualize:
+ 
(
) => 
Texture Transfer
Previously, we generated with texture as the only contraint. Now, we add an additional contraint to match the pixels from another reference image, essentially retain the outline of an image while changing its texture. We do this by augmenting a patch’s cost with the SSD between the patch and the target region, with a multiplier alpha to control the weight. Outputs:
Quite good if I say so myself!
Gradient Domain Fushion
Introduction
The goal of this project is to incorporate objects from a source image in a target image, while blending such that the the resulting image look natural as a whole. This is achievable by taking advantage of humans using image gradients as an indicator of smooth integration, so by matching the images’ gradients, the result would look natural by human standards.
Toy Problem
Mathematically, we want to solve for 
Where v are the variables to solve for (blended image), s is the source image, t is the target image, i index the position of all pixels and j index the four direct neighbors of pixel at i. The difference between pixel i and j represents the gradient from i to j.
By solving the least squares to minimize the above cost with an additional term to match the first pixel’s intensity (since gradients only capture relative intensities), we should be able to recover an image if we input that as the source and target.
The left is the input image and the right is the reconstructed by least squares. Looks identical!
Poisson Blending
Theory remains the same as the toy problem, except now we need to treat pixels differently depending on whether the pixel is part of the source, part of the target, or on the border. The toy problem could be computed very fast despite its gigantic dimensions by exploiting the structured sparsity. The matrix now is sparse but not structured, which makes computing the least squares very slow. Therefore, I cheated here to instead of minimizing each neighbor individually, I minimize a pixel with all four neighbors at the same time such that the matrix is square and can be solved using scipy.linalg.spsolve instead of scipy.linalg.lsqr, speeding up the computation drastically.
These are the masks:
And here are the outputs:
Final Thoughts
I was almost driven crazy by how slow the least squares in Gradient Domain Fushion took (HOURS!) before I implemented the cheat unwillingly 😖. I had no guarantee that the result from the cheat would be good, since the equation I solved was mathematically different, but still captured the same idea, so it was a huge relief to see the outputs. Compared to this, Image Quilting was a lot smoother of a ride and very easy to understand intuitively.
CS194-26 Final Project
Kevin Lin, klinime@berkeley.edu
Image Quilting
Introduction
The goal of this project is to synthesize images with arbitrary size that contain the same texture as a reference image, mainly by autoregressively sampling image patches that minimize their overlap with the generated samples.
Randomly Sampled Texture
The most naive way to perform quilting is to randomly sample a patch and and tile the patch to fit the target dimension. Outputs:
You can see that the border between patches are distinct and the repetition is somewhat dull.
Overlapping Patches
This is the autoregressive sampling that compute a sample’s cost as the sum of squared difference (SSD) between the overlapping area of the sample and the generated image. A tolerance perameter tol is used to filter the samples with cost > min_cost * (1 + tol), and a sample is then randomly chosen. Outputs:
Sometimes and sometimes bad. The border is still distinct but now we have more variation.
Seam Finding
Seam carving is a dynamic programming algorithm that computes the seam across an image (vertically or horizontally) with the lowest total cost. Although the MATLAB code is given to us, since I don’t know MATLAB, I wrote my own seam carving in Python for bells & whistles. 🤣
In our case, the image to carve is the overlapping region and the cost is the SSD. Thus, we find the seam with the minimal difference between our new sample and generated samples, and merge them accordingly. Outputs:
Significantly better! Although you can see some hiccups if you look close enough.
(Bells & Whistles) Seam Carving
The algorithm for seam carving is as following:
Visualize:
+ 
(
) => 
Texture Transfer
Previously, we generated with texture as the only contraint. Now, we add an additional contraint to match the pixels from another reference image, essentially retain the outline of an image while changing its texture. We do this by augmenting a patch’s cost with the SSD between the patch and the target region, with a multiplier alpha to control the weight. Outputs:
Quite good if I say so myself!
Gradient Domain Fushion
Introduction
The goal of this project is to incorporate objects from a source image in a target image, while blending such that the the resulting image look natural as a whole. This is achievable by taking advantage of humans using image gradients as an indicator of smooth integration, so by matching the images’ gradients, the result would look natural by human standards.
Toy Problem
Mathematically, we want to solve for
Where v are the variables to solve for (blended image), s is the source image, t is the target image, i index the position of all pixels and j index the four direct neighbors of pixel at i. The difference between pixel i and j represents the gradient from i to j.
By solving the least squares to minimize the above cost with an additional term to match the first pixel’s intensity (since gradients only capture relative intensities), we should be able to recover an image if we input that as the source and target.
The left is the input image and the right is the reconstructed by least squares. Looks identical!
Poisson Blending
Theory remains the same as the toy problem, except now we need to treat pixels differently depending on whether the pixel is part of the source, part of the target, or on the border. The toy problem could be computed very fast despite its gigantic dimensions by exploiting the structured sparsity. The matrix now is sparse but not structured, which makes computing the least squares very slow. Therefore, I cheated here to instead of minimizing each neighbor individually, I minimize a pixel with all four neighbors at the same time such that the matrix is square and can be solved using scipy.linalg.spsolve instead of scipy.linalg.lsqr, speeding up the computation drastically.
These are the masks:
And here are the outputs:
Final Thoughts
I was almost driven crazy by how slow the least squares in Gradient Domain Fushion took (HOURS!) before I implemented the cheat unwillingly 😖. I had no guarantee that the result from the cheat would be good, since the equation I solved was mathematically different, but still captured the same idea, so it was a huge relief to see the outputs. Compared to this, Image Quilting was a lot smoother of a ride and very easy to understand intuitively.