Table of Contents
In this project, I explored gradient-domain processing and applied a "Poisson blending" technique to seamlessly blend a source image into a target image. This Poisson blending technique entails finding pixel values for the target pixels that maximally preserve the gradient of the source image without changing the background pixels.
Say we have target image "t" and the cutout of the source image, "S". In order to construct a new image "v" (the source image cutout blended into the target image) we need to ensure that the gradients inside of "S" are similar to the gradients of the corresponding cutout region of the target image. Outside of the cutout region of the target image, the pixels of "v" with match "t". We formulate this objective into the following least squares problem:
This least squares problem describes the following constraints:
1. For pixels inside the region "S", the new image "v" should have the same pixels as the source image
2. For pixels on the boundary of "S" (pixels that are in S but have at least one neighbor outside of "S"), we use the pixel intensity value of the target image.
3. For pixels outside of "S", the pixel value will be equal to the pixel value of the target image
In this part of the project, I implemented gradient domain processing on a toy example.
My goal was to reconstruct an image "v" by computing the x and y gradients from an image "s".
I also used one additional constraint to ensure the resulting image "v" was the exact reconstruction
of the original image "s": the top left corners of the two images, "v" and "s", should be the same color.
I solved for the image "v" using least squares constraints (Av - b)^2. "A" is a sparse matrix with "total_entries" columns (the number of pixels in the original image "s") and 2*total_entries+1 rows (I needed total_entries equations for the first objective, total_entries equations for
the second objective, and 1 equation for the third objective), and "b" is a known vector with 2*total_entries+1 entries.
As seen below, I was able to recover the original image using gradient domain processing:
Next, I created blended images using Poisson blending. This process entailed the following steps:
1. Creating a mask for the source image using mask-generating code from the following source
2. Translating the source image so that it is placed in the desired (x,y) coordinates in the target image
3. For each color channel, solving for blended image "v" using constraint equation described in the "Introduction" section using least squares (Av-b)^2.
"A" is a sparse matrix with "total_entries" columns (the number of pixels in the original image "s") and "total_entries" rows, and "b" is a known vector with "total_entries" entries.
Here are my results for Poisson blending:
This result was a failure since the region "S" of the source image contained extraneous pixels (snow in the background behind Santa and the reindeers) that I could not conceal with the mask. This caused poor blending in the resulting image "v" as shown below: the snow could be easily seen, which created a clear boundary between the source and target images.
For my Bells and Whistles, I enhanced the Poisson blending procedure by mixing gradients of both the source and target image in the resulting image "v". This involved solving the following constraint with least squares:
In this case, "d_ij" is set to the gradient of the source image if the absolute value of the gradient
of the source image is greater than the absolute value of the target image gradient. Otherwise, "d_ij" is set to
the gradient of the target image.
First, I performed gradient mixing on the images above created with Poisson blending. As shown below, gradient mixing did not improve
the blending result:
However, I did notice significant improvement while blending an image of text onto another image. As seen below, the "hello" text in the gradient mixing result was blended smoothly into the pink background and did not have any artifacts of the mask in the resulting blended image unlike the Poisson blended result.
In this project, I implemented the image quilting algorithm for texture synthesis and transfer as described in this paper. My first goal was texture synthesis, the creation of a larger texture image from a small sample. Next, my goal was texture transfer, where an object is given the apperance of having the same texture as a sample while still preserving its shape.
The first texture synthesis algorithm I implemented was random texture sampling. This method entails randomly sampling square patches from the sample texture image and tiling these random samples in the output image until it is full.
This texture synthesis technique involved the following steps:
1. Randomly sample an initial patch from the sample tecture image
2. Sample a new patch to overlap with the existing patch. The patch that is selected
to overlap with the existing patch is chosen such that the cost of the overlapping region is
below a certain threshold determined by the minimum overlap cost across all candidate patches.
3. Repeat step 2 and continue to place patches in the output image until it is completely filled.
This technique applies the overlapping patches method described above but also finds the minimum cut path through the overlap SSD in order to obtain a binary mask that specifies which pixels to copy from the newly sampled patch and which to copy from the existing patch. This process is illustrated below. As shown in the illustration, the minimum cost path of the overlap SSD is the seperation between the "0" and "1" regions in the boundary mask.
Here are the results of my texture synthesis using the techniques described above. As seen below, the random sampling approach yields unsatisfactory textures, as there are clear boundaries visible in the output image between each patch. The overlapping patches image result is an improvement, but still creates slightly choppy textures with somewhat visible boundaries. The seam finding approach yields the best texture images out of all of the approaches, as it creates the most realistic looking texture with almost no visible boundaries.
Lastly, I implemented texture transfer. I used the existing seam finding approach but modified the error term such that it also includes the SSD of the pixels in the candidate texture patch and the pixels at the current target image position. This yielded some very interesting texture transfer images as shown below:
