Image Quilting

Overview of Methods

In order to reimplement the SIGGRAPH 2001 by Prof. Efros, we take several steps of sampling and overlapping textures.

Randomly Sampled Texture (10 pts)

First, I found a randomly sampled texture. Here I just looped through every possible block in my output image and assigned it to a subsample of my initial texture image. Of course, this will result in odd artifacts throughout the picture since edges will get cut off. Here are some examples:


As you can see, these results are pretty bad. While the random brick texture could possible past, we see that the text looks awful (perhaps because our block size is too big).

Overlapping Patches (30 pts)

In order to fix the harsh edges caused by random sampling, looking for the best patch to overlap a certain number of pixels across patches is a valid idea. So, for this section of reimplimentation of the paper, I looked at every single patch that could be sampled from the original texture image and found the ssd between the existing patch and the sample and tried to minimize it. Of course, in order to do this, I did not manually search every patch -- rather I used cv2's filtering functions that allowed me to pass in a kernel. I had to split the image up across red, green, and blue channels, though. That's why my total ssd was actually the sum of the filtering throughout the channels. After doing this, I simply overlapped the old existing newly made texture with the newly selected sample. Here are some examples:

Seam Finding (20 pts)

Wow! That's a lot better. Still we see that there are a few places where the bricks get oddly cut off and the text blocks look odd together. That's why we want to maintain jagged edges. As presented in the paper, we now calculate the min cut where the cost of cutting through a pixel is the ssd cost. Of course, the path should only be through the overlapped region (or else you end up with holes!). Here are some examples:




Texture Transfer (30 pts)

Finally, transferring texture is a very similar idea to creating an image quilt, except that when finding the cut, we add on the cost of the correspondance map. I created the correspondance map by looking at the intensities of each image and thus, a single channel ssd. This ssd was added on to the cost of the original cost of cut using weighted averaging with alpha. Changing alpha leads to slightly different results, but I found that this part was very sensitive to how big the box size was. Here are some examples:
Assuming we start with the following two images (first = texture, second = target):

Here is a result: Here is another duo:

Here is the result using block size 8x8:

Here is the result using block size 16x16:

Here is the result using block size 32x32:

Here is the result using block size 64x64:

Eulerian Video Magnification

Laplacian pyramid (20pts)

In order to build the laplacian pyramid, I calculated a gaussian pyramid that repeatedly blurred and downsized an image. Then to extract the higher frequencies, at every level of the pyramid, I subtracted the layer underneath it (making sure to resize accordingly). P.S. I built my Laplacian stack in the YIQ space, but to display, I will bring it back to RGB. Here are some examples of the first frame and it's pyramid.

Temporal filtering (40pts)

In order to temporally filter, I converted my signal to the frequency domain using scipy's fft.rfft functions and then created a mask which 0'ed out any frequencies that weren't in the right range. Finally, I used fft.irfft to convert back to the YIQ space with pixels.

Pixel change magnification (10pts)

Once I got the amplified pixel back, I amplified it using alpha. This is a simple multiplication. Here is an example of what the amplified signal loks like:

Image reconstruction (30pts, which includes evaluation of your results)

Here is face with low cutoff at 0.83, high cutoff at 1, amplification at 100
Here is baby2 with low cutoff at 2.33, high cutoff at 1.67, amplification at 150
Here is a video of me with low cutoff at 0.9, high cutoff at 1.1, amplification at 100
Here is the same video with low = 0.8 and high=1.2 but amplification at 60. Interestingly, this video shows that nothing really changed.
I also attempted to run the face experiment again, this time only amplifying the middle layers of my laplacian pyramid. Here are the results -- it seems that this was more motion amplification :
Middle Layers Amplified Only

As you can see, choosing the parameters makes a huge difference. If I had more time, I would probably experiment with applying different alphas to different layers. I'd also try to invest time in producing a Gaussian Pyramid quickly.

Bells and Whistles

For Bells and Whistles, I plotted the frequency distribution of the I channel of the 2nd Laplacian Layer on the center pixel for this face image (probably somewhere around his nose). I plotted both the Original and filtered versions:
As you can see, the filtering works, and only values in those frequencies were amplified