CS 194-26 Final Project:

Image Quilting and Video Magnification

Leon Xu


Image Quilting


1. Randomly Sampled Textures

Using small base textures, we can randomly sample square patches and place these side-by-side to create a larger output image of the same texture. However, this method produces poor results: we can clearly see seams in the texture.

Original Textures:

Randomly quilted samples:


2. Overlapping Patches

If, rather than placing patches side-by-side, we allow them to overlap slightly; and additionally choose patches whose overlaps have low SSD; we can produce slightly better results.

The SSD error is restricted to be within around a percentage of the error of the error of the first overlapping patch, lower bounded by some reasonable minimum cost. If we are unable to find such a patch after many random samples, we can simply choose the beset match we have found so far.


3. Seam Finding

For further improvement on the algorithm: after finding a patch with low SSD error, we can try to find the nicest way to fit it in with its neighbors: that is, in the overlapping area between neighbors, we can find a good boundary where one side will take its pixels from the left neighbor, and the other will take its pixels from the right neighbor.

This can be accomplished efficiently through a min-cut algorithm, which finds the path from one side of the boundary to the other with minimum SSD in the pixels along the path. In the case where there is an overlap on the top as well as an overlap on the left, we can intersect the top-bottom and left-right min-cut paths.

Compared to the previous methods, this gives the best results. The water works particularly well.


4. Texture Transfer

Finally, we implement an algorithm to transfer a texture onto a target picture. This is done by adding to the SSD error term used to find a matching patch: we add the error between a correspondence map of the sample patch, and of the target at the location the patch will fill. This correspondence maps can be, for example, a brightness map of the images; so we give lower error to sample patches whose brightness matches closely with the target patch.

Target and texture source:

Correspondence maps:

Result:

We can clearly see an outline of Feynman, and the texture is reasonably similar to the source, but the result is not as good as achieved in the paper. This is likely partially just a problem with parameter tuning.

Another possible reason is that the Feynman image has much more extreme bright and dark areas compared to the texture source, which we can see in the correspondence maps. I tried to normalize the correspondence maps, but for some reason this produced worse results even though the correspondence maps still looked correct.


Bells & Whistles

I implemented my own version of the cut function for this project, as the provided code was in Matlab but I wrote my project in Python.


Eulerian Video Magnification


Overview

The idea of this project is to magnify temporal signals of certain frequencies in a video. For example, the human heartbeat, around 0.4 to 4 Hz, causes slight changes in skin coloration. If we can filter out these signals, and add an amplified version back to the original video, we can produce a video where the coloration changes are magnified and easily visible to the naked eye.

This can be accomplished surprisingly easily by considering each pixel location as a temporal signal; we essentially perform the filtering pixel-by-pixel, with some caveats for spatial frequency bands.


Process

The basic process for this project is as follows:

  1. Decompose the video into different spatial frequency bands using Laplacian pyramids for each frame
  2. For each pixel position, band-pass the temporal signal of the YIQ values to extract frequency bands of interest
  3. Multiply these signals by a magnification factor and add to original signals
  4. Reconstruct the video from the modified spatial pyramid


Results

The results look pretty decent. I just did the first 5 seconds for the sake of runtime. We can see coloration more clearly on the first video, and movement on the other two. I used the YIQ color space and multiplied Y/I/Q bandpassed outputs by 5/10/10 when reconstructing.

The first video more clearly shows heartbeat (and even breathing), while the other two show movement.



References

Image quilting paper
Video magnification paper