CS194-26 | FINAL PROJECTS
(1) Fake Miniatures (Simulated Tilt-Shifting)
(2) Eulerian Video Magnification of Subtle Signals
Allen Zeng, CS194-26-aec

(1) Fake Miniatures (Simulated Tilt-Shifting)

In this section, I simulate the tilt-shift effect on plain photographs taken with a small aperture. Meaning, the input image of a scene is clear and not blurry, and then the algorithm converts the image to look like a photograph of a miniature scene. Overall the algorithm consists of two parts: increasing saturation, and increasingly blur the image away form the object in focus.

Increasing saturation is relatively simple. First, the image is converted to HSV space, then the saturation values are boosted by an arbitrary value (25%) and clipped so the pixels have a valid pixel intensity. The saturation-boosted image is converted back to RGB space for further processing. Increasing the saturation of the image helps the illusion that the output image is of a miniature scene.

In a miniature scene, often only a part of the scene is in focus. As we move away from the focus in the scene, the rest of the image becomes increasingly blurry. Additionally, note that for a majority of the cases, miniature scenes are of landscapes or otherwise horizontally oriented environments. E.g. miniature football stadiums or miniature trees in a forest. With that assumption, the user selects a point in the image to focus on. The image is then split vertically into horizontal bars. I chose to have ten bars in my processing. The idea is to blur each of those ten bars increasingly as they are increasingly distanced from the user's click point, then blend all of the bars back together to produce the final variably blurred output. In practice, I produced a ten-level gaussian stack, and each of the horizontal segments served as masks. Then, I linearly blended the original image with each of the gaussian levels, based on the distance from the user's click point, using the horizontal masks.

Combining those two techniques, I produced the following results. The simulated effect works well and has visually-pleasing results. It works best in landscapes, so the Westfield Mall photo is not as believable as a miniature.

Source Images	Tilt-Shifted Outputs
Construction 1 pexels.com, CC0
Construction 2 pexels.com, CC0
Graduation
Westfield Mall, San Francisco
Union Square, San Francisco

BELLS & WHISTLES	Miniature Video
Construction Site pexels.com, CC0

(2) Eulerian Video Magnification of Subtle Signals

It is possible to boost subtle signals in a video, so that the human viewer can more easily extract relevant information. By boosting the subtle signals, we can answer questions such as: "Is the baby breathing?" "Is the suspect fidgeting?" "What is the heartrate of this person?"
Those questions are answered in the videos Baby 2, Face 1, and Wrist. Additional videos with subtle signals are available below as well. This project is based on the paper Eulerian Video Magnification for Revealing Subtle Changes in the World by Wu et al. The sample videos used here are available here.

The overview of the algorithm is as follows: We apply a gaussian filter to all frames of a video. This is so that each pixel contains information of its neighboring region, and that we can capture subtle movements of the video's subject. The gaussian filter is applied an empirical number of times so that high frequency noise is removed. And then the frame is subtracted from an even more blurred image of itself to produce the laplacian of the frame. The laplacian of the frame contains a subset range of the frequencies in the original frame, ideally it should also contain the signal we want to magnify. We then operate temporally on the processed video. A temporal band-pass filter is applied to remove extraneous noise and movements. The processed video is then scaled and added back to the original video, thus the subtle signal is magnified in the output video.

My results were fairly successful, following the parameters as specified in the paper. I used the same cutoff frequencies as specified, and for most videos I used a higher amplification factor to produce more visually apparent results.

My implementation resulted in fairly noisy videos, compared to the videos featured on the paper's webpage. I suspect the noise in my videos is due to my temporal bandpass filter being a box filter on the frequency domain. The sharp removal of extraneous frequencies likely produced artifacts in the signals, resulting in the colorful blotches shown in the results. To augment the output noise, different bandpass filter types could be used.

Original Videos	Signal Magnified Videos
Baby 2	Baby 2 Magnified
Face 1	Face 1 Magnified

BELLS & WHISTLES	Other Videos, Magnified
Baby 1	Baby 1 Magnified
Face 2	Face 2 Magnified
Guitar Low E String Magnified Notice the topmost string	Guitar A String Magnified Notice the second string form the top
Subway	Subway Magnified
Shadow	Shadow Magnified
Wrist	Wrist Magnified Notice the pulse on the wrist