Final Project

by Tiantian Cao (aej) & Haoming Guo (agh)

Depth Refocusing


The objects which are far away from the camera do not vary their position significantly when the camera moves around while keeping the optical axis direction unchanged. The nearby objects, on the other hand, vary their position significantly across images. Averaging all the images in the grid without any shifting will produce an image which is sharp around the far-away objects but blurry around the nearby ones. Similarly, shifting the images 'appropriately' and then averaging allows one to focus on object at different depths. For every image taken by the camera in position (u, v), shift the image by c * ((8-u), (v-8)). This shift the image towards/away from the central image produced by the camera in position (8,8). Then we average all the images at once.

Aperture Adjustment


Averaging a large number of images sampled over the grid perpendicular to the optical axis mimics a camera with a much larger aperture. Using fewer images results in an image that mimics a smaller aperture. We set parameter r as the radius of the aperture. We include all images with position (u,v) if the distance between (u,v) and the center (8,8) is less than r.

I. LightField Camera

Overview


Using lightfield data, we can achieve complex effects with simple shifting and averaging operations. We use data from the Stanford Light Field Archive. Each dataset contains 289 images taken with a 17x17 grid of cameras. In particular, we implement Depth Refocusing and Aperture Adjustment.

Result


Here is a gif that includes refocused images with parameter c ranging from -1 to 3 with an interval of 0.5.

Result


Here are some results with different r.

Summary


We learned a lot about the concepts and machnisms of light field with only a few lines of code.

r = 1

r = 2

r = 4

r = 8

II. Style Transfer

Overview


We reimplement neural style transfer discussed in the paper "A Neural Algorithm of Artistic Style", where we take in a content image and a style image and output a blended image painted in the style of the style image but still keep the content of the content image.

Result


Here are the content, style and final images.

III. Image Quilting

Overview


The goal of texture synthesis is to create a larger texture sample from a pre-existing smaller sample without visible seams. Texture transfer maps the texture of one sample onto another object while preserving that object's appearance.

Random Sampled Texture & Overlapping Patches


We implemented two baseline algorithm. The first one randomly selects a patch and uses it to map the whole output picture. The second one uses SSD cost to select one of the best k patches at each step. K is set to 10 here. Alternatively one can implement a version to select one of the patches with score above a threshold. It involves a variable ‘overlap’ to be set. I set the variable as the paper suggests to be about 1/6 of the patchsize. We can see the second methods works much better than the first one.


Seam Finding


We implemented cut function (Bells & Whistles) and seam finding function to choose the best path to split two patches. The seam finding takes the minimum cost path given an error surface. Then, a corresponding quilting is used to produce the texture synthesis. This also significantly improves the result. We can see that the edges are very hard to observe, except in few places.


Texture Transfer


We implemented texture transfer that combines previous method with one new correspondence error with the target image to transfer the texture to the target image. Here, a weighted sum of the synthesis error and the correspondence error is used, and the weight is manually set according to preference over which to prioritize. To further improve the result, one can implement an iterative method described in the paper.



Method


We first formulate a loss function that matches the content and style of each respective image in the feature space of a deep network, which is a weighted sum of three terms: content loss + style loss + total variation loss. Then we perform gradient descent on the pixels of the image itself. Different from the paper, the deep network we use as a feature extractor is SqueezeNet, a small model that has been trained on ImageNet. We chose SqueezeNet for its small size and efficiency.

Hyperparameters: initial_lr = 3.0, decayed_lr = 0.1, decay_lr_at = 180, iteration = 200.

Content

Style

Result

Here are some of our own images.

This one fails and we think that the reason is that the color and lightness of the content is similar to those of starry night, so the result is not as good as the two before.

Result


Here are the results of random sampled texture, overlapping patches and seam finding.

Original

random sample texture

overlapping patches

seam finding

Here are two other results of seam finding. The first one is good, but the second one fails as you can see there are still visible borders.

Original

Texture

Result