CS 194-26 Computational Photography

Image Quilting

Victor Vong, CS194-26-acq

Ian Lee, CS194-26-ael

Texture Synthesis Overview

Texture synthesis is the creation of a larger texture image from a small sample. With this technique we can generate larger pictures with the appearance of havin gthe same texture as an sample. Here, we present several approaches for texture synthesis.

Random Sample

First we sample patches from the source image and tile the patches onto the target. The results are pretty bad as the borders are obvious. On the left is the sample, and on the right is the result

bricks_small_random bricks_small_random

Overlapping Patches

Instead of just tiling the patches next to each other, we can try to overlap new patches with existing one. In order to construct a better image, we should pick patches that overlap well with the existing ones. To do so, we first randomly sample a bunch of patches, then for each patch, we calculate the SSD between the patch and the existing ones in the overlap region. We randomly pick one in with an error within 20% of the smallest difference. If we always pick the most similar one, we might end up with too many identical patches. The overlapping scheme I employed is one such that a new patch overlaps with existing patches to its left and right above it.

bricks_small_overlap

Seam Finding

Although the results from the above approach is satisfying for the bricks texture, if we use a different one, the results become terrible. The borders are more obvious for texture that are not square or rectangular shaped. Here we introduce seam finding to remove edge artifacts from the overlapping patches. The energy of every pixel in the overlapping region is the squared difference between the existing patch and the new patch. We want to look for a seam that has the lowest energy and we use dynamic programming to do so. (See Bells and Whistles.)

The results are presented here in the order of sample, random sampling, overlap sampling and seam cut sampling. The seam cut sampling result is very satisfying.

white_small white_small_overlap white_small_overlap white_small_overlap

More examples:

white_small white_text_overlap white_text_overlap white_small_overlap

white_small sketch._overlap white_small_overlap

white_small white_text_overlap white_text_overlap white_small_overlap

white_text_overlap white_text_overlap white_small_overlap

Texture Transfer

Now that we know how to do image quilting, we can modify it so that the output image uses the texture from the sample but retains the shape of an target image. I only had to modify the image quilting function's error function to include the difference between the sampled source patch and the target patch at the location to be filled multiplied by a scaling factor alpha. Below is the result of using the sketch as source image and Feynman as a target.

sketch sketch sketch

sketch sketch sketch sketch sketch

Bells & Whistles: Seam Cut

Here, I demonstrate how seam finding works. We first find the overlap region, then we make the SSD of the existing patch and the new patch the energy matrix. Then we use dynamic programming to identify the seam with the smallest energy. The first diagram thows the overlap region and the seam in red. The second diagram is the exisitng patches, and the third one is the next patch to be added.

cost cost cost


Seam Carving



Section I: Overview

In this project we implemented the algorithm in Seam Carving for Content-Aware Image Resizing by Shai Avidan and Ariel Shamir. Our version of the algoithm uses the Sobel(Img) as the energy function and from there we shrink an image (either horizontally or vertically) to a given dimension

Section II: Seam Carving

Seam carving is a content-aware method of resizing an image. Traditional rescaling techniques treat all parts of an image equally and therefore shrink the aspect ratio of the image resizing all objects in the image uniformly. However, there are certain parts of the image that we attribute more importance to and therefore do not want to shrink whatsoever. Seam carving makes the assumption that important sections of the image contain high amounts of energy (in our case vertical and horizontal edges), and tries to remove low-energy seams from the image thereby shrinking the image while mainintaing the original aspect ratio of important objects.

In order to shrink in image by 1 pixel in either the vertical or horizontal direction, we first compute the energy of each pixel in the image via the combined vertical and horizontal Sobel of the image. We then find the vertical/horizontal seam (a connected path from top to bottom or side to side) with the lowest total energy, and remove that seam from the image. This process is repeated until the desired size is met on both axes of the image.

Note: by vertically or horizontally, we meant removing vertical seams and horizontal seams


Original
Shrunk vertically by 150 pixels
Shrunk Horizontally by 150 pixels
Shrunk in both axes by 150 pixels
Original
Shrunk vertically by 200 pixels
Shrunk Horizontally by 200 pixels
Shrunk in both axes by 200 pixels

Original
Shrunk vertically by 50 pixels
Shrunk Horizontally by 50 pixels
Shrunk in both axes by 100 pixels
Original
Shrunk vertically by 50 pixels
Shrunk Horizontally by 50 pixels
Shrunk in both axes by 50 pixels

Original
Shrunk vertically by 50 pixels
Shrunk Horizontally by 50 pixels
Shrunk in both axes by 50 pixels
Original
Shrunk vertically by 100 pixels
Shrunk Horizontally by 100 pixels
Shrunk in both axes by 100 pixels

The above images did pretty well in maintaining important image properties when shrinking. Below are our biggest blunders. You can easily see how warped the important features (to us and not the algorithm) are warped almost beyond recognition :(.


Original
Shrunk vertically by 150 pixels
Shrunk Horizontally by 150 pixels
Shrunk in both axes by 150 pixels
Original
Shrunk vertically by 150 pixels
Shrunk Horizontally by 150 pixels
Shrunk in both axes by 150 pixels

Bells & Whistle: Seam insertion

Instead of removing the seam from the image, we can enlarge the image by inserting a seam at the path with the lowest energy. We do so inserting a pixel which is the average of the neighboring 2 pixels along the seam. If we recompute the seam every single time, it's likely the same seam will be chosen and thus stretched for the specif number of pixels. Instead we compute the non intersecting seams with the lowest energies, store those seams, then proceed to enlarge the image. Below aret he some results.


Original
enlarged in both axes by 40 pixels
Original
enlarged horizontally by 40 pixels
Original
enlarged horizontally by 40 pixels (failure)

Section IV: What We learned

This project was really amazing. When it was first showncase in class we were blown away and definitely knew we wanted to do this for the final project. Shrinking images this way definitely seems like it has better benefits provided the image is well suited for such a shrinking method. It definitely showcased a different way of thinking when it came to shrinking. Overall, we really enjoyed learning about seam carving and the methodology behind it (using an a good energy function is very important).

Fake Miniatures




Section I: Overview

In this project we created fake miniatures by simulating the effect of selective focus cameras, also known as Tilt Shift. In order to replicate this effect we select a focus plane by allowing the user to select a point (creating a horizontal line from that point) and applying a blurring filter to the rest of the image. Effectively this narrows the perceived depth of field in the scene (determined by the depth of field in pixels chosen by the user) and creates the illusion that the lens was really close to the subject causing the image to "seem" miniature.

Section II: Creating Fake Miniatures

The steps needed to create fake miniatures are as follows:

1. Set a focus line by drawing lines or selecting points in the image.

2. Define the size of the fake depth of field (DOF) with respect to the focus plane as several pixels below and above.

3. Increasingly blurr the image around the focus plane to simulate depth of field effect.

We chose our inital sigma value to be 1 and continually multiplied it by 1.2 for every region the size of the depth of field chosen.


Original
Miniature
Original
Miniature

Original
Miniature
Original
Miniature

Original
Miniature
Original
Miniature

Original
Miniature

These two photos are taken at newport beach and Disneyland. They are also our biggest failures. Without many objects to really showcase the focused objects they look rather mundane compared to the above images.


Original
Miniature
Original
Miniature

Section IV: What We learned

This project really emphasized a unique way to use blurring to achieve this effect. We thought blurring was rather mundane, but after this project it shed some light as to the effect blurring can have to the human eye and perception. Amazing.