CS 194-26 Spring 2020

Final Project: Image Quilting

Alex Shiyu Liu, cs194-26-afw

Overview

This project explores the image quilting algorithm for texture synthesis and transfer as described in the SIGGRAPH 2001 paper by Efros and Freeman.

Randomly Sampled Texture

In the first part, I randomly sampled axis-aligned squares of the original flecktarn camo image into the output image in order of left to right and top to bottom to create a basic sythetic texture:

Overlapping Patches

In this part, I randomly sample the first, top right tile for the output image. The remaining tiles overlap the previous tile and above tile by the number of pixels set as a parameter. Each subsequent sampled tile is chosen by minimizing the sum of squared distances of pixel values between in the overlap area of the already sampled tile and every possible tile in the original image. The set of smallest SSD tiles within minc*(1+tol), where minc is the minimum SSD and tol is a set parameter, is randomly sampled as the next tile in the output image. This stochastic process helps with creating more natural textures. The result below on the flecktarn camo image presents a relatively effective synthetic texture:

Seam Finding

In this step of synthesizing textures, seam finding is implemented to remove the edge artifacts between sampled tiles. The seam is a minimum cost path from one side of the overlap section to the other. The minimum cost is based on the sum of squared distances between the two adjacent tiles, computed in the previous section. Using a dynamic programming algorithm for finding this minimum cost seam, a mask can be created and applied to the tiles in order to remove edge artifacts and create a smoother texture. The flecktarn camo image and each texture sythesizing process, ending with the seam finding image, is below:

A few more examples of textures and their seam finding synthetic counterparts are below:





Texture Transfer

In this section, the image quilting algorithm with seam finding is modified to create texture samples with underlying images. The difference in this modified algorithm is that an additional cost term is added to the original SSD in the overlap areas. For every potential sample patch, the SSD in the overlap area against the already placed texture is computed and added with the SSD of the difference in pixel values between the potential sample patch and the underlying image in the area that the potential sample patch would be mimicing. The parameter alpha, where alpha and (1-alpha) are multiplied to one of each of the two SSD terms, is utilized to determine the weight on emphasizing either accurate image recreation or original texture quality. The result of the flecktarn camo image being synthesized to an image of a doberman is below:

Another example is below in which I synthesize the toast image to depict my friend's face:

Bells & Whistles

The original spec gives a MATLAB file for seam finding through dynamic programming. I rewrote a python function for seam finding using dynamic programming which can be found in the code submission.

Conclusion

I learned a lot about image quilting and some of Prof. Efros work. This project gave me experience with interesting methods for texture synthesis and transfer. I was able to gain more experience with using mathematical methods like SSD and dynamic programming for manipulating images.





CS 194-26 Spring 2020

Final Project: Seam Carving

Alex Shiyu Liu, cs194-26-afw

Overview

This project explores seam carving as described in this SIGGRAPH paper by Avidan and Shamir for smart image cropping.

Method

This algorithm shrinks images by first determining the "importance" of each pixel using an energy function. Then until the image has shrunk to the desired dimension: 1. Find the lowest-importance seam in the image. 2. Remove it. The seam carving is implemented by using dynamic programming to find the minimum energy seam from one side of the image to the other. For resized parameters where both the rows and columns are trimmed, I alternated removing rows and columns until one of the two dimensions reaches the final value and then the other dimension is trimmed until the final shape is created.

Energy Functions

I implemented three different energy functions which all had effective results on specific images. The first energy function I used was the sum of the absolute value of gradients, i.e. f(I) = |d/dx(I)| + |d/dy(I)| where I is the image matrix and operations are performed elementwise. This energy function was a good starting point since the gradient values offer some measure of the importance of pixels. The second energy function I implemented was the h values from the Harris Corner Detector algorithm that derived from the eigenvalues of matrices with gradient values of pixels as entries. The third energy function utilized sum of gradients and entropy. This function is g(I) = |d/dx(I)| + |d/dy(I)| + entropy(I) where entropy() is a function of entropy over a 9x9 pixel window. I found this function to be more successful in some of the examples since the entropy over the 9x9 window considers more pixels around a given pixel when determining energy.

Results

House image resized from (512 x 384) to (432 x 344) using the harris energy function:


The image below was a particularly interesting example of this image resizing method. Note that the foliage around the borders was removed along with the uniform water in the middle while the buildings and boats were almost untouched. Japan image resized from (1334 x 750) to (734 x 650) using the harris energy function:


The image below depicts a successfully compressed picture of a New York street where the buildings and cars are all intact, but smaller. However, the energy function was not robust in that the resulting image had curvature in originally straight lines of the buildings. New York image resized from (348 x 348) to (248 x 298) using the harris energy function:


The image was a result of the algorithm that removed a lot of the cloud area, but kept significant patterns and characteristics of the clouds. Utah image resized from (1334 x 750) to (934 x 700) using the entropy energy function:


The image below was a successful example of the algorithm in removing some of the uniform sky structure and compressing the building areas. Los Angeles image resized from (488 x 867) to (413 x 567) using the gradient energy function:


The image below was a relatively successful resizing, but the individuals in the image were slightly altered. Dan image resized from (600 x 900) to (525 x 750) using the entropy energy function:


Failures

This example shows highly altered individuals which is not intended since they are the subjects of the image. This likely occured since the entire image is highly contrasted between pixels, so the energy function was unable to effectiviely assign high values to important pixels. Dan image resized from (600 x 900) to (420 x 600) using the harris energy function:


This image shows a failure case in which the individuals arm is mistakenly removed. Jens image resized from (333 x 500) to (333 x 200) using the harris energy function:



Bells & Whistles

I implemented three different energy functions (gradient, harris, and entropy) in an attempt to better measure energy for more effective image resizing.

Conclusion

I learned a lot about seam carving for smart image resizing. I was able to apply mathematical methods like gradients and dynamic programming to resize image while keeping important artifacts intact. The use of energy functions allowed me to weigh pixels in order to preserve specific aspects of images.