Jeffrey Shen Final Projects

These are my submissions for CS 194-26 Final Projects.

Final Project 1: Image Quilting

In this project we want to generate a larger image from a small sample using the sample to extrapolate the texture from the image. We will try different methods for creating this image including using random patches, using SSD to match patches, and using a cut to match up patches along edges. We can then apply these methods to create a texture transfer from to use one image and transfer texture from a texture sample to create a final image.

Randomly Sampled Texture

As an initial method for generating an image quilt from a sample, we can find patches randomly from the sample image and combine them together to create the final image. This involves placing down the patches orderly from the top left hand side and randomly choose patches from the sample. Here are some examples of the generated images with sample size of 200, patch size of 50 and output size of 400.





Overlapping Patches

An improved method of creating the image quilt is through using overlapping patches and finding similar patches on the overlapping portion through SSD. We can setup the patches to overlap a certain amount, and take the overlap portion, which can be the top edge, left edge, or both and find the best match against all the possible patches in the sample. Then we can choose either the patch below a certain tolerance or between the lowest ssd scores. We are then able to build out the image quilt using this algorithm to match create a quilt that should have continuity. Here are some examples with a sample size of 200, output size of 400, patch size of 51, and overlap of 25. We will choose the patch that matches with the lowest ssd score.







Seam Finding

This method of seam finding is based from the research paper "Image Quilting for Texture Synthesis and Transfer" by Efros and Freeman. Instead of copying the patches over as blocks, we will create cuts along the line that creates the most clean combination of two different patches. In this method we will use the previous overlapping method and create a path that results in the best cut by cutting along the path with the least error. We will do this by generating an error of the overlapping region by pixel and then use a cut method to find the shortest path from one side to the other which will be the slice that we fit the new patch onto the previous image. For each patch, given the sample patch to match and the current patch,
Current patch, sample patch

We want to first find the ssd difference between the matches and run the cut algorithm with the pixelwise ssd difference between the matches to get a cut mask of the path with the least cost.

Finally, we will combine the images using the left side of the mask and filling in the current patch and filling in the right side of the mask with the sample patch.

We will then fill these in as the patches, and be able to generate the image with seams. Here are some samples below using the seam finding algorithm.









Texture Transfer

Finally, we can apply the quilting with seam finding to do texture transfer. Texture transfer involves getting a content image and a texture image and creating an output image with the content image having the texture of the texture image. This is done by changing the error function whereas previously we used SSD to calculate the error between two patches, we now want to calculate the error between the difference in content as well as the difference between texture of the original image and the texture image. We then combine these terms and use an alpha constant to adjust how much we want the texture of the image to influence the final image.





Bells and Whistles

Iterative Texture Transfer

In iterative Texture Transfer, we will also match the patch against what was matched the previous iteration where each iteration uses a different alpha value. This is able to smooth out some of the textures and create a better texture transfer that is less splotchy. Where the non-iterative approach is on the left and iterative approach is the right image.



Final Project 2: Neural Net Style Transfer

In the Neural Net Style Transfer Project, we want to train a neural net to apply a certain art style to an image. We want to be able to transfer the art style to different images while still maintaining the same content. This involves separating the content from the style of the images. We can do this by training a neural net to minimize the style loss and content loss. This project is based off the research paper "A Neural Algorithm of Artistic Style" from Gatys, Ecker, and Bethge.

DataLoader

We first want to standarize the data and we create a DataSet to manipulate the original image by rescaling the image to size 256 and converting the image to a tensor.

Convolutional Neural Net

For the Style Transfer Neural Net, we use the VGG-Network which excels at visual object recognition. I used the VGG 19 network to train on minimizing the style and content loss in order to generate an image with similar style to the style image and similar content to the content image. We also use 5 levels of average pooling to improve our gradient calculations. We use a pretrained network to save computation and setup the model to calculate the conten and style loss at different levels in the neural net.

Loss Function and Optimizer

The content loss is calculated at the 4th convolutional layer. This is because the higher convolutional layers are able to combine context from the previous convolutional layers and is able to better understand the content and object in an image, this allows us to be able to better understand the image and compare the content between images. For the style loss, we will compare at all convolutional layers to ensure our style in consistent across all details of the image as well as the overall image. In order to compute the style loss, we will use the gram matrix which will be computed from the feature map multiplied by a transpose of the feature maps. By applying this loss across all layers, we ensure the neural net is minimizing the style loss at all layers. We use MSE loss as the loss function for comparing the style loss and content loss between the images. I was able to train the network using 35 epochs and toward the end, it was converging on a loss and decreasing slower than previous iterations. Here is the Training total loss graph.

Results

The network was able to carry over the style well for most pictures, however sometimes if the images colors did not match up exactly with the art, it was unable to come up with colors. Some of the results were very interesting.



For this image, it was unable to get the details in the face, possibly because the original art image has one shade of color for each section or the general color of the face was similar and decided it should be one shade.

One interesting example I wanted to try was using one of my pastel art pieces of a road in SF and a picture of the same road in SF in a similar picture to see how similar the image would look to the art piece.

Although some of the final image was washing out with colors, the trees look surprisingly very similar and the car was converted very well.