For my final projects, I decided to work on two projects centered around combining the content and style of two source images to form a hybrid image. The first project uses CNN feature maps to achieve the task while the second uses patch sampling and quilting.

1. Reimplement: A Neural Algorithm of Artistic Style

For this project, I will reimplement the method detailed in the paper "A Neural Algorithm of Artistic Style"[https://arxiv.org/pdf/1508.06576.pdf]. The core idea here is that we are given two target images, one for matching content and one for matching style. We want to match the convolution activation maps of specific layers when a pretrained VGG19 model is run on both the content and result images and match the gram matrix of convolutional activation maps for the style and result images.

In order to implement this method, I created two custom nn.Module layers that act as an identity layer which caches a loss(either content or style) when a forward pass is run on the result image. I used a pretrained VGG19 model as outlined in the paper and added these special loss layers after the convolutional layers which were selected to match on style or content. These losses are initialized with a feature map or gram matrix representation of the content or style image we want to compare against. Finally, I froze the weights of all tensors except for the input(or result) image that we want to optimize and ran a training loop with the optim.LBFGS optimizer which has been documented to work well with this setup.

I mentioned on a private piazza post and will do so here again that I found an official PyTorch tutorial[https://pytorch.org/tutorials/advanced/neural_style_tutorial.html] which walks through a lot of this project. I comment where I may have taken inspiration from some of the tutorial's high level code design, like using a special loss layer to cache the loss when running a forward pass of the model, but I never referenced the tutorial in actually coding up my solution. In fact, I had originally used an adam optimizer and only changed my code after doing some more research and finding out that optim.LBFGS is better suited for this problem(something that was also mentioned in the tutorial).

1.1 Results

Here are some of my highlighted results:

1.2 Experiments and Tuning

In order to obtain results that looked good, I had to spend quite some time tuning parameters for the algorithm. I tried different values for iterations, alpha, beta, content_layers, and style_layers. My alpha/beta ratio ended up being around 1e-5 to 1e-7 in range. While this differs from the optimal values listed in the paper(1e-3 to 1e-4), I also did implement some of the losses differently in not dividing by a normalizing constant(like number of layers that contribute to loss). Thus when taking the different implementation into account, my alpha/beta values are not too far off from the paper results.

I also found tuning alpha and beta to be the most difficult part of this project, so I separated the content and style losses and graphed them for every experiment in order to figure out how I could improve my alpha and beta values. Here is an example of the loss graphs for a single run of the algorithm.

More information about my experimentation can be seen in my jupyter notebook code submission.

2. Image Quilting

In this project, I will implement the image quilting algorithm described in the paper "Image Quilting for Texture Synthesis and Transfer"[http://www.eecs.berkeley.edu/Research/Projects/CS/vision/papers/efros-siggraph01.pdf].

2.1. Randomly Sampled Texture

The most naive method is to simply randomly sample patches from the source image into a grid pattern. This method was fairly simple to implement and here are some of my source images and texture results:

2.2. Overlapping Patches

The next method I implemented was overlapping patches. I utilized two helper functions as suggested by the project spec:

1. ssd_patch returns a cost image where every pixel is the cost of selecting a patch centered at that pixel

2. choose_sample takes in a cost image and returns a selected sample patch from the source image given a tol threshold which determines how much we care about selecting a low cost patch

As a sanity check, I confirmed my code works by running the algorithm with tol=1 to ensure that a partial copy of the source image is formed:

Here are some of my results for overlapping patch textures with a tolerance of 10-100(randomly selecting one of the "tol" lowest cost patches):

2.3. Seam Finding

For seam finding, we were provided with a cut.m helper function to find the min-cost contiguous path of a cost image. Since this helper was written in matlab and I used python for this project, I utilized the cut.py file provided from https://yxw.cs.illinois.edu/course/CS445/F21/projects/quilting/ComputationalPhotography_ProjectQuilting.html which appears to be the current semester's project spec for this same project.

Here are some figures to demonstrate the seam finding process:

For the full quilting algorithm using seam finding, the overall algorithm remains the same as that of overlapping patches, except that now for every overlapping section we run seam finding to find the min-cost path and combine the two overlapping patches using this path as shown above. Depending on the patch location in the resulting image, this could mean 0, 1, or 2 overlapping regions. For 2 overlapping regions(top and left), we simply combine the two paths at where they intersect on the corner to get the full seam.

Here are some results from seam finding texture synthesis:

2.4. Comparison of random, overlapping, and seam-finding

Here, I compare the results of the random, overlapping, and seam-finding methods for generating textures. We can see that the generated textures improve from random→overlap→seam-finding which is expected. The overlap method is able to find better suited patches than the random method, but lacks the ability to form a nice seam along the border that the seam-finding method can achieve.

2.5. Texture Transfer

Finally, I implement the full texture transfer algorithm. This algorithm is largely the same as the seam-finding method, except now the cost term is comprised of two parts: the same overlap cost as before along with a new content cost that is determined by the difference in grayscale pixel values of the texture and content patches. These two cost components are summed with a weighting of $\alpha$ and $(1-\alpha)$ respectively.

Some of my results are displayed below. I played around with adjusting the patch_size, overlap_size, alpha, and image sizes in order to create better looking results. I also used a blurred grayscale version of the texture and target images for the content cost calculations which improved my results.

Here are some of my results for texture transfer:

2.6 Bells & Whistles: Disguised Toast on a Toast

Disguised Toast is a popular streamer. I'm going to put his face on a toast for my bells & whistles of this project. I first created a toast textured image of Disguised Toast's face, then reused some code from project 2 to perform Laplacian pyramid blending between the toast-textured face image and the full toast image using a binary mask. Here are my results: