CS 194-26 Final Project

Calvin Tang | calvintang@berkeley.edu | 3031974882

Project 1: Seam Carving

The first project I decided to do in the whole final project is Seam Carving. I've been told that this has been covered in some semesters of CS 61B, but since I didn't do it my semester, I wanted to take a look at it. The process of seam carving is used in content-aware image resizing, such as the resizing of web pages or viewing different images on different screen sizes. The concept of seam carving is simple and quite elegant. After reading the paper and following this tutorial, I was able to understand the process of seam carving and apply it to my own images.

Energy Map

The metric for determining which pixels are to be deleted is defined as the energy of a pixel. There are various "energy functions" that can be used to determine the energy of individual pixels. For the sake of this project, I have chosen the simple energy function as provided in the paper. Here it is below:

Similar to Project 2, we find the partial derivatives of the image. There are various ways to do this. In this project, I experimented with both Sobel filters and Scharr filters, shown below.

By applying these filters to the image, we're able to generate an energy map of the input image. Some examples are shown below, with the Sobel version on the left and the Scharr version on the right. As we can see, they're very similar. I wanted to experiement on the impact they would have on the result images.

Seam Selection

The next step of the algorithm is to remove the minimum energy "seams" of the image in increasing order until the desired image size is achieved. In the paper, the best solution to this portion of the algorithm is dynamic programming. Below is the idea for the subproblem of dynamic programming:

The general idea is that we want to populate a 2-dimensional array with the same size of the input image that keeps track of the minimum energy seam thus far. As we can see in the above subproblem, we are taking the energy at the current pixel and finding the minimum energy in a neighboring pixel. By doing this, we can find seams across the image of certain energy levels and determine which seams will need to be removed and in what order. It's important to note that there needs to be special cases for the edges of the image, where i = 0 or j = 0. To account for this, you can see a special j = 0 case in my loops.

Since we are using DP and it involves a double for loop across all pixels of the image, this can be a lengthy process! This portion of the code took the longest to run, since it does most of the processing on the image itself. I took a tip at the bottom of the tutorial to utilize Numba to convert Python code into machine code such that it speeds up this portion of my code, improving the runtime drastically.

Seam Removal

After the process of seam selection, we are ready to remove seams from the image until the desired output size is reached. We are able to find the seam of minimum energy in the image from the last portion of our algorithm. From there, we select the pixels in that seam, remove the pixels, then move on to the next minimum seam in the image. We repeat this process until the output image reaches the desired size.

Results

Below are some of the results of my output, with both vertical and horizontal seam carving. Most of my output were generated using the Sobel filters. Using Scharr filters did not make a large difference in the output image (especially since the energy maps were so similar to Sobel's), so I decided to stick with my original results of using Sobel. Each of these images have been reduced to 50% of their original height or width. This parameter can be adjusted in my code.

City

Image credit to TA Shivam Parikh.
Top: Original image
Left: Sobel output
Right: Scharr output

Yosemite

Lake

Credit to Pietro De Grande on Unsplash.

Cannon Beach

Project 2: Image Quilting and Texutre Transfer

I decided to do image quilting for my second project. I was interested in implementing it after hearing Prof. Efros talk about it in class. Also, part of the process of image quilting involves seam carving, so I thought it'd be a good idea to carry over my code from my first project. Image quilting is the process of sampling patches from an input image and generating images of varying sizes using the textures from the input image. There are few methods discussed below with various degrees of success in each of them.

Randomly Sampled Texture

This was the simplest algorithm to implement. This function takes in an input image, a desired output shape, and patch size parameter. The algorithm randomly samples a patch of size patchsize x patchsize and places it on the image. This algorithm will fill up an output image and return a randomly sampled texture image.

Of course, we expect to see very messy results. There is no metric for successful blending here, as we're randomly selecting patches. Even when passing in a textured image, we see a lot of harsh lines and can easily tell where patches are in the image. Below are some results of the algorithm, with the input image on the left and the output image on the right.

Overlapping Textures

Now we begin putting in efforts to produce more seamless results. We do this by allowing a certain amount of overlap between patches. Doing this, we can take the overlap sections of two patches and compute the loss between them. Instead of looping through the image and incrementing each step by patchsize, we step by (patchsize - overlap), where overlap is an integer value that we want patches to overlap by. It is important to note that there are three cases for overlapping: overlapping on the left, overlapping on the top, and overlapping on both the left and top. Because of these different cases, we must be aware of the case at hand and compute loss on the overlap section(s) of interest accordingly.

The loss function I used was a simple SSD function. To reduce repetitiveness, we include a tolerance variable. I often found tol = 1.1 worked as a good value. When computing the overlap loss from our desired patch to all other patches from the texture image, I take the minimum loss patch and multiply its loss by tol. I then gather all patches that have a loss less than or equal to that computed loss value, where each patch is somewhat "close enough" of a neighbor to the patch in question. From there, I randomly select one of my patches in the group, meaning it may not be the patch with the lowest cost.

Below are some of the results of my algorithm. Still, we see that it is not perfect, but a huge improvement from randomly sampling. The algorithm found it particularly difficult to find optimal patches that overlap on both the left and top side of the patch. Nevertheless, I found the results to be satisfying.

For this output, I experimented with different output sizes. The image in the center has a dimension of 200x200px and the image on the right has a dimension of 500x500px.

Seam Carved Textures

To improve the image quilting algorithm even further, we utilize seam carving. The algorithm for this portion of the project works very similar to the previous one. However, there is one key difference. In the overlap sections of patches, we detect the minimum cost seam (as defined in Project 1 above). Anything to the left or top of that seam (depending on the case of the overlap situation), we replace with the pixels from the existing output image. In this way, we want to ensure that the blend is as seamless as possible and works with the pixels that have already been placed on our output image. Below is an example of a seam found in a patch and pixel replacement with another random patch.

This part in particular was quite difficult for me to understand. Luckily, I could use my code from the last section to find the minimum seam of an overlap section easily. After that, it was a matter of getting all coordinates of seam pixels in the overlap and then used creative slicing to replace the pixels in the patch with those of the existing output image.

My results are below, where the input image is on the left and the output texture image is on the left. There were some hiccups here and there, especially with the middle case of overlapping on both the left and top of a patch. I'm still pleased with the results of some of my textures and still an improvement in my eyes. I'm proud of myself for understanding this portion of the project and implementing it.

Texture Transfer

Finally, we move into texture transferring. This is a process in which we take the texture from an image and overlay it onto another image. This gives the appearance that the target photo takes on the texture of the input image, which produces a pretty cool effect! Luckily, this algorithm behaves very similarly to the previous seam carving section just discussed. The only difference in the algorithm is that we compare the selected patch to the corresponding location in the target image to check how similar they are to one another. Adding this metric to the loss function of selecting optimal patches allows the algorithm to replicate the look of the target image using the textures solely from the input image.

An extra parameter alpha is also passed into the functon. This parameter functions similarly to the image blending project from earlier in the semester, but it is only used when finding the best patch to put into the output image. This alpha parameter, which is a float value between 0 and 1, determines how much weight is put on the SSD of the overlap sections of patches and how much weight is put on the SSD between the patch and the destination patch on the target image.

Unfortunately, I was not able to solve my issue with this portion of the code. I understand the algorithm and believe it was coded fine, but my program just stops running after a certain row. Please do read into my code to take a peek at my understanding. Below are my two input images, my output image, and the desired output (from the paper).

Conclusion

These projects were a lot more fun than I anticipated. I'm glad we had a large variety of projects to choose from and I'm glad I chose the ones I did. Thank you for an enjoyable class!