Project 3: Fun with Frequencies and Gradients!

In this project, we aimed to create hybrid images. We explored different ways of doing so, most namely in the frequency domain, and the gradient domain. In the frequency domain, we used Gaussian and Laplacian stacks to blend together multiresolition images. In the gradient domain, we used Poisson Blending in order to insert objects from a source image into a target image.

---

Part 1.1: Warmup

For his part, we used the unsharp masking technique in order to to sharpen images. I first obtained the low frequency portions of an image by convoluting it with a Gaussian filter, and then subtracted the low frequency portions from the normal image to obtain the high frequency portions. Then, I multiplied the high frequency portions by a constant alpha (> 1) to sharpen images, and added the low frequencies back to the accentuated high frequencies to obtain the resulting image.

Note the details in the black splotch of fur on his head get more detailed. The wall patterns become sharper as well.

---

Part 1.2: Hybrid Pictures

We then moved on to creating hybrid images, which look different at different distances. To create them, you can blend the high frequency portions of an image with the low frequency portions of another. The high frequency images will be dominant at close range, and the low frequency portions will be dominant at a distance. This creates the effect of two different images being "blended together" in one.

Part 1.2 Frequency Domain Analysis

Here are a few pictures of the frequency domain version of these photos. As you can see, there are portions of Ollicat in the merged spectrum (the well-defined + shape), and there are portions of ScaredCat in the spectrum (the diagonal line corresponding to y = x), not spread out. Note that there are some features of OlliCat that didn't make it into the final picture, the high frequency parts (the line corresponding to y = -x)

Part 1.2 Bells and Whistles

Using the full color channel for both low and high resolution images works the best. Here is the colored technique on a cartoon image, which blends in suboptimally if either channel is grayscale.

---

Part 1.3 Gaussian and Laplacian Stacks

For this portion of the project, we repeatedly extracted high frequency and low frequency components of a picture, creating two stacks which contain information on different parts of the frequency spectrum. This led to some interesting observations on unique pictures, such as Salvador Dali's painting Gala Contemplating the Mediterranean Sea which at a distance of 20 meters is transformed into the portrait of Abraham Lincoln.

Let's take a look at the gaussian and laplacian stacks for the hybrid image that we generated. In the gaussians, it is clear that Happy Cat Ollie dominates the lower frequencies. In the laplacians, it is clear that Scared Cat is the dominant image.

---

Part 1.4 Multiresolution Blending

Now that we have generated gaussian and laplacian stacks, we can use them to perform multiresolution blending. We first mask both images that we wish to blend together, then add both images together. This creates a very rough edge between the two photos. Then, we take this rough image, and generate a gaussian stack off of it. Then, we generate a laplacian stack for both of the source images. We use the values of the gaussian stack to weight the pixel values of the laplacian stacks. The combination of all of blended laplacian stacks and the last image in the gaussian stack results in our blended hybrid photo.

Part 1.4 Bells and Whistles

Using the full color channel, effects are enhanced.

Here, we display the example laplacians for the sun file.

---

Part 2.1: Toy Problem

Now, instead of working in the frequency domain, we can work in the gradient domain. Humans are sensitive to changes in images, thus if we take these "gradient" differences between pixels in an image and translate them to another image, we can effectively create hybrid images this way. We can frame this problem as a least squares optimization problem where we solve for each pixel to be placed in the target image.

We start off with a toy problem: reconstructing an image from its own gradient.

Part 2.1: Poisson Blending

Now, we employ this strategy in order to blend together portions of images. From the source image, we put a mask over the object to be inserted and make a "hole" in the target image. Then, we set up the optimization problem by adding constraints on all the vertical and horizontal gradients in the image. Then, we add additional constraints on the boundaries of the object to be inserted, forcing the difference between the inserted image and the normal image to be the least amount possible. Finally, we solve this sparse least squares problem and fill the solved into the target image.

Here is an example of one that did not work too well. The squirrel did not work well because the gradient in the background of the squirrel was too close to the squirrel color, thus the squirrel became "blue", as the optimization problem changed it to be ~ the same color the background took on.

Here, Poisson blending performs better than Multiresolution blending. Although it was not optimal for the same reason as the blue squirrel, (the base of the campanille has varying gradients, from grey granite to green foliage). It performs remarkably better because the gradients of the hoover tower is preserved, whereas in multiresolution blending the image of the Hoover tower is faint. There is no good way to balance the seam between the two images and the strength of the Hoover Tower.

Here, the jagged lines for the multiresolution blending isn't great. However, thomas the tank engine's facial colors are preserved, and the yellow and blue of the train don't leak into the face, which is good. However, we sacrifice the smoothness of the gradient.

For the Poisson blending, the seams are much better. However, we trade off a bit of color Thomas's face. This is because the intensity difference for thomas's face wasn't preserved, rather the gradients.

The most important thing that I learned about the project was that sparse least squares can be optimized a lot using lsqr and csr matrices. Also, I learned that higher order functions were useful, because I had to use one to implement my im2vec function. Interestingly enough, gradient domain seems to be more effective for generating hybrid images than frequency domain, which aligns with the observation that the human eye is more sensitive to immediate gradients than other subtle features like high and low frequencies.

Random Stuff