Image Quilting is the idea of taking a single small image of a texture and extrapolating that texture to create a larger image. One can think of it as expanding the texture of an image. Below, we will see examples of image quilting.
We begin with a naive approach where we randomly sample square patches from our image to expand our texture. If we are building out an image, we randomly sample what the next patch may be. Examples for random sampling are seen below.
|
|
|
|
|
|
|
|
Now we will make our quilting better by sampling patches to overlap with existing ones. This way, when we expand our texture we will join together pixels that are in the same neighborhood and thus smoother.
|
|
|
|
|
|
|
We can already see a major improvement!
The last step is to incorporate seam finding. The problem before with finding overlaps is that sometimes the overlap wasn't always uniform across the image. But with seam finding, we can take these overlapping regions and "split them at the seams" to build out our texture with the patches that directly match!
|
|
|
|
|
|
|
We can see the final results for all the textures
|
|
|
|
|
|
|
|
|
|
|
|
The last idea that we can accomplish is that of texture transfer. The idea is to take two images and apply the texture of one onto the shape of the other. We do this using our quilting process from before but we add additional cost from the second image that we are grabbing the shape from. For example,
|
|
|
We also see some failure cases
|
|
|
|
|
|
|
To improve our texture transfer, we will implement the iterative texture transfer approach from the paper where we modify the error in overlapping regions iteratively so that we start with large blocks then progress to smaller blocks.
|
|
|
|
|
|
|
|
|
|
|
The lightfield represents every ray of light in every direction in space. To make a lightfield camera, we need a way to capture all the different amounts of light at all different positions. Naively, we can make a grid of cameras, each capturing a different set of light. Then we have this grid of cameras take a picture at the same time and there we have our rudimentary light field photograph. The only problem is that our lightfield is split across many images, not just one. In this part of the project, we will see how we can mimic depth refocusing and aperture adjustment using this grid of cameras.
Since our grid of cameras involve different cameras taking pictures in different locations, there will be some variation in the shape of our images. Namely, objects in the background will be sharper while objects in the foreground will be blurrier. Because of this, a simple average will only yield a sharp background.
Either way, we have achieved something -- the background is in focus after averaging. We can use this knowledge to refocus different areas of the image by aligning our images before averaging. For example, if we align the foreground in all images, then the background will be blurry in the average. Thus, we will now align images with the point that we want to be in focus then average the aligned images. Here are different areas refocused from foregroound to background.
|
|
|
|
|
|
Much remains the same in aperture adjustment except now we will simulate changing our aperture by either averaging a variable number of photos from the grid. Each photo contributes light information. The more pictures we have, the more "light" we have. When our aperture is large, we are letting more light in. When our aperture is small, we are letting less light in. Thus, to simulate aperture adjustment we will simply adjust the number of images we average.
|
|
|
Overall, these two projects taught me different ways to appreciate photos as information. Whether it is one photo that we can quilt a texture from or a grid of photos that we can simulate a light field with, at the end of the day photos are just another form of data, albeit prettier than most data.