Final Project: Image Quilting & Light-Field Camera
Project 1 : Image Quilting, Ken Guan
In this project, we will first generate a smooth texture image from a much smaller sample, then transfer the texture to another image.
We generate a texture image by selecting and combining patches from the source image. We first try to randomly select samples, but the result is not very smooth, as one expects.
Left: source image. Right: random patches 4x4
To achieve smoother results, we must select source patches that result in continuous transition. We introduce an overlap between every two neighboring patches in the final image. For each new patch to be sampled, we identify its overlap with neighboring patches find source patches that have low SSD with these overlaps. For this project we randomly select from the 10 candidate patches with lowest SSD. The result appears slightly better, although edge effects are still quite obvious.
Left: random patches 4x4. Right: SSD patch selection. The bricks are lined up much better.
To further smooth out the edge effects, we do not simply overwrite old patches at the overlap. Instead, for each overlap, we find a lowest cost "seam" that cuts the region into two halves. We then use the previous patch on one half and the newly sampled patch on the other half.
From left to right: the two overlap regions; The SD cost at each point; The lowest cost seam; The combined image.
The resulting image is also quite a bit nicer:
Left: SSD patch selection. Right: SSD patch selection + combination with seam finding
A few more results from this method:
-> 
-> 
-> 
The idea of texture transfer is such that during patch selection, we account for a weighted sum of the alignment error (overlap SSD) and the correspondence error, which is defined by a how well the sampled patch imitates a target image. I used SSD on grayscale as the correspondence error heuristic. The results are not very ideal.
-> 
From left to right: alpha = 0.3, 0.5, 0.7, 0.9. Lower alpha means more weight on shape. Higher alpha means more weight on smoothness.
Using smaller patches help a bit with correspondence, but hurts continuity.
->
For future improvement, we could try a different correspondence metric.
I implemented my own version of the seam finding algorithm. I use DP to calculate minimum cumulative error at each point and perform a back traverse from the minimal error point in the last row. Code is in the Notebook.
Project 2 : Light-Field Camera, Ken Guan
In this project, we will explore two special effects that one can make from multiple images over a plane orthogonal to the optical axis. All images used are taken in a 17x17 grid by The Stanford Light Field Archive .
If we stack all the images in a series together, the back region of the image appears clear while the front appears blurry. This is because the objects that are far away moved less in angle relative to the camera.
If we shift all the images toward the center image (the (8, 8) image in our case) by a certain amount such that the front region is better aligned, the back region will then be blurred. We can thus generate a series of images with different focuse depth by altering the amount of shift. Here are the results:
We could also imitate pictures taken by cameras with different apertures. How? Simply cut out some images from the outer ring of the grid, and the image will become blurred in the outer ring! The more we cut, the smaller aperture the camera appears to have.
While these are only two of the many special effects achievable, it is still stunning how simple, elegant the method is and how beautiful the results are. I'm curious whether the final aperture adjustment method could be applied with panomara stitching to generate a panomara with consistently high aperture even with a lower aperture camera.