Final Project: Lightfield Camera

CS 194

By Won Ryu

Overview

In this paper (http://graphics.stanford.edu/papers/lfcamera/lfcamera-150dpi.pdf) by UC Berkeley Professor Ren Ng who also founded Lytro camera, there are benefits of capturing multiple images over a plane orthogonal to the optical axis as we can use those multiple images to simulate some properties of the camera after taking the images using operations such as shifting and averaging. This project is about using the lightfield camera data which are multiple images taken over a regularly spaced grid to reproduce the effects that a camera can make.

Depth Refocusing

Depending on the focal length of the lens, a camera can have different depth of focus. With lightfield camera data, we can achieve a similar effect by refocusing the depth after taking the images. This is possible due to that objects which are further away from the camera do not vary their position significantly when the camera moves around while not changing the optical axis direction. However, for the close objects, their position changes much more across the lightfield camera images. As a result, averaging all the lightfield camera images without any shifting will produce an image which is sharp for the objects that are further away but blurry for the objects close by. Also, shifting the images to the center lightfield camera image (the middle one in the grid) and then averaging makes the close objects to be sharp and the ones that are further away to be blurry. Depending on how much the shift occurs will determine which depth the focus is on.

As a result, the depth refocusing was done by using the x and y positions of the center image and calling it midX and midY respectively. Then for each image, I called the x and y position as X and Y respectively. Then I would shift the image by c(X-midX) in the horizontal and c(Y-midY) in the vertical direction. Finally I took the average of the shifted images. The parameter to vary here was c which would determine how much shift would happen which would determine at which depth the focus would occur.

The range of c where the image had a part that was in focus was between -0.15 and 0.65. Here is the gif of images between those range:

This is the gif. If the gif appears to not be moving, it has finished playing. Right click and Open Image in New Tab. Then hit refresh and it’ll start again.

bears

Here is also a sample images when c = -0.05

bears

Here is also a sample images when c = 0.45

bears

Another example

This is the gif. If the gif appears to not be moving, it has finished playing. Right click and Open Image in New Tab. Then hit refresh and it’ll start again.

bears

Here is also a sample images when c = 0.05

bears

Here is also a sample images when c = -0.45

bears

Aperture Adjustment

Just like a camera being able to adjust how blurry the parts that are not in focus are by changing the size of the aperture (large aperture the more blurry and smaller aperture less blurry), we can adjust how blurry the parts that are not in focus are with the lightfield camera image data. When we include more images from the image dataset to average, we get a more blurred image due to the slight differences in the position of the objects. As a result, the more images we include the shift and averaging process, the blurrier the parts that aren’t in focus will be. In order to simulate the effects of adjusting the aperture, we will follow the same procedure as before for depth refocusing of shifting and averaging the images. However this time, we will pick a constant c for all images as this will keep the object in focus the same and instead the parameter we will change which will act as changing the aperture will be r which will be related to the number of images included in the shifting and averaging to create the processed images. When r is 0, it means only the middle image is used and as r goes up by 1 we add two more images (one from left and one from right) to the subset of images we use to make the processed image.

This is the gif. If the gif appears to not be moving, it has finished playing. Right click and Open Image in New Tab. Then hit refresh and it’ll start again.

bears

Here is also a sample images when r = 2

bears

Here is also a sample images when r = 90

bears

Another example

This is the gif. If the gif appears to not be moving, it has finished playing. Right click and Open Image in New Tab. Then hit refresh and it’ll start again.

bears

Here is also a sample images when r = 1

bears

Here is also a sample images when r = 35

bears

Bells & Whistles: Using Real Data

Now that this was done with data from a lightfield camera, I wanted to see if it was possible for me to collect my own data to perform these effects. I used an iPhone for my camera and I drew up a 5X5 grid of points with each point being 5cm from each other in width and height.

bears

Then I placed the corner of my iPhone on each of the points and took a photo from each of the 25 points. With these data I performed the Depth Refocusing and Aperture Adjustment as before.

Depth Refocusing

This is the gif. If the gif appears to not be moving, it has finished playing. Right click and Open Image in New Tab. Then hit refresh and it’ll start again.

bears

Here is also a sample images when c = 117.5

bears

Here is also a sample images when c = 159.5

bears

Aperture Adjustment

This is the gif. If the gif appears to not be moving, it has finished playing. Right click and Open Image in New Tab. Then hit refresh and it’ll start again.

bears

Here is also a sample images when r = 0

bears

Here is also a sample images when r = 12

bears

These pictures I took myself worked decently. It is definitely not as sharp as the lightfield camera data and that is likely due to that when I took these images by hand the images were not perfectly all equally separated by 5cm in width and height. Also it was hard to keep the iPhone straight with no rotations between the images and as a result between the images there were some slight rotations that were unintended. Due to imperfections of rotations and slight shifts, the pixels were not proportionally aligned even when shifted making even the parts that should be sharp a bit blurry. Also since I took 25 pictures in a 5X5 grid, I only had 25 images as opposed to 289 images from the lightfield camera data which made the aperture adjustment not a lot of fine adjustment possible.

Summary

I learned that these effects that you can have on photos by adjusting settings on a camera can actually be applied using shifting and averaging when there is a grid of images on the same view.

Final Project: Image Quilting

CS 194

By Won Ryu

Overview

This project was about texture synthesis and texture transfer based on this paper by Professor Efros and Freeman (https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/papers/efros-siggraph01.pdf). Texture synthesis is when you have a sample texture that you want a larger texture in the same texture as the sample texture. The main idea in texture synthesis is sampling patch by patch and laying them down to look smooth and three different texture synthesis methods are done in this project. Texture transfer is when there is a sample texture and a target image and we want the target image to retain its noticeable appearances but have the texture of the sample texture.

Randomly Sampled Texture

To begin with for texture synthesis, we will start with a simple idea of random sampling the texture. When we have a sample, a desired output size and a patch size of the texture we will be laying out one by one, we can take a random patch from the sample and lay it out on the output we’re building and repeat this process until the image is full.

Example:

bears

Overlapping Patches

Random sampling as the patches were randomly sampled makes the boundaries between the patches very obvious. To make the boundaries look smoother, we’re going to take advantage of that if there are overlapping parts and those overlapping parts are similar to each other, then the boundaries will look smoother. For each patch in the output we’re building, we are going to start with the upper left corner to be randomly sampled. To fill the rest of the patches, we’re going to calculate the sum of squared differences (SSD) of the overlapping regions of the upper neighboring patch and the left neighboring patch for each possible sample patch as the cost as the SSD will be used to determine how similar the overlapping regions are. Then we will select one of the patches of the sample that have a cost that is lower than the_lowest_cost * (1+tol) at random onto the output image we’re building to ensure a repetitive pattern doesn’t occur. If there are no left neighbor or upper neighbor (i.e. we’re on the upper parts or left most parts) then we only consider the overlapping region with the neighbor that exists to calculate cost. When the sample patch is chosen, we copy over to the output directly into the corresponding position in the output image.

Example:

bears

Seam Finding

There are still some noticeable straight borders between the laid patches and to mitigate this we will include seam finding. In order to include seam finding, I took the two overlapping parts of the patches and found the min-cost contiguous path using a dynamic programming algorithm where cost is the SSD summed across the three color channels for each 2D pixel position. This cut was used to create a mask where the part above the cut was set to 0 and the part below was set to 1. For the parts that have top and left overlaps, the mask was mask of upper binary and with mask of left. Then using the mask, the overlapping part had parts from the neighboring patch and parts from the new sample patch to put and the path determined where the cut was.

Example:

bears

To illustrate the seam finding process, here are 2 overlapping patches.

bears bears

Their cost image with the min path plotted on top. bears

Comparing random, overlapping, and seam-finding methods side by side

Original sample texture:

bears

Random bears Overlapping bears Seam-finding bears

As visible, the boundaries between the patches become less noticeable from random to overlapping to seam cut.

Some more image quilting texture synthesis (texture sampple to the left output synthesis to the right):

Texture of my dog fur

bears bears

Texture of the sea

bears bears

Texture of my couch

bears bears

Texture of toast

bears bears

Texture Transfer

Using the algorithm for texture synthesis where we picked the patch from the sample that had a low cost and then used seam finding to stitch them together, we can perform texture transfer where we make a target image to have the texture we desire. For this we change the cost to not only be the SSD of the overlapping patches but also include the SSD between the patch and the part of the target we’re trying to fill. This way we pick a patch from the sample that is also similar to the part of the target image we’re trying to fill. As a result the cost becomes alpha * SSD_with_overlaps + (1-alpha) * SSD_with_target and we have alpha as a parameter that determines how much importance we put on making the patch similar to the target as opposed to making the patch borders smooth.

Feynman

Texture bears

Target image bears

Texture transfer bears

Biden

Texture bears

Target image bears

Texture transfer bears

Bells & Whistles: Own version of cut

For this project, I implemented my version of cut which takes a 2D cost matrix as input and returns a 2D matrix of the same shape which is a matrix of 0’s above the cut and 1’s below the cut. The cost matrix used for this function would be a 2D matrix of the SSD summed over the RGB color channels of 2 overlapping patches. The algorithm to find the continuous cheapest cut was a dynamic algorithm. For each pixel coordinate in the overlap, E(i,j) was computed as InputMatrix(i,j) + min(E(i-1,j-1), E(i-1,j), E(i-1,j+1)) and which of the 3 paths it took was also kept track. Then the lowest values in the last column was where the cheapest cut would end and the full cut was found by tracing back which path it took.

Bells & Whistles: Iterative texture transfer

For texture transfer, the non-iterative method was improved by iterating upon the output. This was done by having texture transfer done in several iterations where the first iteration was done as before and then at each of the rest of the iterations, we used the previous result to be a part of the cost by including the SSD of the difference with the result from the previous iteration in the cost. This ensured it would have similarities with the result from previous iterations. Across iterations, I also decreased the block size by half (the paper divided the block by 3 but that was too much for my images as the sample textures I was using worked well with a small initial patch size which meant between iterations it needed to be made smaller by less than what the paper used) to ensure that between iterations the adjustments would be finer in detail. Also, across iterations the alpha value increased by this formula from the paper alpha(ith iteration) = 0.8*(i-1)/(N-1) + 1 where N was the number of iterations in total which was 3 for me. This meant as the iterations went on the alpha value increased and we cared more about having the images be similar to the previous iterations and overlaps to avoid over fitting to the target image and losing a form of structure set by the iterations with the larger patch sizes.

Biden

Non-iterative texture transfer

bears

Iterative texture transfer

bears

Feynman

Non-iterative texture transfer

bears

Iterative texture transfer

bears