Final Projects
Seam Carving
The first project I chose to do was seam carving. The main idea is that we want to be able to resize an image while still preserving the “interesting” parts of it. We can do that by finding ‘seams’ to remove instead of naively removing straight up columns and rows from an image to resize it.
A really good overview of this project can be found here: https://datastructur.es/sp17/materials/hw/hw5/hw5.html
Energy Function
The energy function is as described in the specification linked above. It’s a dual-gradient function, using the R/G/B values from each color channel for every pixel.
Finding a Seam
First, we need to compute the minimum energy path from the top to the bottom. This is the first pass; the base case is the first row/col depending on whether you are carving vertically or horizontally.
Next, in the second pass we can actually find the indices that correspond to a seam. We just need to choose the pixel with the minimum energy path, then from the pixels ‘connected’ to it, continue choosing pixels with the minimum energy path until we reach the other side of the image.
Carving Output
set2a |
|
set2a - 67 horizontally carved |
|
set3a |
|
set3a - 22 horizontally carved |
|
set4a |
|
set4a - 30 horizontally carved |
|
set4a - 101 horizontally carved |
|
Failure Cases
Because the algorithm is not incredibly robust, and most of my input images have a lot of similar-looking space horizontally (i.e clouds, ocean etc), I thought it would perform poorly when carving many vertical seams.
Surprisingly it did pretty poorly even with a few vertical seams for one input:
set2a - 30 vertically carved |
|
set3a - 102 vertically carved |
|
set4a - 100 vertically carved |
|
Bells and Whistles
I did pre-compute for bells and whistles; implementation-wise, that just meant I needed to first compute seams as arrays of indices to remove, in order, and then finally just ran the removal function with the pre-computed seams.
Lightfield Camera
The second project I chose was lightfield carving.
I feel like this project was difficult to conceptualize but surprisingly didn’t require too much complexity in code - the results are super cool though so definitely one of my favorites. Pictures being unfocused is basically my biggest fear as a photographer, so being able to adjust and essentially reconstruct your perfect photo is a really interesting concept!
I used two datasets to see if results differed:
Lightfield images are reconstructed by combining many images that are taken over an evenly spaced grid.
First, we simply naively average all the input images.
Eucalyptus Flowers
Lego
Notice that the Eucalyptus Flowers photo does seem to focus on the far side of objects as we expect, while the Lego Knights doesn’t really.
Depth Focusing
A key concept here is that the grid is 17x17 and evenly spaced. This makes it easy to align all images to one “point of view” by shifting from one to another by calculating a physical offset and scaling it (I used np.roll
).
Naively, we can just use the values that are in the filenames of the Stanford dataset instead of relying on them being exactly equally spaced.
However, the choice of ‘target’ image to align to isn’t actually important. Aligning all the images will always produce the same image, if shifted a little overall. Thus, the variable we must change for depth focusing is actually how much to shift by, i.e the scaling factor we apply to the physical offset.
I found that the “best” values varied by image subject and image size. Past a certain threshold, the entire image will just appear blurred.
Example
I’ve included some output labeled by where it focuses the photo.
Eucalyptus Flowers
“Center” |
“Mid-back” |
“Far back” |
|
|
|
Lego Knights
Aperture
The results are even better when you combine the ‘focus’ effect by shifting along with taking a different number of images for the average to simulate different apertures, so I used factors from the previous part which made it focus on the “center” of the image.
For a “small” aperture photo, I used only a 5x5 grid of input images; “medium”, 10x10 grid; “large”, the original 17x17 grid.
Eucalyptus Flowers
Lego Knights
Final Projects
Seam Carving
The first project I chose to do was seam carving. The main idea is that we want to be able to resize an image while still preserving the “interesting” parts of it. We can do that by finding ‘seams’ to remove instead of naively removing straight up columns and rows from an image to resize it.
A really good overview of this project can be found here: https://datastructur.es/sp17/materials/hw/hw5/hw5.html
Energy Function
The energy function is as described in the specification linked above. It’s a dual-gradient function, using the R/G/B values from each color channel for every pixel.
Finding a Seam
First, we need to compute the minimum energy path from the top to the bottom. This is the first pass; the base case is the first row/col depending on whether you are carving vertically or horizontally.
Next, in the second pass we can actually find the indices that correspond to a seam. We just need to choose the pixel with the minimum energy path, then from the pixels ‘connected’ to it, continue choosing pixels with the minimum energy path until we reach the other side of the image.
Carving Output
Failure Cases
Because the algorithm is not incredibly robust, and most of my input images have a lot of similar-looking space horizontally (i.e clouds, ocean etc), I thought it would perform poorly when carving many vertical seams.
Surprisingly it did pretty poorly even with a few vertical seams for one input:
Bells and Whistles
I did pre-compute for bells and whistles; implementation-wise, that just meant I needed to first compute seams as arrays of indices to remove, in order, and then finally just ran the removal function with the pre-computed seams.
Lightfield Camera
The second project I chose was lightfield carving.
I feel like this project was difficult to conceptualize but surprisingly didn’t require too much complexity in code - the results are super cool though so definitely one of my favorites. Pictures being unfocused is basically my biggest fear as a photographer, so being able to adjust and essentially reconstruct your perfect photo is a really interesting concept!
I used two datasets to see if results differed:
Lightfield images are reconstructed by combining many images that are taken over an evenly spaced grid.
First, we simply naively average all the input images.
Eucalyptus Flowers
Lego
Notice that the Eucalyptus Flowers photo does seem to focus on the far side of objects as we expect, while the Lego Knights doesn’t really.
Depth Focusing
A key concept here is that the grid is 17x17 and evenly spaced. This makes it easy to align all images to one “point of view” by shifting from one to another by calculating a physical offset and scaling it (I used
np.roll
).Naively, we can just use the values that are in the filenames of the Stanford dataset instead of relying on them being exactly equally spaced.
However, the choice of ‘target’ image to align to isn’t actually important. Aligning all the images will always produce the same image, if shifted a little overall. Thus, the variable we must change for depth focusing is actually how much to shift by, i.e the scaling factor we apply to the physical offset.
I found that the “best” values varied by image subject and image size. Past a certain threshold, the entire image will just appear blurred.
Example
I’ve included some output labeled by where it focuses the photo.
Eucalyptus Flowers
Lego Knights
Aperture
The results are even better when you combine the ‘focus’ effect by shifting along with taking a different number of images for the average to simulate different apertures, so I used factors from the previous part which made it focus on the “center” of the image.
For a “small” aperture photo, I used only a 5x5 grid of input images; “medium”, 10x10 grid; “large”, the original 17x17 grid.
Eucalyptus Flowers
Lego Knights