Soham Kale CS194 Final Project
I started out by testing my gradient algorithm by recreating an image from a single pixel and gradients off of that pixel.
I created a mapping between every single pixel in my image at position (x, y) to a linear index with the transformation
>>> index = x * im_width + y
with a backwards transformation>>> (x, y) = (index // w, index % w)
where // is floor division and where % is the modulo operator.I then used SciPy's CSR sparse matrix for all my transformations with my A matrix. I built up my matrix lazily by adding to a list the non zero elements, as well as the row and column of that element, before converting to a sparse matrix.
The relationship that my overdetermined system would represent was given by the following for my x-derivatives:
>>> A[i, indexof (x+1, y)] = 1
>>> A[i, indexof (x, y)] = -1
>>> b[i] = s[y, x+1] - s[y, x]
For every pixel index i, and similarly for my y-derivatives. I also had one final constraint on the top-left pixel:>>> A[i, indexof (0, 0)] = 1
>>> b[i] = s[0, 0]
For this part, I used a very similar overdetermined system as in the toy problem, however, I replaced the final constraint by a single pixel with a set of constraints that were determined by the boundary of my source:
>>> A[i, indexof (x+1, y)] = 1
>>> A[i, indexof (x, y)] = -1
>>> b[i] = t[y, x+1] - t[y, x]
for every (x, y) lying on the boundary of the source region, and similarly for the y derivatives.Bells & Whistles: I also implemented mixed Poisson blending. I'll be showing both the regular and mixed versions.
I got the following results:
Name | Naive Approach | Poisson | Mixed Poisson |
---|---|---|---|
|
|||
|
|||
|
|||
|
|||
|
|||
|
For this project, I implemented seam carving.
I used the dual-gradientenergy algorithm, using a Sobel filter to calculate the x and y derivatives:
>>> E = dx(x, y)^2 + dy(x, y)^2
where>>> dx(x, y)^2 = dx,r(x, y)^2 + dx,g(x, y)^2 + dx,b(x, y)^2
>>> dy(x, y)^2 = dy,r(x, y)^2 + dy,g(x, y)^2 + dy,b(x, y)^2
and>>> dx,r(x, y)^2 = (im[x + 1, y] - im[x - 1, y])^2
and with similar rules for the other channels and gradient directions.I implemented seam finding using a dynamic programming algorithm that finds the lowest energy seam. For each pixel, I cached the energy of the lowest energy seam from the top of the image to that pixel.
>>> cache[x,y] = energy[x,y] + min{cache[x-1,y-1], cache[x,y-1], cache[x+1,y-1]}
I also cached which direction above that pixel that particular seam originates from. I then removed the seam using numpy slicing techniques.To convert from just scaling in the x direction to scaling in the y direction, I tranposed the image first, scaled by the appropriate amount, and then transposed back.
Original | Scaled |
---|---|
I also implemented precomputation into my seam removal. I created a function that would precompute a series of seam removals, up to a maximum amount. I then created an online function that would remove seams according to the cached order that they were presented in the precomputation function.
Though I am not able to show a dynamic version of this on my website, my results are in my code. The speed ups for a 500px width image are shown below:
Computation | Time (seconds) |
---|---|
Precomputing seams to 100px width | 129.63 |
Resizing to 400px width | 0.29 |
Resizing to 300px width | 1.22 |
Resizing to 200px width | 1.18 |
The online component is much faster and could be implemented in a fairly dynamic manner.
The most important thing I learned from this project is how to properly optimize sequential code. I spent time timing each of my different functions to learn where my bottlenecks where, and looked into how to properly optimize each one.
For example, I found that my seam removal function was a bottleneck because I was using nested loops. I moved to using Numpy's slicing operations and I got nearly a 40x improvement. I also decided to use to a DP approach for my seam finding rather than the naive version I was using.