Soham Kale CS194 Final Project

Project 1: Gradient Domain Fusion

Toy Problem

I started out by testing my gradient algorithm by recreating an image from a single pixel and gradients off of that pixel.

Creating an index map

I created a mapping between every single pixel in my image at position (x, y) to a linear index with the transformation

>>> index = x * im_width + y

with a backwards transformation

>>> (x, y) = (index // w, index % w)

where // is floor division and where % is the modulo operator.

Generating a sparse representation

I then used SciPy's CSR sparse matrix for all my transformations with my A matrix. I built up my matrix lazily by adding to a list the non zero elements, as well as the row and column of that element, before converting to a sparse matrix.

Creating an overdetermined system

The relationship that my overdetermined system would represent was given by the following for my x-derivatives:

>>> A[i, indexof (x+1, y)] = 1

>>> A[i, indexof (x, y)] = -1

>>> b[i] = s[y, x+1] - s[y, x]

For every pixel index i, and similarly for my y-derivatives. I also had one final constraint on the top-left pixel:

>>> A[i, indexof (0, 0)] = 1

>>> b[i] = s[0, 0]

Result

Original
Original

Poisson Blending

For this part, I used a very similar overdetermined system as in the toy problem, however, I replaced the final constraint by a single pixel with a set of constraints that were determined by the boundary of my source:

>>> A[i, indexof (x+1, y)] = 1

>>> A[i, indexof (x, y)] = -1

>>> b[i] = t[y, x+1] - t[y, x]

for every (x, y) lying on the boundary of the source region, and similarly for the y derivatives.

Bells & Whistles: I also implemented mixed Poisson blending. I'll be showing both the regular and mixed versions.

I got the following results:

Name Naive Approach Poisson Mixed Poisson
Human-penguin relations
Lone ranger
An unfortunate Tuesday
Pupper protection detail
I think it's a Banksy
Math is everywhere

Project 2: Seam Carving

For this project, I implemented seam carving.

Calculating the energy

I used the dual-gradientenergy algorithm, using a Sobel filter to calculate the x and y derivatives:

>>> E = dx(x, y)^2 + dy(x, y)^2

where

>>> dx(x, y)^2 = dx,r(x, y)^2 + dx,g(x, y)^2 + dx,b(x, y)^2

>>> dy(x, y)^2 = dy,r(x, y)^2 + dy,g(x, y)^2 + dy,b(x, y)^2

and

>>> dx,r(x, y)^2 = (im[x + 1, y] - im[x - 1, y])^2

and with similar rules for the other channels and gradient directions.

Finding a seam

I implemented seam finding using a dynamic programming algorithm that finds the lowest energy seam. For each pixel, I cached the energy of the lowest energy seam from the top of the image to that pixel.

>>> cache[x,y] = energy[x,y] + min{cache[x-1,y-1], cache[x,y-1], cache[x+1,y-1]}

I also cached which direction above that pixel that particular seam originates from. I then removed the seam using numpy slicing techniques.

Making it bi-directional

To convert from just scaling in the x direction to scaling in the y direction, I tranposed the image first, scaled by the appropriate amount, and then transposed back.

Results

Original Scaled

Failures

Bells and Whistles

I also implemented precomputation into my seam removal. I created a function that would precompute a series of seam removals, up to a maximum amount. I then created an online function that would remove seams according to the cached order that they were presented in the precomputation function.

Though I am not able to show a dynamic version of this on my website, my results are in my code. The speed ups for a 500px width image are shown below:

Computation Time (seconds)
Precomputing seams to 100px width 129.63
Resizing to 400px width 0.29
Resizing to 300px width 1.22
Resizing to 200px width 1.18

The online component is much faster and could be implemented in a fairly dynamic manner.

What I learned

The most important thing I learned from this project is how to properly optimize sequential code. I spent time timing each of my different functions to learn where my bottlenecks where, and looked into how to properly optimize each one.

For example, I found that my seam removal function was a bottleneck because I was using nested loops. I moved to using Numpy's slicing operations and I got nearly a 40x improvement. I also decided to use to a DP approach for my seam finding rather than the naive version I was using.