Project #1: Reconstruction of HDR Images

Here are the images we'll use for our reconstruction. There is a very high dynamic range between the very bright images and the near black images.

We want to find a nice, smooth mapping from each pixel value (0-255) and exposure time to a particular intensity. We set up a nice system of linear equations to determine this mapping as described in this paper.

We can then use this to get a radiance map of our image. We see that trying to just linearly normalize the radiance map, we end up with a very dark image. We then use a couple global tone-mapping functions (log, sqrt) and we see some pretty nice results!

We can do better, though, using local methods which map the pixels based on the content around them. Using a simplified version of the algorithm in Durand 2002, and applying some gamma compression, we get an amazing image with detail all over the place:

Project #2: Neural Style Transfer

In this project I'll generate an image which takes it's stylistic cues from one image and it's content-based cues from another. I'll be using this Leonid Afremov painting and picture of Sather Gate as an example.

We'll use some of the convolutional features of a trained VGG19 network in order to perform our optimization. First, we run our images through the network and take some of the layer outputs as features to match. For the content of the image, we take the first convolutional layer outputs. For the style, we'll take the output of several of the early convolutional layers and computer their "gram matrices" which are like covariance matrices of the different filter outputs. Here is what they look like.

Initializing our generated image to the content image and then optimizing for the L2 loss of the gram matrices of the style image and the generated image and the layer outputs of the content image and the generated image.

If we use later layers, we get a very strange looking image that is somewhat stylized.

Increasing the style loss even more, we get an even more styled image. We can see though, that there is some noise in the sky that looks weird, but it seems fine.

Here is a style transfer using the Renoir Painting Bal du moulin de la Galette

Overall the images come out pretty good, if a little noisy. Here's another example: