Fall 2020 CS 194-26 Project 1

Author: Youqi Chen

Project Overview:


In this project, I implemented a program that converts 3 color-channel images into a single colored image. For both low resoltion images (.jpg) and high resolution images (.tif), Sum of Squared Differences (SSD) is used to score pixels for alignment. While low resolution images are exhaustively searched, an image pyramid is used for recursively searching high resolution images and compute displacement.

Approach:


To start with, images are loaded into the program and divided into three equally sized parts representing Blue, Green, and Red channels. For each output, the program align Green and Red channels with the Blue channel to produce the colored image.

Low Resolution .jpg Images: Exhaustive Search

For the low resolution images, the program iterated through a [-15, 15] pixel window to exhaustively search for the best displacement. When applying the matching metric to the images, 10% of each side of the images are cropped. A Sum of Squared Differences (SSD) score is then placed on each pixel and the pixel with the smallest SSD is picked as the displacement for the two input images. After extracting the best (Blue, Green) and (Blue, Red) displacements, the program then align the three *original* images based on the displacements.

High Resolution .tif Images: Image Pyramid

For the higher resolution images, Image Pyramid is used to downscale the image to roughly the same size as the low resolution images, and then accumulating displacements at each level to yield the final displacements. The alignment at each level applies the same cropping and SSD metric as it is for the low resoltion images, and by recursively callling pyramid align program, displacements are accumulated to quickly align the images at their original scale.

Difficulties:


Due to the unnatural borders of the images, some alignments do not work well when using the original image. The program cropped 10% of each border when computing displacements, and the outcomes are more satisfactory.

'emir.tif' does not align well under the plain SSD implementation because the difference in brightness created noises for finding the best alignment. Therefore, instead of comparing pixels, the program applied sobel filter to the original images to detect edges and align based on the filtered image. This gives better aligned outputs than the original solution.

Results:


All of the images except emir.tif are aligned well under the current implementation. In the Sobel solution, the unnatural border are cropped with a fixed value. As explained in the 'Difficulties' section, the brightness issue is addressed with Sobel filter that enables alignment by edges. Some Sobel solutions are not displayed on the website as there is little change in displacement for that image comparing to the SSD solution.

Image name SSD Output SSD Displacement Sobel Output Sobel Displacement
castle Red: (0, 96)
Green: (0, 32)
cathedral Red: (3, 12)
Green: (2, 5)
emir Red: (-240, 96)
Green: (16, 48)
Red: (32, 112)
Green: (32, 48)
harvesters Red: (16, 128)
Green: (16, 64)
icon Red: (16, 96)
Green: (16, 48)
lady Red: (16, 112)
Green: (16, 48)
melons Red: (16, 176)
Green: (0, 80)
Red: (16, 176)
Green: (16, 80)
monastery Red: (2, 3)
Green: (2, -3)
onion_church Red: (36, 112)
Green: (32, 48)
Red: (32, 112)
Green: (32, 48)
self_portrait Red: (32, 176)
Green: (32, 80)
three_generations Red: (16, 112)
Green: (16, 48)
tobolsk Red: (3, 6)
Green: (3, 3)
train Red: (32, 96)
Green: (0, 48)
Red: (32, 80)
Green: (0, 48)
workshop Red: (-16, 96)
Green: (0, 48)
Red: (-16, 112)
Green: (0, 48)
solovetskii Red: (0, 112)
Green: (0, 32)
railroad Red: (-16, 96)
Green: (0, 48)
Red: (0, 128)
Green: (0, 48)
foreman Red: (0, 16)
Green: (0, 64)