Project 4: Classification and Segmentation

I used the Fashion MNIST dataset available in torchvision.datasets.FashionMNIST to train an image classification neural net. The dataset has 10 different classes:

There are 60,000 images in the training set, and 10,000 images in the testing set. I split the training set data into 50,000 train images and 10,000 validation images. Here are some samples images from the training set:

Fashion MNIST Training Set Sample Images

My architecture for this CNN is as follows:

I started out using the recommended 2 convolution layers with 32 channels each, but in order to increase my accuracy, I decided to increase the channels in the second convolutional layer to 64 channels.

I used the following loss function and optimizer:

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

I trained the network using the 50,000 test images, and tuned the hyperparameters based on the accuracy on the 10,000 validation set images. I trained over 10 epochs, using batches of 2,000 images at a time.

Here is a plot of the training and validation accuracy per epoch.

Training and Validation Accuracy

Below is the per-class accuracy of the validation set & the testing set:

The network has the most trouble differentiating between T-Shirts, Coats, Shirts, and Pullovers. This is understandable because they all have similar image structures and can be easily confused.

For each class, here are some examples of correctly and incorrectly classified images:

Classified As	Correct	Incorrect
0	T-Shirt T-Shirt	Pullover Dress
1	Trouser Trouser	Dress Shirt
2	Pullover Pullover	Coat Shirt
3	Dress Dress	Trouser Coat
4	Coat Coat	Pullover Dress
5	Sandal Sandal	T-Shirt Bag
6	Shirt Shirt	T-Shirt Pullover
7	Sneaker Sneaker	Ankle Boot Sandal
8	Bag Bag	T-Shirt Dress
9	Ankle Boot Ankle Boot	Sandal Sneaker

Here are the visualized filters for the first layer of convolution. I had 32 channels, with each channel consisting of a 5x5 filter.

I used the MiniFacade dataset to implement Semantic Segmentation in order to classify individual pixels in an image like so:

There are 906 images in the training set, and 114 images in the testing set. I split the training set data into 725 train images and 181 validation images. Here is an example of a training image, and its corresponding ground truth segmentation.

Original Image

Segmentation

My architecture for this CNN is as follows:

I chose this architecture after looking at how the network performed on the validation set. I decided to only do one MaxPool operation because MaxPool tends to collapse individual pixel features, which would work well for image classification but so well for image segmentation. I chose to do a U-Net type architecture similar to the one described in this paper

I used the following loss function and optimizer:

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

I trained the network using the 725 test images, and tuned the hyperparameters based on the accuracy on the 181 validation set images. I trained over 50 epochs.

Below is a plot of the training and validation accuracy per epoch.

Training and Validation Loss

Below is the average precision of my network calculated on the testing set:

Average AP: 0.51897080071

Here are some examples of my network's output from the testing set:

Input Image

Ground Truth

Network Output

Input Image

Ground Truth

Network Output

Input Image

Ground Truth

Network Output

Input Image

Network Output

The network generally performs well when its identifying the facade and windows. It does not do as well with pillars and balconies. This may be for a variety of reasons; for one, every training image had facade & windows while only a smaller subset had balconies, and and even smaller subset had pillars. Additionally, windows have straighter edges & are much easier to visually identify. Therefore, it is expected that the network perform well in segmenting windows. However, balconies and pillars are harder to identify because their shapes are not standardized.

The last image & network output pair is a picture I took of the Royal Palace in Madrid, Spain that I cropped to (256, 256, 3) to run through the network. The network does not do well in identifying the pillars on the palace, and instead classifies it as part of the facade. It recognizes some of the balcony on the middle and top floors, but not very clearly. The network could perhaps not be performing as well because of the large amount of sky present in the picture, which was not present in many of the training images.