Part 1: Nose Tip Detection

Sample Images and the Nose Keypoints

/Users/tusharchopra/anaconda3/envs/194/lib/python3.7/site-packages/torch/nn/modules/loss.py:446: UserWarning: Using a target size (torch.Size([1, 1, 2])) that is different to the input size (torch.Size([1, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

Epoch: 1 	Training Loss: 11.993408 	Validation Loss: 12.076559
Validation loss decreased (inf --> 12.076559).  Saving model ...
Epoch: 2 	Training Loss: 7.404642 	Validation Loss: 6.591562
Validation loss decreased (12.076559 --> 6.591562).  Saving model ...
Epoch: 3 	Training Loss: 7.539657 	Validation Loss: 5.394070
Validation loss decreased (6.591562 --> 5.394070).  Saving model ...
Epoch: 4 	Training Loss: 6.931491 	Validation Loss: 5.553184
Epoch: 5 	Training Loss: 6.898903 	Validation Loss: 13.310530
Epoch: 6 	Training Loss: 6.282839 	Validation Loss: 5.614550
Epoch: 7 	Training Loss: 5.841912 	Validation Loss: 11.728922
Epoch: 8 	Training Loss: 6.078113 	Validation Loss: 5.018254
Validation loss decreased (5.394070 --> 5.018254).  Saving model ...
Epoch: 9 	Training Loss: 5.111738 	Validation Loss: 6.634047
Epoch: 10 	Training Loss: 4.989370 	Validation Loss: 5.279741
Epoch: 11 	Training Loss: 4.720344 	Validation Loss: 4.221449
Validation loss decreased (5.018254 --> 4.221449).  Saving model ...
Epoch: 12 	Training Loss: 4.014762 	Validation Loss: 4.643963
Epoch: 13 	Training Loss: 4.259456 	Validation Loss: 8.123965
Epoch: 14 	Training Loss: 3.563928 	Validation Loss: 4.111737
Validation loss decreased (4.221449 --> 4.111737).  Saving model ...
Epoch: 15 	Training Loss: 3.593453 	Validation Loss: 3.551310
Validation loss decreased (4.111737 --> 3.551310).  Saving model ...
Epoch: 16 	Training Loss: 3.083703 	Validation Loss: 2.754069
Validation loss decreased (3.551310 --> 2.754069).  Saving model ...
Epoch: 17 	Training Loss: 3.179502 	Validation Loss: 2.906153
Epoch: 18 	Training Loss: 2.596480 	Validation Loss: 3.112926
Epoch: 19 	Training Loss: 3.176289 	Validation Loss: 2.820181
Epoch: 20 	Training Loss: 2.432105 	Validation Loss: 2.507370
Validation loss decreased (2.754069 --> 2.507370).  Saving model ...
Epoch: 21 	Training Loss: 2.283013 	Validation Loss: 2.426327
Validation loss decreased (2.507370 --> 2.426327).  Saving model ...
Epoch: 22 	Training Loss: 2.345575 	Validation Loss: 4.367297
Epoch: 23 	Training Loss: 2.299767 	Validation Loss: 2.649786
Epoch: 24 	Training Loss: 1.595773 	Validation Loss: 2.408165
Validation loss decreased (2.426327 --> 2.408165).  Saving model ...
Epoch: 25 	Training Loss: 2.179755 	Validation Loss: 3.702160

Graph of training loss and valid loss

<matplotlib.legend.Legend at 0x7ff2e645db90>

array([42.79999261, 30.17728832])

Predicting Nose Points

Failures

<matplotlib.collections.PathCollection at 0x7ff2e2e45210>

<matplotlib.collections.PathCollection at 0x7ff2e3bc9790>

I think these fail because they're not looking straight into the camera, and there's not enough data for the model to learn that faces can rotate

Success

<matplotlib.collections.PathCollection at 0x7ff2e18c2e50>

<matplotlib.collections.PathCollection at 0x7ff2e436bcd0>

Part 2: Full Facial Keypoints Detection

My CNN model has 6 convolutions and 3 linear layers. After each convolution I apply the max pool 2d of the ReLu output. I also apply the ReLu to the first two linear layers.

Epoch: 1 	Training Loss: 36.345543 	Validation Loss: 11.980581
Validation loss decreased (inf --> 11.980581).  Saving model ...
Epoch: 2 	Training Loss: 14.361457 	Validation Loss: 24.072775
Epoch: 3 	Training Loss: 18.306268 	Validation Loss: 14.149394
Epoch: 4 	Training Loss: 14.132140 	Validation Loss: 10.958599
Validation loss decreased (11.980581 --> 10.958599).  Saving model ...
Epoch: 5 	Training Loss: 11.547324 	Validation Loss: 10.400283
Validation loss decreased (10.958599 --> 10.400283).  Saving model ...
Epoch: 6 	Training Loss: 12.298830 	Validation Loss: 11.287458
Epoch: 7 	Training Loss: 12.653927 	Validation Loss: 12.927188
Epoch: 8 	Training Loss: 12.159833 	Validation Loss: 10.240041
Validation loss decreased (10.400283 --> 10.240041).  Saving model ...
Epoch: 9 	Training Loss: 12.604101 	Validation Loss: 10.077740
Validation loss decreased (10.240041 --> 10.077740).  Saving model ...
Epoch: 10 	Training Loss: 10.737818 	Validation Loss: 10.255321

plot of train and valid loss

<matplotlib.legend.Legend at 0x7ff2d8721610>

Predictions

Failures

white is actual, red is predicted

<matplotlib.collections.PathCollection at 0x7ff2e628ea90>

<matplotlib.collections.PathCollection at 0x7ff2e617a0d0>

I think these aren't as good because the face is looking in different direction and the mouth is open in the first one.

Success

<matplotlib.collections.PathCollection at 0x7ff2e61b94d0>

<matplotlib.collections.PathCollection at 0x7ff2d40da590>

Filters

Part 3: Train with Larger Dataset

<matplotlib.collections.PathCollection at 0x7fe5603453c8>

Chose the Wide Resnet 101-2 model as it had the lowest Top-5 Error on the pytorch website. A wide resnet reduces the training time but with the expense of increasing the number of parameters due to the widening of the network. Chose a batch size of 4 to increase accuracy but also not have a long training time. I tried learning rate of 0.001, but I think it got stuck in a local minimum.

training
Epoch: 1 	Training Loss: 41.942136 	Validation Loss: 556.061587
Validation loss decreased (inf --> 556.061587).  Saving model ...
Epoch: 2 	Training Loss: 32.693303 	Validation Loss: 37.495496
Validation loss decreased (556.061587 --> 37.495496).  Saving model ...
Epoch: 3 	Training Loss: 32.324002 	Validation Loss: 42.331469
Epoch: 4 	Training Loss: 32.196633 	Validation Loss: 36.570268
Validation loss decreased (37.495496 --> 36.570268).  Saving model ...
Epoch: 5 	Training Loss: 31.676811 	Validation Loss: 36.433591
Validation loss decreased (36.570268 --> 36.433591).  Saving model ...
Epoch: 6 	Training Loss: 31.652815 	Validation Loss: 35.608265
Validation loss decreased (36.433591 --> 35.608265).  Saving model ...
Epoch: 7 	Training Loss: 31.585463 	Validation Loss: 39.584903
Epoch: 8 	Training Loss: 31.329903 	Validation Loss: 36.816471
Epoch: 9 	Training Loss: 31.035557 	Validation Loss: 37.020407
Epoch: 10 	Training Loss: 30.964417 	Validation Loss: 38.029210
Epoch: 11 	Training Loss: 30.434661 	Validation Loss: 32.956229
Validation loss decreased (35.608265 --> 32.956229).  Saving model ...
Epoch: 12 	Training Loss: 29.709201 	Validation Loss: 34.989040
Epoch: 13 	Training Loss: 28.706096 	Validation Loss: 31.454396
Validation loss decreased (32.956229 --> 31.454396).  Saving model ...
Epoch: 14 	Training Loss: 27.810327 	Validation Loss: 36.474533
Epoch: 15 	Training Loss: 26.784513 	Validation Loss: 32.639664
Epoch: 16 	Training Loss: 25.856699 	Validation Loss: 35.379851
Epoch: 17 	Training Loss: 24.865633 	Validation Loss: 37.417058
Epoch: 18 	Training Loss: 23.769616 	Validation Loss: 36.492834
Epoch: 19 	Training Loss: 22.674175 	Validation Loss: 34.137691
Epoch: 20 	Training Loss: 21.561496 	Validation Loss: 42.660261
Epoch: 21 	Training Loss: 20.484348 	Validation Loss: 39.119189
Epoch: 22 	Training Loss: 19.399423 	Validation Loss: 46.976135
Epoch: 23 	Training Loss: 18.311935 	Validation Loss: 46.072699
Epoch: 24 	Training Loss: 17.335451 	Validation Loss: 41.131859

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-63-40fca8a04394> in <module>()
      2                                    model,criterion, optimizer,
      3                                    n_epochs=500,
----> 4                                    saved_model='model.pt')

<ipython-input-62-92bbc88b65dc> in train(train_loader, valid_loader, model, criterion, optimizer, n_epochs, saved_model)
     19             output = torch.reshape(output, (len(batch['image']),68,2)).to(device)
     20             loss = criterion(output, landmarks)
---> 21             loss.backward()
     22             optimizer.step()
     23             train_loss += loss.item()*batch['image'].size(0)

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--> 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222
    223     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--> 132         allow_unreachable=True)  # allow_unreachable flag
    133
    134

KeyboardInterrupt: 

Plot for valid and training loss

<matplotlib.legend.Legend at 0x7fe8c2121ac8>

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py:63: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return default_collate([torch.as_tensor(b) for b in batch])

<matplotlib.image.AxesImage at 0x7fe8a8356f60>

'749'

<matplotlib.collections.PathCollection at 0x7fe8c1fec240>

<matplotlib.collections.PathCollection at 0x7fe8c1eb26a0>

<matplotlib.collections.PathCollection at 0x7fe8c1e12b38>

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-83-d1e9e1e20b82> in <module>()
      2 img = cv2.resize(tushar, dsize=(224, 224))
      3 img = color.rgb2gray(img) - 0.5
----> 4 output = model(torch.from_numpy(np.expand_dims(img, 0))).cpu().numpy()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torchvision/models/resnet.py in forward(self, x)
    218
    219     def forward(self, x):
--> 220         return self._forward_impl(x)
    221
    222

/usr/local/lib/python3.6/dist-packages/torchvision/models/resnet.py in _forward_impl(self, x)
    201     def _forward_impl(self, x):
    202         # See note [TorchScript super()]
--> 203         x = self.conv1(x)
    204         x = self.bn1(x)
    205         x = self.relu(x)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    421
    422     def forward(self, input: Tensor) -> Tensor:
--> 423         return self._conv_forward(input, self.weight)
    424
    425 class Conv3d(_ConvNd):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    418                             _pair(0), self.dilation, self.groups)
    419         return F.conv2d(input, weight, self.bias, self.stride,
--> 420                         self.padding, self.dilation, self.groups)
    421
    422     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 1, 3, 3], but got 3-dimensional input of size [1, 224, 224] instead

range(0, 137088)

[598.2648449882132,
 700.9292132268187,
 607.2505164849954,
 756.0530377997727,
 621.2033386855829,
 809.0346099353228,
 640.0809264261214,
 859.4482176733799,
 669.6081971965853,
 904.5307472103932,
 711.3290251434827,
 938.4672356277216,
 752.4627205270235,
 953.3058641777664,
 788.2039554783555,
 949.7510410996734,
 827.2981267209913,
 934.4640582975794,
 876.6373736272093,
 922.1513776935515,
 929.6824145707928,
 913.1921361704342,
 987.5180348880956,
 904.5070490602587,
 1024.3668818239307,
 870.7543470038743,
 1043.5848413686283,
 822.9763288654265,
 1052.1223764888575,
 770.1229010409996,
 1055.319649117892,
 716.8502640020652,
 1052.765377357358,
 664.1712538922419,
 636.1264191299189,
 661.3405819251889,
 664.6984147869173,
 636.5289287879818,
 705.4320488601435,
 627.6967673379867,
 748.4551521676485,
 628.9010071050925,
 791.3212175212923,
 637.7925480076524,
 864.4514630427126,
 634.3674697875977,
 902.899013331679,
 616.9498070888832,
 943.6310719974706,
 606.9902153640497,
 984.4900682793289,
 608.4226964731686,
 1016.5712630475155,
 628.5594872646644,
 832.5305576011783,
 678.4037192923124,
 836.6399911348938,
 713.0140631003458,
 840.7094528948675,
 747.4327937892226,
 845.3173352851243,
 783.2444863241227,
 798.448411034756,
 810.4822752905674,
 821.52595119789,
 815.9083807272989,
 846.5616054847592,
 819.6783642378009,
 869.2274279985272,
 811.3887379130379,
 889.3463194800205,
 802.0099687419954,
 687.574529053735,
 692.9587577444609,
 713.2432373547163,
 677.4494413782338,
 744.3002999727844,
 675.1158774954374,
 771.8614496950244,
 691.7659899602171,
 744.0824747554591,
 699.4614865662622,
 712.3424394951492,
 701.3301376592917,
 890.0987944055776,
 683.5445083868308,
 915.9064150950949,
 662.3624682817303,
 947.0360972920403,
 659.0972687768154,
 973.1807546146581,
 670.3108010213883,
 951.0253976290344,
 683.0369757980598,
 919.983334900903,
 685.78135230893,
 764.7854736828413,
 878.9628566054048,
 796.2538181992828,
 865.2678062564037,
 827.6573866547132,
 855.127679543417,
 849.4972264024078,
 859.0030477555072,
 870.626892590132,
 851.2510040783491,
 901.3761641705623,
 854.9777812019723,
 933.1902770996094,
 863.2005700283364,
 906.5760478035348,
 891.3267211914062,
 879.1349642394018,
 905.6950738625449,
 855.6820748751281,
 910.4573964603612,
 831.5472854864402,
 911.0099622382493,
 799.1247971331487,
 900.4238946633261,
 779.0058888920018,
 878.5098431696657,
 828.799624083472,
 874.2162545626281,
 850.6556368968526,
 874.4181748687245,
 872.6805199795083,
 869.658064545178,
 920.8728702732774,
 865.1561869636911,
 874.8449621982262,
 878.1269186051165,
 853.0499210044986,
 882.5210986528241,
 830.2377364361873,
 882.4428310706967]

	Predicted
Id
0	138.569743
1	330.279691
2	137.539780
3	353.279256
4	139.343905
...	...
137083	139.199152
137084	316.611529
137085	139.228152
137086	312.545877
137087	138.366608

137088 rows × 1 columns