2023/08/02 08:33:32 279082 |  TRACE   | Building model '20230802T083332-30ts_test_model_Malawi'..
2023/08/02 08:33:32 552802 |  TRACE   | Start the training:
2023/08/02 08:33:32 554996 |  TRACE   |  - Maximum steps: 1000000000
2023/08/02 08:33:32 557165 |  TRACE   |  - Validation steps: 500
2023/08/02 08:33:32 558790 |  TRACE   |  - Batch-size: 512
2023/08/02 08:33:32 560574 |  TRACE   |  - Loss function: CustomLoss(type=cross_entropy_focal)
2023/08/02 08:33:32 562103 |  TRACE   |  - Early-stopping tolerance: 5
2023/08/02 08:33:32 563787 |  TRACE   |  - Early-stopping improvement: 0.0001
2023/08/02 08:33:32 565489 |  TRACE   |  - Early-stopping metric: f1_class
2023/08/02 08:33:32 566801 |  TRACE   |  - Optimizer: Adam
2023/08/02 08:33:32 568119 |  TRACE   |  - Learning rate scheduling: LearningRateScheduler(scheduler=reduce_on_plateau, mode=max, lr=0.001)
2023/08/02 08:33:32 569567 |  TRACE   |  - Training balanced by: target_id
2023/08/02 08:33:32 570965 |  TRACE   |  - Device: cuda
2023/08/02 08:33:32 573268 |  TRACE   |  - N params: 1033819
2023/08/02 08:33:32 574828 |  TRACE   | 

2023/08/02 08:33:36 552468 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:33:36 560051 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:33:36 561816 |  TRACE   |  - Patience: 0/1
2023/08/02 08:33:36 563382 |  TRACE   | Initial validation score for 'f1_class': 0.20309
2023/08/02 08:33:36 564879 |  TRACE   | 

2023/08/02 08:33:36 566303 |  TRACE   | Training for 500 steps (current step: 0/1000000000)
2023/08/02 08:34:16 322818 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:34:16 332166 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:34:16 333805 |  TRACE   |  - Patience: 0/1
2023/08/02 08:34:16 335177 |  TRACE   | Early stopping:
2023/08/02 08:34:16 336282 |  TRACE   |  - New best f1_class: 0.48355
2023/08/02 08:34:16 337394 |  TRACE   |  - Improvement over previous best: 0.28046
2023/08/02 08:34:16 338685 |  TRACE   |  - Best model updated!
2023/08/02 08:34:16 393217 |  TRACE   | 

2023/08/02 08:34:16 394370 |  TRACE   | Training for 500 steps (current step: 500/1000000000)
2023/08/02 08:34:52 599705 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:34:52 608546 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:34:52 610258 |  TRACE   |  - Patience: 0/1
2023/08/02 08:34:52 611584 |  TRACE   | Early stopping:
2023/08/02 08:34:52 612896 |  TRACE   |  - New best f1_class: 0.49513
2023/08/02 08:34:52 614032 |  TRACE   |  - Improvement over previous best: 0.01158
2023/08/02 08:34:52 615126 |  TRACE   |  - Best model updated!
2023/08/02 08:34:52 666582 |  TRACE   | 

2023/08/02 08:34:52 668042 |  TRACE   | Training for 500 steps (current step: 1000/1000000000)
2023/08/02 08:35:33 673597 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:35:33 682359 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:35:33 683927 |  TRACE   |  - Patience: 1/1
2023/08/02 08:35:33 685309 |  TRACE   | Early stopping:
2023/08/02 08:35:33 686733 |  TRACE   |  - No improvement for f1_class: 0.48611 (current best: 0.49513)
2023/08/02 08:35:33 687902 |  TRACE   |  - Early stopping patience: 1/5
2023/08/02 08:35:33 688945 |  TRACE   | 

2023/08/02 08:35:33 690097 |  TRACE   | Training for 500 steps (current step: 1500/1000000000)
2023/08/02 08:36:09 904651 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:36:09 915074 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:36:09 917048 |  TRACE   |  - Patience: 0/1
2023/08/02 08:36:09 918547 |  TRACE   | Early stopping:
2023/08/02 08:36:09 919827 |  TRACE   |  - New best f1_class: 0.50834
2023/08/02 08:36:09 921128 |  TRACE   |  - Improvement over previous best: 0.01321
2023/08/02 08:36:09 922367 |  TRACE   |  - Best model updated!
2023/08/02 08:36:09 990909 |  TRACE   | 

2023/08/02 08:36:09 992209 |  TRACE   | Training for 500 steps (current step: 2000/1000000000)
2023/08/02 08:36:45 994292 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:36:46 002606 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:36:46 004442 |  TRACE   |  - Patience: 1/1
2023/08/02 08:36:46 005786 |  TRACE   | Early stopping:
2023/08/02 08:36:46 006943 |  TRACE   |  - No improvement for f1_class: 0.50370 (current best: 0.50834)
2023/08/02 08:36:46 008221 |  TRACE   |  - Early stopping patience: 1/5
2023/08/02 08:36:46 009318 |  TRACE   | 

2023/08/02 08:36:46 010364 |  TRACE   | Training for 500 steps (current step: 2500/1000000000)
2023/08/02 08:37:22 200051 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:37:22 208619 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:37:22 210617 |  TRACE   |  - Patience: 0/1
2023/08/02 08:37:22 211973 |  TRACE   | Early stopping:
2023/08/02 08:37:22 213272 |  TRACE   |  - New best f1_class: 0.51986
2023/08/02 08:37:22 214354 |  TRACE   |  - Improvement over previous best: 0.01152
2023/08/02 08:37:22 215689 |  TRACE   |  - Best model updated!
2023/08/02 08:37:22 271364 |  TRACE   | 

2023/08/02 08:37:22 272843 |  TRACE   | Training for 500 steps (current step: 3000/1000000000)
2023/08/02 08:37:58 430311 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:37:58 438719 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:37:58 440094 |  TRACE   |  - Patience: 1/1
2023/08/02 08:37:58 441310 |  TRACE   | Early stopping:
2023/08/02 08:37:58 442680 |  TRACE   |  - No improvement for f1_class: 0.48721 (current best: 0.51986)
2023/08/02 08:37:58 443858 |  TRACE   |  - Early stopping patience: 1/5
2023/08/02 08:37:58 444955 |  TRACE   | 

2023/08/02 08:37:58 445970 |  TRACE   | Training for 500 steps (current step: 3500/1000000000)
2023/08/02 08:38:34 435480 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:38:34 444047 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:38:34 445660 |  TRACE   |  - Patience: 0/1
2023/08/02 08:38:34 447107 |  TRACE   | Early stopping:
2023/08/02 08:38:34 448560 |  TRACE   |  - New best f1_class: 0.54629
2023/08/02 08:38:34 449801 |  TRACE   |  - Improvement over previous best: 0.02644
2023/08/02 08:38:34 450916 |  TRACE   |  - Best model updated!
2023/08/02 08:38:34 505319 |  TRACE   | 

2023/08/02 08:38:34 506709 |  TRACE   | Training for 500 steps (current step: 4000/1000000000)
2023/08/02 08:39:10 686121 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:39:10 694462 |  TRACE   |  - Current learning rate: 0.001
2023/08/02 08:39:10 695975 |  TRACE   |  - Patience: 1/1
2023/08/02 08:39:10 697362 |  TRACE   | Early stopping:
2023/08/02 08:39:10 698648 |  TRACE   |  - No improvement for f1_class: 0.46433 (current best: 0.54629)
2023/08/02 08:39:10 700021 |  TRACE   |  - Early stopping patience: 1/5
2023/08/02 08:39:10 701247 |  TRACE   | 

2023/08/02 08:39:10 702454 |  TRACE   | Training for 500 steps (current step: 4500/1000000000)
2023/08/02 08:39:46 554150 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:39:46 562814 |  TRACE   |  - Current learning rate: 0.0001
2023/08/02 08:39:46 564064 |  TRACE   |  - Patience: 0/1
2023/08/02 08:39:46 565125 |  TRACE   | Early stopping:
2023/08/02 08:39:46 566159 |  TRACE   |  - No improvement for f1_class: 0.48154 (current best: 0.54629)
2023/08/02 08:39:46 567163 |  TRACE   |  - Early stopping patience: 2/5
2023/08/02 08:39:46 568231 |  TRACE   | 

2023/08/02 08:39:46 569287 |  TRACE   | Training for 500 steps (current step: 5000/1000000000)
2023/08/02 08:40:22 221651 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:40:22 230309 |  TRACE   |  - Current learning rate: 0.0001
2023/08/02 08:40:22 231746 |  TRACE   |  - Patience: 1/1
2023/08/02 08:40:22 232929 |  TRACE   | Early stopping:
2023/08/02 08:40:22 233991 |  TRACE   |  - No improvement for f1_class: 0.47686 (current best: 0.54629)
2023/08/02 08:40:22 235094 |  TRACE   |  - Early stopping patience: 3/5
2023/08/02 08:40:22 236138 |  TRACE   | 

2023/08/02 08:40:22 237186 |  TRACE   | Training for 500 steps (current step: 5500/1000000000)
2023/08/02 08:40:59 123284 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:40:59 132428 |  TRACE   |  - Current learning rate: 1e-05
2023/08/02 08:40:59 134013 |  TRACE   |  - Patience: 0/1
2023/08/02 08:40:59 135243 |  TRACE   | Early stopping:
2023/08/02 08:40:59 136557 |  TRACE   |  - No improvement for f1_class: 0.47732 (current best: 0.54629)
2023/08/02 08:40:59 137753 |  TRACE   |  - Early stopping patience: 4/5
2023/08/02 08:40:59 139013 |  TRACE   | 

2023/08/02 08:40:59 140226 |  TRACE   | Training for 500 steps (current step: 6000/1000000000)
2023/08/02 08:41:35 013285 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/08/02 08:41:35 021654 |  TRACE   |  - Current learning rate: 1e-05
2023/08/02 08:41:35 023085 |  TRACE   |  - Patience: 1/1
2023/08/02 08:41:35 024452 |  TRACE   | Early stopping:
2023/08/02 08:41:35 025632 |  TRACE   |  - No improvement for f1_class: 0.46780 (current best: 0.54629)
2023/08/02 08:41:35 027028 |  TRACE   |  - Early stopping patience: 5/5
2023/08/02 08:41:35 028211 |  TRACE   |  - Early Stopping tolerance reached. Training stopped!
2023/08/02 08:41:35 029476 |  TRACE   | 

2023/08/02 08:41:35 118068 |  TRACE   | Evaluating:
2023/08/02 08:41:35 120242 |  TRACE   |  - Writing results to /home/vito/bontek/projects/TAP/HRL/Malawi/models/20230802T083332-30ts_test_model_Malawi/evaluation
2023/08/02 08:41:35 122233 |  TRACE   |  - Generating report..
2023/08/02 08:41:35 139129 |  TRACE   | 
              precision    recall  f1-score   support

     1-1-0-0       0.54      0.56      0.55       102
     1-2-0-0       0.55      0.63      0.59        92
     1-3-0-0       0.40      0.21      0.28        38

    accuracy                           0.53       232
   macro avg       0.49      0.47      0.47       232
weighted avg       0.52      0.53      0.52       232

2023/08/02 08:41:35 140351 |  TRACE   |  - Plotting confusion matrices..
2023/08/02 08:41:35 758347 |  TRACE   |  - Plotting F1 scores..

