2023/06/01 15:58:59 658014 |  TRACE   | Building model '20230601T155859-24ts_default-interspring'..
2023/06/01 15:59:41 081547 |  TRACE   | Start the training:
2023/06/01 15:59:41 090855 |  TRACE   |  - Maximum steps: 1000000000
2023/06/01 15:59:41 098389 |  TRACE   |  - Validation steps: 500
2023/06/01 15:59:41 101930 |  TRACE   |  - Batch-size: 512
2023/06/01 15:59:41 211543 |  TRACE   |  - Loss function: CustomLoss(type=cross_entropy_focal)
2023/06/01 15:59:41 217489 |  TRACE   |  - Early-stopping tolerance: 5
2023/06/01 15:59:41 222312 |  TRACE   |  - Early-stopping improvement: 0.0001
2023/06/01 15:59:41 227786 |  TRACE   |  - Early-stopping metric: f1_class
2023/06/01 15:59:41 233650 |  TRACE   |  - Optimizer: Adam
2023/06/01 15:59:41 238273 |  TRACE   |  - Learning rate scheduling: LearningRateScheduler(scheduler=reduce_on_plateau, mode=max, lr=0.001)
2023/06/01 15:59:41 242692 |  TRACE   |  - Training balanced by: target_id
2023/06/01 15:59:41 247305 |  TRACE   |  - Device: cuda
2023/06/01 15:59:41 359407 |  TRACE   |  - N params: 1035882
2023/06/01 15:59:41 362956 |  TRACE   | 

2023/06/01 15:59:57 292314 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 15:59:57 328361 |  TRACE   |  - Current learning rate: 0.001
2023/06/01 15:59:57 332541 |  TRACE   |  - Patience: 0/1
2023/06/01 15:59:57 339417 |  TRACE   | Initial validation score for 'f1_class': 0.01434
2023/06/01 15:59:57 344677 |  TRACE   | 

2023/06/01 15:59:57 348698 |  TRACE   | Training for 500 steps (current step: 0/1000000000)
2023/06/01 16:03:20 570977 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:03:20 614943 |  TRACE   |  - Current learning rate: 0.001
2023/06/01 16:03:20 618398 |  TRACE   |  - Patience: 0/1
2023/06/01 16:03:20 619661 |  TRACE   | Early stopping:
2023/06/01 16:03:20 621607 |  TRACE   |  - New best f1_class: 0.60240
2023/06/01 16:03:20 624082 |  TRACE   |  - Improvement over previous best: 0.58806
2023/06/01 16:03:20 626237 |  TRACE   |  - Best model updated!
2023/06/01 16:03:20 690209 |  TRACE   | 

2023/06/01 16:03:20 691974 |  TRACE   | Training for 500 steps (current step: 500/1000000000)
2023/06/01 16:06:03 647323 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:06:03 683070 |  TRACE   |  - Current learning rate: 0.001
2023/06/01 16:06:03 692573 |  TRACE   |  - Patience: 0/1
2023/06/01 16:06:03 694082 |  TRACE   | Early stopping:
2023/06/01 16:06:03 695388 |  TRACE   |  - New best f1_class: 0.64695
2023/06/01 16:06:03 697509 |  TRACE   |  - Improvement over previous best: 0.04455
2023/06/01 16:06:03 699888 |  TRACE   |  - Best model updated!
2023/06/01 16:06:03 762453 |  TRACE   | 

2023/06/01 16:06:03 764181 |  TRACE   | Training for 500 steps (current step: 1000/1000000000)
2023/06/01 16:08:18 965026 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:08:19 004360 |  TRACE   |  - Current learning rate: 0.001
2023/06/01 16:08:19 006990 |  TRACE   |  - Patience: 0/1
2023/06/01 16:08:19 008450 |  TRACE   | Early stopping:
2023/06/01 16:08:19 011631 |  TRACE   |  - New best f1_class: 0.66943
2023/06/01 16:08:19 013480 |  TRACE   |  - Improvement over previous best: 0.02247
2023/06/01 16:08:19 015007 |  TRACE   |  - Best model updated!
2023/06/01 16:08:19 084534 |  TRACE   | 

2023/06/01 16:08:19 086270 |  TRACE   | Training for 500 steps (current step: 1500/1000000000)
2023/06/01 16:10:44 923757 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:10:44 966407 |  TRACE   |  - Current learning rate: 0.001
2023/06/01 16:10:44 969452 |  TRACE   |  - Patience: 0/1
2023/06/01 16:10:45 077376 |  TRACE   | Early stopping:
2023/06/01 16:10:45 079716 |  TRACE   |  - New best f1_class: 0.69868
2023/06/01 16:10:45 082632 |  TRACE   |  - Improvement over previous best: 0.02926
2023/06/01 16:10:45 085333 |  TRACE   |  - Best model updated!
2023/06/01 16:10:45 166492 |  TRACE   | 

2023/06/01 16:10:45 168564 |  TRACE   | Training for 500 steps (current step: 2000/1000000000)
2023/06/01 16:12:28 607081 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:12:28 646448 |  TRACE   |  - Current learning rate: 0.001
2023/06/01 16:12:28 650557 |  TRACE   |  - Patience: 0/1
2023/06/01 16:12:28 653143 |  TRACE   | Early stopping:
2023/06/01 16:12:28 655540 |  TRACE   |  - New best f1_class: 0.71224
2023/06/01 16:12:28 657163 |  TRACE   |  - Improvement over previous best: 0.01356
2023/06/01 16:12:28 658803 |  TRACE   |  - Best model updated!
2023/06/01 16:12:28 730430 |  TRACE   | 

2023/06/01 16:12:28 732318 |  TRACE   | Training for 500 steps (current step: 2500/1000000000)
2023/06/01 16:14:42 411919 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:14:42 453570 |  TRACE   |  - Current learning rate: 0.001
2023/06/01 16:14:42 455466 |  TRACE   |  - Patience: 0/1
2023/06/01 16:14:42 456776 |  TRACE   | Early stopping:
2023/06/01 16:14:42 458115 |  TRACE   |  - New best f1_class: 0.72358
2023/06/01 16:14:42 459431 |  TRACE   |  - Improvement over previous best: 0.01133
2023/06/01 16:14:42 460897 |  TRACE   |  - Best model updated!
2023/06/01 16:14:42 557114 |  TRACE   | 

2023/06/01 16:14:42 560086 |  TRACE   | Training for 500 steps (current step: 3000/1000000000)
2023/06/01 16:18:58 155933 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:18:58 196026 |  TRACE   |  - Current learning rate: 0.001
2023/06/01 16:18:58 199218 |  TRACE   |  - Patience: 1/1
2023/06/01 16:18:58 202544 |  TRACE   | Early stopping:
2023/06/01 16:18:58 205953 |  TRACE   |  - No improvement for f1_class: 0.70482 (current best: 0.72358)
2023/06/01 16:18:58 209352 |  TRACE   |  - Early stopping patience: 1/5
2023/06/01 16:18:58 211258 |  TRACE   | 

2023/06/01 16:18:58 213514 |  TRACE   | Training for 500 steps (current step: 3500/1000000000)
2023/06/01 16:22:12 511416 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:22:12 551218 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:22:12 554879 |  TRACE   |  - Patience: 0/1
2023/06/01 16:22:12 556330 |  TRACE   | Early stopping:
2023/06/01 16:22:12 561979 |  TRACE   |  - No improvement for f1_class: 0.71929 (current best: 0.72358)
2023/06/01 16:22:12 564510 |  TRACE   |  - Early stopping patience: 2/5
2023/06/01 16:22:12 668380 |  TRACE   | 

2023/06/01 16:22:12 671243 |  TRACE   | Training for 500 steps (current step: 4000/1000000000)
2023/06/01 16:24:37 203754 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:24:37 243132 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:24:37 245805 |  TRACE   |  - Patience: 0/1
2023/06/01 16:24:37 248099 |  TRACE   | Early stopping:
2023/06/01 16:24:37 250345 |  TRACE   |  - New best f1_class: 0.75525
2023/06/01 16:24:37 252751 |  TRACE   |  - Improvement over previous best: 0.03168
2023/06/01 16:24:37 254959 |  TRACE   |  - Best model updated!
2023/06/01 16:24:37 339065 |  TRACE   | 

2023/06/01 16:24:37 341046 |  TRACE   | Training for 500 steps (current step: 4500/1000000000)
2023/06/01 16:29:06 359204 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:29:06 399269 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:29:06 402728 |  TRACE   |  - Patience: 0/1
2023/06/01 16:29:06 405968 |  TRACE   | Early stopping:
2023/06/01 16:29:06 408465 |  TRACE   |  - New best f1_class: 0.75647
2023/06/01 16:29:06 411104 |  TRACE   |  - Improvement over previous best: 0.00122
2023/06/01 16:29:06 414069 |  TRACE   |  - Best model updated!
2023/06/01 16:29:06 613247 |  TRACE   | 

2023/06/01 16:29:06 617073 |  TRACE   | Training for 500 steps (current step: 5000/1000000000)
2023/06/01 16:33:44 068940 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:33:44 108714 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:33:44 111638 |  TRACE   |  - Patience: 0/1
2023/06/01 16:33:44 114407 |  TRACE   | Early stopping:
2023/06/01 16:33:44 120227 |  TRACE   |  - New best f1_class: 0.76040
2023/06/01 16:33:44 122912 |  TRACE   |  - Improvement over previous best: 0.00393
2023/06/01 16:33:44 126151 |  TRACE   |  - Best model updated!
2023/06/01 16:33:44 221558 |  TRACE   | 

2023/06/01 16:33:44 225385 |  TRACE   | Training for 500 steps (current step: 5500/1000000000)
2023/06/01 16:36:20 840690 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:36:20 879986 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:36:20 888426 |  TRACE   |  - Patience: 0/1
2023/06/01 16:36:20 892771 |  TRACE   | Early stopping:
2023/06/01 16:36:20 896360 |  TRACE   |  - New best f1_class: 0.76213
2023/06/01 16:36:20 904226 |  TRACE   |  - Improvement over previous best: 0.00172
2023/06/01 16:36:21 009069 |  TRACE   |  - Best model updated!
2023/06/01 16:36:21 115131 |  TRACE   | 

2023/06/01 16:36:21 119585 |  TRACE   | Training for 500 steps (current step: 6000/1000000000)
2023/06/01 16:40:57 813630 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:40:57 853524 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:40:57 856731 |  TRACE   |  - Patience: 0/1
2023/06/01 16:40:57 858549 |  TRACE   | Early stopping:
2023/06/01 16:40:57 861820 |  TRACE   |  - New best f1_class: 0.76393
2023/06/01 16:40:57 865103 |  TRACE   |  - Improvement over previous best: 0.00181
2023/06/01 16:40:57 868138 |  TRACE   |  - Best model updated!
2023/06/01 16:40:57 947621 |  TRACE   | 

2023/06/01 16:40:57 950245 |  TRACE   | Training for 500 steps (current step: 6500/1000000000)
2023/06/01 16:46:41 489955 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:46:41 529875 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:46:41 532708 |  TRACE   |  - Patience: 0/1
2023/06/01 16:46:41 636448 |  TRACE   | Early stopping:
2023/06/01 16:46:41 639668 |  TRACE   |  - New best f1_class: 0.76587
2023/06/01 16:46:41 643135 |  TRACE   |  - Improvement over previous best: 0.00193
2023/06/01 16:46:41 645947 |  TRACE   |  - Best model updated!
2023/06/01 16:46:41 728472 |  TRACE   | 

2023/06/01 16:46:41 731214 |  TRACE   | Training for 500 steps (current step: 7000/1000000000)
2023/06/01 16:50:04 820574 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:50:04 859758 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:50:04 862785 |  TRACE   |  - Patience: 0/1
2023/06/01 16:50:04 864211 |  TRACE   | Early stopping:
2023/06/01 16:50:04 866133 |  TRACE   |  - New best f1_class: 0.76683
2023/06/01 16:50:04 867959 |  TRACE   |  - Improvement over previous best: 0.00096
2023/06/01 16:50:04 869891 |  TRACE   |  - Best model updated!
2023/06/01 16:50:04 946527 |  TRACE   | 

2023/06/01 16:50:04 948530 |  TRACE   | Training for 500 steps (current step: 7500/1000000000)
2023/06/01 16:52:39 021537 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:52:39 063206 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:52:39 066491 |  TRACE   |  - Patience: 1/1
2023/06/01 16:52:39 068463 |  TRACE   | Early stopping:
2023/06/01 16:52:39 070581 |  TRACE   |  - No improvement for f1_class: 0.76345 (current best: 0.76683)
2023/06/01 16:52:39 072414 |  TRACE   |  - Early stopping patience: 1/5
2023/06/01 16:52:39 073956 |  TRACE   | 

2023/06/01 16:52:39 075542 |  TRACE   | Training for 500 steps (current step: 8000/1000000000)
2023/06/01 16:55:34 191184 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:55:34 230683 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:55:34 233825 |  TRACE   |  - Patience: 0/1
2023/06/01 16:55:34 236177 |  TRACE   | Early stopping:
2023/06/01 16:55:34 238111 |  TRACE   |  - New best f1_class: 0.76816
2023/06/01 16:55:34 239960 |  TRACE   |  - Improvement over previous best: 0.00133
2023/06/01 16:55:34 241723 |  TRACE   |  - Best model updated!
2023/06/01 16:55:34 319428 |  TRACE   | 

2023/06/01 16:55:34 321181 |  TRACE   | Training for 500 steps (current step: 8500/1000000000)
2023/06/01 16:58:15 354151 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 16:58:15 393722 |  TRACE   |  - Current learning rate: 0.0001
2023/06/01 16:58:15 398398 |  TRACE   |  - Patience: 1/1
2023/06/01 16:58:15 400064 |  TRACE   | Early stopping:
2023/06/01 16:58:15 401883 |  TRACE   |  - No improvement for f1_class: 0.76298 (current best: 0.76816)
2023/06/01 16:58:15 404198 |  TRACE   |  - Early stopping patience: 1/5
2023/06/01 16:58:15 405852 |  TRACE   | 

2023/06/01 16:58:15 407697 |  TRACE   | Training for 500 steps (current step: 9000/1000000000)
2023/06/01 17:01:00 111797 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 17:01:00 152172 |  TRACE   |  - Current learning rate: 1e-05
2023/06/01 17:01:00 154271 |  TRACE   |  - Patience: 0/1
2023/06/01 17:01:00 155533 |  TRACE   | Early stopping:
2023/06/01 17:01:00 157695 |  TRACE   |  - No improvement for f1_class: 0.76469 (current best: 0.76816)
2023/06/01 17:01:00 160003 |  TRACE   |  - Early stopping patience: 2/5
2023/06/01 17:01:00 161627 |  TRACE   | 

2023/06/01 17:01:00 163457 |  TRACE   | Training for 500 steps (current step: 9500/1000000000)
2023/06/01 17:03:38 064332 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 17:03:38 103834 |  TRACE   |  - Current learning rate: 1e-05
2023/06/01 17:03:38 107977 |  TRACE   |  - Patience: 0/1
2023/06/01 17:03:38 109573 |  TRACE   | Early stopping:
2023/06/01 17:03:38 111459 |  TRACE   |  - New best f1_class: 0.77150
2023/06/01 17:03:38 113419 |  TRACE   |  - Improvement over previous best: 0.00334
2023/06/01 17:03:38 115378 |  TRACE   |  - Best model updated!
2023/06/01 17:03:38 188441 |  TRACE   | 

2023/06/01 17:03:38 190013 |  TRACE   | Training for 500 steps (current step: 10000/1000000000)
2023/06/01 17:06:51 320320 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 17:06:51 359801 |  TRACE   |  - Current learning rate: 1e-05
2023/06/01 17:06:51 363932 |  TRACE   |  - Patience: 1/1
2023/06/01 17:06:51 365528 |  TRACE   | Early stopping:
2023/06/01 17:06:51 367528 |  TRACE   |  - No improvement for f1_class: 0.77035 (current best: 0.77150)
2023/06/01 17:06:51 368783 |  TRACE   |  - Early stopping patience: 1/5
2023/06/01 17:06:51 370594 |  TRACE   | 

2023/06/01 17:06:51 372348 |  TRACE   | Training for 500 steps (current step: 10500/1000000000)
2023/06/01 17:09:57 370483 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 17:09:57 409798 |  TRACE   |  - Current learning rate: 1.0000000000000002e-06
2023/06/01 17:09:57 413046 |  TRACE   |  - Patience: 0/1
2023/06/01 17:09:57 416151 |  TRACE   | Early stopping:
2023/06/01 17:09:57 419255 |  TRACE   |  - No improvement for f1_class: 0.77055 (current best: 0.77150)
2023/06/01 17:09:57 421580 |  TRACE   |  - Early stopping patience: 2/5
2023/06/01 17:09:57 423133 |  TRACE   | 

2023/06/01 17:09:57 529321 |  TRACE   | Training for 500 steps (current step: 11000/1000000000)
2023/06/01 17:12:49 427649 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 17:12:49 467546 |  TRACE   |  - Current learning rate: 1.0000000000000002e-06
2023/06/01 17:12:49 470243 |  TRACE   |  - Patience: 1/1
2023/06/01 17:12:49 471853 |  TRACE   | Early stopping:
2023/06/01 17:12:49 473908 |  TRACE   |  - No improvement for f1_class: 0.77098 (current best: 0.77150)
2023/06/01 17:12:49 476201 |  TRACE   |  - Early stopping patience: 3/5
2023/06/01 17:12:49 477834 |  TRACE   | 

2023/06/01 17:12:49 479604 |  TRACE   | Training for 500 steps (current step: 11500/1000000000)
2023/06/01 17:15:57 848373 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 17:15:57 888608 |  TRACE   |  - Current learning rate: 1.0000000000000002e-07
2023/06/01 17:15:57 891456 |  TRACE   |  - Patience: 0/1
2023/06/01 17:15:57 892880 |  TRACE   | Early stopping:
2023/06/01 17:15:57 894969 |  TRACE   |  - No improvement for f1_class: 0.77149 (current best: 0.77150)
2023/06/01 17:15:57 897363 |  TRACE   |  - Early stopping patience: 4/5
2023/06/01 17:15:57 898604 |  TRACE   | 

2023/06/01 17:15:57 900620 |  TRACE   | Training for 500 steps (current step: 12000/1000000000)
2023/06/01 17:18:36 242460 |  TRACE   | Learning rate scheduler: reduce_on_plateau
2023/06/01 17:18:36 283336 |  TRACE   |  - Current learning rate: 1.0000000000000002e-07
2023/06/01 17:18:36 286130 |  TRACE   |  - Patience: 1/1
2023/06/01 17:18:36 288926 |  TRACE   | Early stopping:
2023/06/01 17:18:36 291757 |  TRACE   |  - No improvement for f1_class: 0.77155 (current best: 0.77150)
2023/06/01 17:18:36 294460 |  TRACE   |  - Early stopping patience: 5/5
2023/06/01 17:18:36 297056 |  TRACE   |  - Early Stopping tolerance reached. Training stopped!
2023/06/01 17:18:36 299303 |  TRACE   | 

2023/06/01 17:18:49 493940 |  TRACE   | Evaluating:
2023/06/01 17:18:49 497816 |  TRACE   |  - Writing results to /home/vito/couchard/vito_crop_classification/models/20230601T155859-24ts_default-interspring/evaluation
2023/06/01 17:18:49 503673 |  TRACE   |  - Generating report..
2023/06/01 17:18:55 731768 |  TRACE   | 
              precision    recall  f1-score   support

     1-1-1-0       0.82      0.77      0.80     13373
     1-1-2-0       0.79      0.81      0.80      9881
     1-1-3-0       0.91      0.93      0.92      7802
     1-1-4-0       0.99      0.94      0.97       141
     1-1-5-0       0.70      0.71      0.71     10938
     1-2-1-0       0.69      0.67      0.68      3201
     1-2-2-0       0.68      0.75      0.71      3112
     1-3-1-0       0.77      0.82      0.80      2375
     1-3-2-0       0.93      0.92      0.93      1630
     1-4-1-0       0.85      0.86      0.86      1750
     1-4-2-0       0.82      0.87      0.84      1009
     1-4-3-0       0.93      0.92      0.93      4098
     1-4-4-0       0.70      0.72      0.71       595
     1-5-0-0       0.89      0.84      0.86     20947
     2-1-0-0       0.69      0.70      0.69      1553
     2-2-0-0       0.60      0.72      0.66      1429
     2-3-1-0       0.56      0.66      0.61      3141
     2-3-2-0       0.45      0.46      0.46      1032

    accuracy                           0.80     88007
   macro avg       0.77      0.78      0.77     88007
weighted avg       0.80      0.80      0.80     88007

2023/06/01 17:18:55 736161 |  TRACE   |  - Plotting confusion matrices..
2023/06/01 17:19:00 207381 |  TRACE   |  - Plotting F1 scores..

