2023/05/04 17:59:21 547360 |  TRACE   | Start the training:
2023/05/04 17:59:21 550145 |  TRACE   |  - Model: InpaintingModel(20230504T175919_cnn_transformer)
2023/05/04 17:59:21 552110 |  TRACE   |  - Number of trainable parameters: 27,799,425
2023/05/04 17:59:21 553373 |  TRACE   |  - Number of frozen parameters: 0
2023/05/04 17:59:21 554544 |  TRACE   |  - Validation steps: 300
2023/05/04 17:59:21 555697 |  TRACE   |  - Maximum steps: None
2023/05/04 17:59:21 557132 |  TRACE   |  - Batch-size: 12
2023/05/04 17:59:21 557926 |  TRACE   |  - Early-stopping tolerance: 10
2023/05/04 17:59:21 559061 |  TRACE   |  - Early-stopping improvement: 0.0
2023/05/04 17:59:21 560383 |  TRACE   |  - Optimizer: Adam
2023/05/04 17:59:21 561765 |  TRACE   |  - Learning rate scheduling: LearningRateScheduler(lr=0.0, warmup=600, cycle=0)
2023/05/04 17:59:21 562971 |  TRACE   |  - Loss function: VITOLoss(mae_masked=1.00, mae_unmasked=1.00, ssim=0.50, mae_time_reg=0.10, ssim_time_reg=0.10)
2023/05/04 17:59:21 564151 |  TRACE   |  - Device: cuda
2023/05/04 17:59:21 565211 |  TRACE   | 

2023/05/04 18:00:11 579670 |  TRACE   | Initial validation loss: 1.20659
2023/05/04 18:00:11 607973 |  TRACE   | 

2023/05/04 18:00:18 540850 |  TRACE   | Training for 300 steps (current step: 0)
2023/05/04 18:11:55 140695 |  TRACE   | Early stopping:
2023/05/04 18:11:55 156804 |  TRACE   |  - New best loss: 0.39022
2023/05/04 18:11:55 157746 |  TRACE   |  - Improvement over previous best: -0.81637
2023/05/04 18:11:55 158490 |  TRACE   |  - Best model updated!
2023/05/04 18:12:01 670285 |  TRACE   | 

2023/05/04 18:12:01 671352 |  TRACE   | Training for 300 steps (current step: 300)
2023/05/04 18:24:35 107991 |  TRACE   | Learning rate scheduler:
2023/05/04 18:24:35 108948 |  TRACE   |  - Validation improved (0.39022 -> 0.31374), resetting plateau patience
2023/05/04 18:24:35 109791 |  TRACE   | Early stopping:
2023/05/04 18:24:35 110579 |  TRACE   |  - New best loss: 0.31374
2023/05/04 18:24:35 111383 |  TRACE   |  - Improvement over previous best: -0.07648
2023/05/04 18:24:35 112068 |  TRACE   |  - Best model updated!
2023/05/04 18:24:42 289982 |  TRACE   | 

2023/05/04 18:24:42 291246 |  TRACE   | Training for 300 steps (current step: 600)
2023/05/04 18:37:15 443542 |  TRACE   | Learning rate scheduler:
2023/05/04 18:37:15 446277 |  TRACE   |  - Validation improved (0.31374 -> 0.26334), resetting plateau patience
2023/05/04 18:37:15 447754 |  TRACE   | Early stopping:
2023/05/04 18:37:15 449380 |  TRACE   |  - New best loss: 0.26334
2023/05/04 18:37:15 451428 |  TRACE   |  - Improvement over previous best: -0.05040
2023/05/04 18:37:15 452729 |  TRACE   |  - Best model updated!
2023/05/04 18:37:24 113119 |  TRACE   | 

2023/05/04 18:37:24 115451 |  TRACE   | Training for 300 steps (current step: 900)
2023/05/04 18:50:10 723144 |  TRACE   | Learning rate scheduler:
2023/05/04 18:50:10 749399 |  TRACE   |  - Validation did not improve (0.26334 -> 0.26941), increasing plateau patience (1/3)
2023/05/04 18:50:10 750305 |  TRACE   | Early stopping:
2023/05/04 18:50:10 751193 |  TRACE   |  - No improvement: 0.26941 (current best: 0.26334)
2023/05/04 18:50:10 751911 |  TRACE   |  - Patience: 1/10
2023/05/04 18:50:10 752693 |  TRACE   | 

2023/05/04 18:50:10 753408 |  TRACE   | Training for 300 steps (current step: 1200)
2023/05/04 19:02:38 758647 |  TRACE   | Learning rate scheduler:
2023/05/04 19:02:38 794383 |  TRACE   |  - Validation improved (0.26334 -> 0.24480), resetting plateau patience
2023/05/04 19:02:38 795426 |  TRACE   | Early stopping:
2023/05/04 19:02:38 796203 |  TRACE   |  - New best loss: 0.24480
2023/05/04 19:02:38 796889 |  TRACE   |  - Improvement over previous best: -0.01854
2023/05/04 19:02:38 797652 |  TRACE   |  - Best model updated!
2023/05/04 19:02:45 383697 |  TRACE   | 

2023/05/04 19:02:45 384620 |  TRACE   | Training for 300 steps (current step: 1500)
2023/05/04 19:15:41 536476 |  TRACE   | Learning rate scheduler:
2023/05/04 19:15:41 563896 |  TRACE   |  - Validation improved (0.24480 -> 0.22342), resetting plateau patience
2023/05/04 19:15:41 564909 |  TRACE   | Early stopping:
2023/05/04 19:15:41 566290 |  TRACE   |  - New best loss: 0.22342
2023/05/04 19:15:41 567293 |  TRACE   |  - Improvement over previous best: -0.02139
2023/05/04 19:15:41 568204 |  TRACE   |  - Best model updated!
2023/05/04 19:15:48 684996 |  TRACE   | 

2023/05/04 19:15:48 686346 |  TRACE   | Training for 300 steps (current step: 1800)
2023/05/04 19:28:00 645477 |  TRACE   | Learning rate scheduler:
2023/05/04 19:28:00 697056 |  TRACE   |  - Validation improved (0.22342 -> 0.21540), resetting plateau patience
2023/05/04 19:28:00 698516 |  TRACE   | Early stopping:
2023/05/04 19:28:00 699722 |  TRACE   |  - New best loss: 0.21540
2023/05/04 19:28:00 700778 |  TRACE   |  - Improvement over previous best: -0.00802
2023/05/04 19:28:00 701784 |  TRACE   |  - Best model updated!
2023/05/04 19:28:07 895696 |  TRACE   | 

2023/05/04 19:28:07 897356 |  TRACE   | Training for 300 steps (current step: 2100)
2023/05/04 19:40:36 706263 |  TRACE   | Learning rate scheduler:
2023/05/04 19:40:36 725232 |  TRACE   |  - Validation improved (0.21540 -> 0.20538), resetting plateau patience
2023/05/04 19:40:36 726162 |  TRACE   | Early stopping:
2023/05/04 19:40:36 726916 |  TRACE   |  - New best loss: 0.20538
2023/05/04 19:40:36 727822 |  TRACE   |  - Improvement over previous best: -0.01002
2023/05/04 19:40:36 728613 |  TRACE   |  - Best model updated!
2023/05/04 19:40:43 818151 |  TRACE   | 

2023/05/04 19:40:43 819231 |  TRACE   | Training for 300 steps (current step: 2400)
2023/05/04 19:53:21 200104 |  TRACE   | Learning rate scheduler:
2023/05/04 19:53:21 225409 |  TRACE   |  - Validation did not improve (0.20538 -> 0.22066), increasing plateau patience (1/3)
2023/05/04 19:53:21 226261 |  TRACE   | Early stopping:
2023/05/04 19:53:21 227191 |  TRACE   |  - No improvement: 0.22066 (current best: 0.20538)
2023/05/04 19:53:21 228006 |  TRACE   |  - Patience: 1/10
2023/05/04 19:53:21 228841 |  TRACE   | 

2023/05/04 19:53:21 229657 |  TRACE   | Training for 300 steps (current step: 2700)
2023/05/04 20:06:04 695372 |  TRACE   | Learning rate scheduler:
2023/05/04 20:06:04 696917 |  TRACE   |  - Validation did not improve (0.20538 -> 0.20638), increasing plateau patience (2/3)
2023/05/04 20:06:04 698055 |  TRACE   | Early stopping:
2023/05/04 20:06:04 699599 |  TRACE   |  - No improvement: 0.20638 (current best: 0.20538)
2023/05/04 20:06:04 805037 |  TRACE   |  - Patience: 2/10
2023/05/04 20:06:04 806400 |  TRACE   | 

2023/05/04 20:06:04 807655 |  TRACE   | Training for 300 steps (current step: 3000)
2023/05/04 20:18:54 952566 |  TRACE   | Learning rate scheduler:
2023/05/04 20:18:54 977744 |  TRACE   |  - Validation improved (0.20538 -> 0.20221), resetting plateau patience
2023/05/04 20:18:54 978554 |  TRACE   | Early stopping:
2023/05/04 20:18:54 979304 |  TRACE   |  - New best loss: 0.20221
2023/05/04 20:18:54 980029 |  TRACE   |  - Improvement over previous best: -0.00318
2023/05/04 20:18:54 980720 |  TRACE   |  - Best model updated!
2023/05/04 20:19:03 696227 |  TRACE   | 

2023/05/04 20:19:03 697255 |  TRACE   | Training for 300 steps (current step: 3300)
2023/05/04 20:31:46 262725 |  TRACE   | Learning rate scheduler:
2023/05/04 20:31:46 335443 |  TRACE   |  - Validation did not improve (0.20221 -> 0.20766), increasing plateau patience (1/3)
2023/05/04 20:31:46 336354 |  TRACE   | Early stopping:
2023/05/04 20:31:46 337136 |  TRACE   |  - No improvement: 0.20766 (current best: 0.20221)
2023/05/04 20:31:46 337930 |  TRACE   |  - Patience: 1/10
2023/05/04 20:31:46 338680 |  TRACE   | 

2023/05/04 20:31:46 339380 |  TRACE   | Training for 300 steps (current step: 3600)
2023/05/04 20:44:03 628843 |  TRACE   | Learning rate scheduler:
2023/05/04 20:44:03 654264 |  TRACE   |  - Validation improved (0.20221 -> 0.18596), resetting plateau patience
2023/05/04 20:44:03 655221 |  TRACE   | Early stopping:
2023/05/04 20:44:03 655997 |  TRACE   |  - New best loss: 0.18596
2023/05/04 20:44:03 656860 |  TRACE   |  - Improvement over previous best: -0.01624
2023/05/04 20:44:03 657569 |  TRACE   |  - Best model updated!
2023/05/04 20:44:12 247034 |  TRACE   | 

2023/05/04 20:44:12 248732 |  TRACE   | Training for 300 steps (current step: 3900)
2023/05/04 20:56:48 115178 |  TRACE   | Learning rate scheduler:
2023/05/04 20:56:48 134088 |  TRACE   |  - Validation did not improve (0.18596 -> 0.18981), increasing plateau patience (1/3)
2023/05/04 20:56:48 134935 |  TRACE   | Early stopping:
2023/05/04 20:56:48 135686 |  TRACE   |  - No improvement: 0.18981 (current best: 0.18596)
2023/05/04 20:56:48 136394 |  TRACE   |  - Patience: 1/10
2023/05/04 20:56:48 137099 |  TRACE   | 

2023/05/04 20:56:48 137837 |  TRACE   | Training for 300 steps (current step: 4200)
2023/05/04 21:09:12 730432 |  TRACE   | Learning rate scheduler:
2023/05/04 21:09:12 731716 |  TRACE   |  - Validation did not improve (0.18596 -> 0.19468), increasing plateau patience (2/3)
2023/05/04 21:09:12 732603 |  TRACE   | Early stopping:
2023/05/04 21:09:12 733376 |  TRACE   |  - No improvement: 0.19468 (current best: 0.18596)
2023/05/04 21:09:12 734137 |  TRACE   |  - Patience: 2/10
2023/05/04 21:09:12 734862 |  TRACE   | 

2023/05/04 21:09:12 735526 |  TRACE   | Training for 300 steps (current step: 4500)
2023/05/04 21:21:56 194426 |  TRACE   | Learning rate scheduler:
2023/05/04 21:21:56 230416 |  TRACE   |  - Validation did not improve (0.18596 -> 0.19001), increasing plateau patience (3/3)
2023/05/04 21:21:56 231752 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/04 21:21:56 232692 |  TRACE   |  - New learning rates: lr_min=2.5e-05 - lr_max=5e-05
2023/05/04 21:21:56 233551 |  TRACE   | Early stopping:
2023/05/04 21:21:56 234289 |  TRACE   |  - No improvement: 0.19001 (current best: 0.18596)
2023/05/04 21:21:56 235309 |  TRACE   |  - Patience: 3/10
2023/05/04 21:21:56 236088 |  TRACE   | 

2023/05/04 21:21:56 237292 |  TRACE   | Training for 300 steps (current step: 4800)
2023/05/04 21:34:36 700568 |  TRACE   | Learning rate scheduler:
2023/05/04 21:34:36 702037 |  TRACE   |  - Validation improved (0.18596 -> 0.17810), resetting plateau patience
2023/05/04 21:34:36 702878 |  TRACE   | Early stopping:
2023/05/04 21:34:36 703644 |  TRACE   |  - New best loss: 0.17810
2023/05/04 21:34:36 704348 |  TRACE   |  - Improvement over previous best: -0.00786
2023/05/04 21:34:36 705181 |  TRACE   |  - Best model updated!
2023/05/04 21:34:45 092012 |  TRACE   | 

2023/05/04 21:34:45 093198 |  TRACE   | Training for 300 steps (current step: 5100)
2023/05/04 21:47:29 385131 |  TRACE   | Learning rate scheduler:
2023/05/04 21:47:29 413886 |  TRACE   |  - Validation did not improve (0.17810 -> 0.17881), increasing plateau patience (1/3)
2023/05/04 21:47:29 414829 |  TRACE   | Early stopping:
2023/05/04 21:47:29 415573 |  TRACE   |  - No improvement: 0.17881 (current best: 0.17810)
2023/05/04 21:47:29 416352 |  TRACE   |  - Patience: 1/10
2023/05/04 21:47:29 417060 |  TRACE   | 

2023/05/04 21:47:29 417823 |  TRACE   | Training for 300 steps (current step: 5400)
2023/05/04 21:59:30 752123 |  TRACE   | Learning rate scheduler:
2023/05/04 21:59:30 785439 |  TRACE   |  - Validation did not improve (0.17810 -> 0.18774), increasing plateau patience (2/3)
2023/05/04 21:59:30 786359 |  TRACE   | Early stopping:
2023/05/04 21:59:30 787158 |  TRACE   |  - No improvement: 0.18774 (current best: 0.17810)
2023/05/04 21:59:30 788043 |  TRACE   |  - Patience: 2/10
2023/05/04 21:59:30 788801 |  TRACE   | 

2023/05/04 21:59:30 789523 |  TRACE   | Training for 300 steps (current step: 5700)
2023/05/04 22:11:51 396612 |  TRACE   | Learning rate scheduler:
2023/05/04 22:11:51 421462 |  TRACE   |  - Validation did not improve (0.17810 -> 0.18502), increasing plateau patience (3/3)
2023/05/04 22:11:51 422441 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/04 22:11:51 423303 |  TRACE   |  - New learning rates: lr_min=1.25e-05 - lr_max=2.5e-05
2023/05/04 22:11:51 424205 |  TRACE   | Early stopping:
2023/05/04 22:11:51 425009 |  TRACE   |  - No improvement: 0.18502 (current best: 0.17810)
2023/05/04 22:11:51 425815 |  TRACE   |  - Patience: 3/10
2023/05/04 22:11:51 427071 |  TRACE   | 

2023/05/04 22:11:51 427852 |  TRACE   | Training for 300 steps (current step: 6000)
2023/05/04 22:24:04 722273 |  TRACE   | Learning rate scheduler:
2023/05/04 22:24:04 748679 |  TRACE   |  - Validation improved (0.17810 -> 0.17397), resetting plateau patience
2023/05/04 22:24:04 749506 |  TRACE   | Early stopping:
2023/05/04 22:24:04 750250 |  TRACE   |  - New best loss: 0.17397
2023/05/04 22:24:04 751026 |  TRACE   |  - Improvement over previous best: -0.00413
2023/05/04 22:24:04 751827 |  TRACE   |  - Best model updated!
2023/05/04 22:24:12 862782 |  TRACE   | 

2023/05/04 22:24:12 864074 |  TRACE   | Training for 300 steps (current step: 6300)
2023/05/04 22:36:33 708646 |  TRACE   | Learning rate scheduler:
2023/05/04 22:36:33 733996 |  TRACE   |  - Validation did not improve (0.17397 -> 0.17534), increasing plateau patience (1/3)
2023/05/04 22:36:33 735088 |  TRACE   | Early stopping:
2023/05/04 22:36:33 735956 |  TRACE   |  - No improvement: 0.17534 (current best: 0.17397)
2023/05/04 22:36:33 736729 |  TRACE   |  - Patience: 1/10
2023/05/04 22:36:33 737518 |  TRACE   | 

2023/05/04 22:36:33 738316 |  TRACE   | Training for 300 steps (current step: 6600)
2023/05/04 22:48:50 338778 |  TRACE   | Learning rate scheduler:
2023/05/04 22:48:50 390304 |  TRACE   |  - Validation did not improve (0.17397 -> 0.17413), increasing plateau patience (2/3)
2023/05/04 22:48:50 391274 |  TRACE   | Early stopping:
2023/05/04 22:48:50 392430 |  TRACE   |  - No improvement: 0.17413 (current best: 0.17397)
2023/05/04 22:48:50 393604 |  TRACE   |  - Patience: 2/10
2023/05/04 22:48:50 394749 |  TRACE   | 

2023/05/04 22:48:50 395809 |  TRACE   | Training for 300 steps (current step: 6900)
2023/05/04 23:01:55 086838 |  TRACE   | Learning rate scheduler:
2023/05/04 23:01:55 088473 |  TRACE   |  - Validation did not improve (0.17397 -> 0.17636), increasing plateau patience (3/3)
2023/05/04 23:01:55 089348 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/04 23:01:55 090149 |  TRACE   |  - New learning rates: lr_min=6.25e-06 - lr_max=1.25e-05
2023/05/04 23:01:55 090894 |  TRACE   | Early stopping:
2023/05/04 23:01:55 091599 |  TRACE   |  - No improvement: 0.17636 (current best: 0.17397)
2023/05/04 23:01:55 092352 |  TRACE   |  - Patience: 3/10
2023/05/04 23:01:55 093168 |  TRACE   | 

2023/05/04 23:01:55 093912 |  TRACE   | Training for 300 steps (current step: 7200)
2023/05/04 23:14:38 610225 |  TRACE   | Learning rate scheduler:
2023/05/04 23:14:38 611686 |  TRACE   |  - Validation improved (0.17397 -> 0.17188), resetting plateau patience
2023/05/04 23:14:38 612692 |  TRACE   | Early stopping:
2023/05/04 23:14:38 613559 |  TRACE   |  - New best loss: 0.17188
2023/05/04 23:14:38 614358 |  TRACE   |  - Improvement over previous best: -0.00209
2023/05/04 23:14:38 615774 |  TRACE   |  - Best model updated!
2023/05/04 23:14:48 548631 |  TRACE   | 

2023/05/04 23:14:48 549800 |  TRACE   | Training for 300 steps (current step: 7500)
2023/05/04 23:27:56 707417 |  TRACE   | Learning rate scheduler:
2023/05/04 23:27:56 708758 |  TRACE   |  - Validation did not improve (0.17188 -> 0.17265), increasing plateau patience (1/3)
2023/05/04 23:27:56 709577 |  TRACE   | Early stopping:
2023/05/04 23:27:56 710265 |  TRACE   |  - No improvement: 0.17265 (current best: 0.17188)
2023/05/04 23:27:56 711018 |  TRACE   |  - Patience: 1/10
2023/05/04 23:27:56 711820 |  TRACE   | 

2023/05/04 23:27:56 712586 |  TRACE   | Training for 300 steps (current step: 7800)
2023/05/04 23:40:26 545187 |  TRACE   | Learning rate scheduler:
2023/05/04 23:40:26 564393 |  TRACE   |  - Validation improved (0.17188 -> 0.17136), resetting plateau patience
2023/05/04 23:40:26 565465 |  TRACE   | Early stopping:
2023/05/04 23:40:26 566422 |  TRACE   |  - New best loss: 0.17136
2023/05/04 23:40:26 567267 |  TRACE   |  - Improvement over previous best: -0.00053
2023/05/04 23:40:26 568088 |  TRACE   |  - Best model updated!
2023/05/04 23:40:35 522805 |  TRACE   | 

2023/05/04 23:40:35 524348 |  TRACE   | Training for 300 steps (current step: 8100)
2023/05/04 23:52:59 623034 |  TRACE   | Learning rate scheduler:
2023/05/04 23:52:59 624664 |  TRACE   |  - Validation improved (0.17136 -> 0.17035), resetting plateau patience
2023/05/04 23:52:59 625848 |  TRACE   | Early stopping:
2023/05/04 23:52:59 629169 |  TRACE   |  - New best loss: 0.17035
2023/05/04 23:52:59 630467 |  TRACE   |  - Improvement over previous best: -0.00101
2023/05/04 23:52:59 631548 |  TRACE   |  - Best model updated!
2023/05/04 23:53:11 266633 |  TRACE   | 

2023/05/04 23:53:11 268279 |  TRACE   | Training for 300 steps (current step: 8400)
2023/05/05 00:05:38 295545 |  TRACE   | Learning rate scheduler:
2023/05/05 00:05:38 313032 |  TRACE   |  - Validation improved (0.17035 -> 0.17013), resetting plateau patience
2023/05/05 00:05:38 313990 |  TRACE   | Early stopping:
2023/05/05 00:05:38 314807 |  TRACE   |  - New best loss: 0.17013
2023/05/05 00:05:38 315622 |  TRACE   |  - Improvement over previous best: -0.00022
2023/05/05 00:05:38 316421 |  TRACE   |  - Best model updated!
2023/05/05 00:05:46 363009 |  TRACE   | 

2023/05/05 00:05:46 364661 |  TRACE   | Training for 300 steps (current step: 8700)
2023/05/05 00:18:51 657262 |  TRACE   | Learning rate scheduler:
2023/05/05 00:18:51 674742 |  TRACE   |  - Validation did not improve (0.17013 -> 0.17212), increasing plateau patience (1/3)
2023/05/05 00:18:51 675673 |  TRACE   | Early stopping:
2023/05/05 00:18:51 676605 |  TRACE   |  - No improvement: 0.17212 (current best: 0.17013)
2023/05/05 00:18:51 677405 |  TRACE   |  - Patience: 1/10
2023/05/05 00:18:51 678219 |  TRACE   | 

2023/05/05 00:18:51 679049 |  TRACE   | Training for 300 steps (current step: 9000)
2023/05/05 00:31:10 981176 |  TRACE   | Learning rate scheduler:
2023/05/05 00:31:10 982335 |  TRACE   |  - Validation improved (0.17013 -> 0.16972), resetting plateau patience
2023/05/05 00:31:10 983136 |  TRACE   | Early stopping:
2023/05/05 00:31:10 983907 |  TRACE   |  - New best loss: 0.16972
2023/05/05 00:31:10 984660 |  TRACE   |  - Improvement over previous best: -0.00041
2023/05/05 00:31:10 985367 |  TRACE   |  - Best model updated!
2023/05/05 00:31:19 242768 |  TRACE   | 

2023/05/05 00:31:19 244134 |  TRACE   | Training for 300 steps (current step: 9300)
2023/05/05 00:44:10 497343 |  TRACE   | Learning rate scheduler:
2023/05/05 00:44:10 528242 |  TRACE   |  - Validation did not improve (0.16972 -> 0.16973), increasing plateau patience (1/3)
2023/05/05 00:44:10 530819 |  TRACE   | Early stopping:
2023/05/05 00:44:10 531718 |  TRACE   |  - No improvement: 0.16973 (current best: 0.16972)
2023/05/05 00:44:10 532684 |  TRACE   |  - Patience: 1/10
2023/05/05 00:44:10 533503 |  TRACE   | 

2023/05/05 00:44:10 534301 |  TRACE   | Training for 300 steps (current step: 9600)
2023/05/05 00:56:39 981683 |  TRACE   | Learning rate scheduler:
2023/05/05 00:56:39 983122 |  TRACE   |  - Validation did not improve (0.16972 -> 0.17020), increasing plateau patience (2/3)
2023/05/05 00:56:39 984022 |  TRACE   | Early stopping:
2023/05/05 00:56:39 984951 |  TRACE   |  - No improvement: 0.17020 (current best: 0.16972)
2023/05/05 00:56:39 985656 |  TRACE   |  - Patience: 2/10
2023/05/05 00:56:39 986437 |  TRACE   | 

2023/05/05 00:56:39 987214 |  TRACE   | Training for 300 steps (current step: 9900)
2023/05/05 01:08:43 001139 |  TRACE   | Learning rate scheduler:
2023/05/05 01:08:43 029563 |  TRACE   |  - Validation did not improve (0.16972 -> 0.17062), increasing plateau patience (3/3)
2023/05/05 01:08:43 030546 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/05 01:08:43 031301 |  TRACE   |  - New learning rates: lr_min=3.125e-06 - lr_max=6.25e-06
2023/05/05 01:08:43 032069 |  TRACE   | Early stopping:
2023/05/05 01:08:43 032867 |  TRACE   |  - No improvement: 0.17062 (current best: 0.16972)
2023/05/05 01:08:43 034194 |  TRACE   |  - Patience: 3/10
2023/05/05 01:08:43 034952 |  TRACE   | 

2023/05/05 01:08:43 035724 |  TRACE   | Training for 300 steps (current step: 10200)
2023/05/05 01:21:46 710215 |  TRACE   | Learning rate scheduler:
2023/05/05 01:21:46 712666 |  TRACE   |  - Validation improved (0.16972 -> 0.16881), resetting plateau patience
2023/05/05 01:21:46 713896 |  TRACE   | Early stopping:
2023/05/05 01:21:46 714663 |  TRACE   |  - New best loss: 0.16881
2023/05/05 01:21:46 715463 |  TRACE   |  - Improvement over previous best: -0.00091
2023/05/05 01:21:46 716135 |  TRACE   |  - Best model updated!
2023/05/05 01:21:55 697100 |  TRACE   | 

2023/05/05 01:21:55 698283 |  TRACE   | Training for 300 steps (current step: 10500)
2023/05/05 01:34:16 809700 |  TRACE   | Learning rate scheduler:
2023/05/05 01:34:16 829547 |  TRACE   |  - Validation improved (0.16881 -> 0.16831), resetting plateau patience
2023/05/05 01:34:16 830398 |  TRACE   | Early stopping:
2023/05/05 01:34:16 831289 |  TRACE   |  - New best loss: 0.16831
2023/05/05 01:34:16 832047 |  TRACE   |  - Improvement over previous best: -0.00050
2023/05/05 01:34:16 832730 |  TRACE   |  - Best model updated!
2023/05/05 01:34:25 015899 |  TRACE   | 

2023/05/05 01:34:25 017191 |  TRACE   | Training for 300 steps (current step: 10800)
2023/05/05 01:46:15 696028 |  TRACE   | Learning rate scheduler:
2023/05/05 01:46:15 697499 |  TRACE   |  - Validation did not improve (0.16831 -> 0.16916), increasing plateau patience (1/3)
2023/05/05 01:46:15 698370 |  TRACE   | Early stopping:
2023/05/05 01:46:15 699034 |  TRACE   |  - No improvement: 0.16916 (current best: 0.16831)
2023/05/05 01:46:15 699740 |  TRACE   |  - Patience: 1/10
2023/05/05 01:46:15 700510 |  TRACE   | 

2023/05/05 01:46:15 701180 |  TRACE   | Training for 300 steps (current step: 11100)
2023/05/05 01:58:50 160984 |  TRACE   | Learning rate scheduler:
2023/05/05 01:58:50 162864 |  TRACE   |  - Validation did not improve (0.16831 -> 0.16835), increasing plateau patience (2/3)
2023/05/05 01:58:50 163889 |  TRACE   | Early stopping:
2023/05/05 01:58:50 165112 |  TRACE   |  - No improvement: 0.16835 (current best: 0.16831)
2023/05/05 01:58:50 166274 |  TRACE   |  - Patience: 2/10
2023/05/05 01:58:50 167467 |  TRACE   | 

2023/05/05 01:58:50 168304 |  TRACE   | Training for 300 steps (current step: 11400)
2023/05/05 02:11:35 554040 |  TRACE   | Learning rate scheduler:
2023/05/05 02:11:35 557232 |  TRACE   |  - Validation did not improve (0.16831 -> 0.16872), increasing plateau patience (3/3)
2023/05/05 02:11:35 558270 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/05 02:11:35 559266 |  TRACE   |  - New learning rates: lr_min=1.5625e-06 - lr_max=3.125e-06
2023/05/05 02:11:35 560115 |  TRACE   | Early stopping:
2023/05/05 02:11:35 561003 |  TRACE   |  - No improvement: 0.16872 (current best: 0.16831)
2023/05/05 02:11:35 561875 |  TRACE   |  - Patience: 3/10
2023/05/05 02:11:35 562753 |  TRACE   | 

2023/05/05 02:11:35 563930 |  TRACE   | Training for 300 steps (current step: 11700)
2023/05/05 02:23:24 537050 |  TRACE   | Learning rate scheduler:
2023/05/05 02:23:24 560286 |  TRACE   |  - Validation improved (0.16831 -> 0.16812), resetting plateau patience
2023/05/05 02:23:24 561232 |  TRACE   | Early stopping:
2023/05/05 02:23:24 562188 |  TRACE   |  - New best loss: 0.16812
2023/05/05 02:23:24 563054 |  TRACE   |  - Improvement over previous best: -0.00018
2023/05/05 02:23:24 563939 |  TRACE   |  - Best model updated!
2023/05/05 02:23:32 824330 |  TRACE   | 

2023/05/05 02:23:32 825464 |  TRACE   | Training for 300 steps (current step: 12000)
2023/05/05 02:36:16 952714 |  TRACE   | Learning rate scheduler:
2023/05/05 02:36:16 973099 |  TRACE   |  - Validation improved (0.16812 -> 0.16781), resetting plateau patience
2023/05/05 02:36:16 973963 |  TRACE   | Early stopping:
2023/05/05 02:36:16 974797 |  TRACE   |  - New best loss: 0.16781
2023/05/05 02:36:16 975684 |  TRACE   |  - Improvement over previous best: -0.00031
2023/05/05 02:36:16 976425 |  TRACE   |  - Best model updated!
2023/05/05 02:36:25 047287 |  TRACE   | 

2023/05/05 02:36:25 048481 |  TRACE   | Training for 300 steps (current step: 12300)
2023/05/05 02:49:34 371875 |  TRACE   | Learning rate scheduler:
2023/05/05 02:49:34 373239 |  TRACE   |  - Validation did not improve (0.16781 -> 0.16899), increasing plateau patience (1/3)
2023/05/05 02:49:34 374550 |  TRACE   | Early stopping:
2023/05/05 02:49:34 375470 |  TRACE   |  - No improvement: 0.16899 (current best: 0.16781)
2023/05/05 02:49:34 376185 |  TRACE   |  - Patience: 1/10
2023/05/05 02:49:34 376930 |  TRACE   | 

2023/05/05 02:49:34 377753 |  TRACE   | Training for 300 steps (current step: 12600)
2023/05/05 03:02:30 245367 |  TRACE   | Learning rate scheduler:
2023/05/05 03:02:30 246617 |  TRACE   |  - Validation improved (0.16781 -> 0.16774), resetting plateau patience
2023/05/05 03:02:30 247554 |  TRACE   | Early stopping:
2023/05/05 03:02:30 248339 |  TRACE   |  - New best loss: 0.16774
2023/05/05 03:02:30 249020 |  TRACE   |  - Improvement over previous best: -0.00008
2023/05/05 03:02:30 249740 |  TRACE   |  - Best model updated!
2023/05/05 03:02:38 890636 |  TRACE   | 

2023/05/05 03:02:38 892535 |  TRACE   | Training for 300 steps (current step: 12900)
2023/05/05 03:15:29 162994 |  TRACE   | Learning rate scheduler:
2023/05/05 03:15:29 190315 |  TRACE   |  - Validation improved (0.16774 -> 0.16764), resetting plateau patience
2023/05/05 03:15:29 191288 |  TRACE   | Early stopping:
2023/05/05 03:15:29 192127 |  TRACE   |  - New best loss: 0.16764
2023/05/05 03:15:29 192892 |  TRACE   |  - Improvement over previous best: -0.00010
2023/05/05 03:15:29 193824 |  TRACE   |  - Best model updated!
2023/05/05 03:15:37 370299 |  TRACE   | 

2023/05/05 03:15:37 371328 |  TRACE   | Training for 300 steps (current step: 13200)
2023/05/05 03:28:12 850730 |  TRACE   | Learning rate scheduler:
2023/05/05 03:28:12 852341 |  TRACE   |  - Validation did not improve (0.16764 -> 0.16776), increasing plateau patience (1/3)
2023/05/05 03:28:12 853254 |  TRACE   | Early stopping:
2023/05/05 03:28:12 854070 |  TRACE   |  - No improvement: 0.16776 (current best: 0.16764)
2023/05/05 03:28:12 854874 |  TRACE   |  - Patience: 1/10
2023/05/05 03:28:12 855615 |  TRACE   | 

2023/05/05 03:28:12 856445 |  TRACE   | Training for 300 steps (current step: 13500)
2023/05/05 03:40:19 478162 |  TRACE   | Learning rate scheduler:
2023/05/05 03:40:19 506164 |  TRACE   |  - Validation did not improve (0.16764 -> 0.16773), increasing plateau patience (2/3)
2023/05/05 03:40:19 507118 |  TRACE   | Early stopping:
2023/05/05 03:40:19 508061 |  TRACE   |  - No improvement: 0.16773 (current best: 0.16764)
2023/05/05 03:40:19 508957 |  TRACE   |  - Patience: 2/10
2023/05/05 03:40:19 509748 |  TRACE   | 

2023/05/05 03:40:19 510612 |  TRACE   | Training for 300 steps (current step: 13800)
2023/05/05 03:52:48 255013 |  TRACE   | Learning rate scheduler:
2023/05/05 03:52:48 256811 |  TRACE   |  - Validation improved (0.16764 -> 0.16697), resetting plateau patience
2023/05/05 03:52:48 257606 |  TRACE   | Early stopping:
2023/05/05 03:52:48 258825 |  TRACE   |  - New best loss: 0.16697
2023/05/05 03:52:48 260296 |  TRACE   |  - Improvement over previous best: -0.00068
2023/05/05 03:52:48 261028 |  TRACE   |  - Best model updated!
2023/05/05 03:52:56 718782 |  TRACE   | 

2023/05/05 03:52:56 720004 |  TRACE   | Training for 300 steps (current step: 14100)
2023/05/05 04:05:25 810046 |  TRACE   | Learning rate scheduler:
2023/05/05 04:05:25 811923 |  TRACE   |  - Validation improved (0.16697 -> 0.16673), resetting plateau patience
2023/05/05 04:05:25 813005 |  TRACE   | Early stopping:
2023/05/05 04:05:25 813795 |  TRACE   |  - New best loss: 0.16673
2023/05/05 04:05:25 814635 |  TRACE   |  - Improvement over previous best: -0.00023
2023/05/05 04:05:25 815555 |  TRACE   |  - Best model updated!
2023/05/05 04:05:34 533186 |  TRACE   | 

2023/05/05 04:05:34 534492 |  TRACE   | Training for 300 steps (current step: 14400)
2023/05/05 04:18:13 401867 |  TRACE   | Learning rate scheduler:
2023/05/05 04:18:13 435495 |  TRACE   |  - Validation did not improve (0.16673 -> 0.16757), increasing plateau patience (1/3)
2023/05/05 04:18:13 436432 |  TRACE   | Early stopping:
2023/05/05 04:18:13 437208 |  TRACE   |  - No improvement: 0.16757 (current best: 0.16673)
2023/05/05 04:18:13 438799 |  TRACE   |  - Patience: 1/10
2023/05/05 04:18:13 439692 |  TRACE   | 

2023/05/05 04:18:13 440542 |  TRACE   | Training for 300 steps (current step: 14700)
2023/05/05 04:30:33 732406 |  TRACE   | Learning rate scheduler:
2023/05/05 04:30:33 733940 |  TRACE   |  - Validation did not improve (0.16673 -> 0.16804), increasing plateau patience (2/3)
2023/05/05 04:30:33 734777 |  TRACE   | Early stopping:
2023/05/05 04:30:33 735571 |  TRACE   |  - No improvement: 0.16804 (current best: 0.16673)
2023/05/05 04:30:33 736352 |  TRACE   |  - Patience: 2/10
2023/05/05 04:30:33 737096 |  TRACE   | 

2023/05/05 04:30:33 737825 |  TRACE   | Training for 300 steps (current step: 15000)
2023/05/05 04:42:36 348777 |  TRACE   | Learning rate scheduler:
2023/05/05 04:42:36 350480 |  TRACE   |  - Validation did not improve (0.16673 -> 0.16769), increasing plateau patience (3/3)
2023/05/05 04:42:36 351367 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/05 04:42:36 352154 |  TRACE   |  - New learning rates: lr_min=1e-06 - lr_max=1.5625e-06
2023/05/05 04:42:36 352915 |  TRACE   | Early stopping:
2023/05/05 04:42:36 353702 |  TRACE   |  - No improvement: 0.16769 (current best: 0.16673)
2023/05/05 04:42:36 354626 |  TRACE   |  - Patience: 3/10
2023/05/05 04:42:36 355367 |  TRACE   | 

2023/05/05 04:42:36 356178 |  TRACE   | Training for 300 steps (current step: 15300)
2023/05/05 04:55:12 840801 |  TRACE   | Learning rate scheduler:
2023/05/05 04:55:12 842135 |  TRACE   |  - Validation did not improve (0.16673 -> 0.16729), increasing plateau patience (1/3)
2023/05/05 04:55:12 844161 |  TRACE   | Early stopping:
2023/05/05 04:55:12 849325 |  TRACE   |  - No improvement: 0.16729 (current best: 0.16673)
2023/05/05 04:55:12 854595 |  TRACE   |  - Patience: 4/10
2023/05/05 04:55:12 859900 |  TRACE   | 

2023/05/05 04:55:12 865102 |  TRACE   | Training for 300 steps (current step: 15600)
2023/05/05 05:08:14 183366 |  TRACE   | Learning rate scheduler:
2023/05/05 05:08:14 184743 |  TRACE   |  - Validation did not improve (0.16673 -> 0.16703), increasing plateau patience (2/3)
2023/05/05 05:08:14 185660 |  TRACE   | Early stopping:
2023/05/05 05:08:14 186456 |  TRACE   |  - No improvement: 0.16703 (current best: 0.16673)
2023/05/05 05:08:14 187286 |  TRACE   |  - Patience: 5/10
2023/05/05 05:08:14 188127 |  TRACE   | 

2023/05/05 05:08:14 188886 |  TRACE   | Training for 300 steps (current step: 15900)
2023/05/05 05:21:25 381915 |  TRACE   | Learning rate scheduler:
2023/05/05 05:21:25 383279 |  TRACE   |  - Validation did not improve (0.16673 -> 0.16692), increasing plateau patience (3/3)
2023/05/05 05:21:25 384140 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/05 05:21:25 385003 |  TRACE   |  - New learning rates: lr_min=1e-06 - lr_max=1e-06
2023/05/05 05:21:25 385914 |  TRACE   | Early stopping:
2023/05/05 05:21:25 386684 |  TRACE   |  - No improvement: 0.16692 (current best: 0.16673)
2023/05/05 05:21:25 387487 |  TRACE   |  - Patience: 6/10
2023/05/05 05:21:25 388287 |  TRACE   | 

2023/05/05 05:21:25 389138 |  TRACE   | Training for 300 steps (current step: 16200)
2023/05/05 05:34:00 915400 |  TRACE   | Learning rate scheduler:
2023/05/05 05:34:00 917672 |  TRACE   |  - Validation did not improve (0.16673 -> 0.16706), increasing plateau patience (1/3)
2023/05/05 05:34:00 918494 |  TRACE   | Early stopping:
2023/05/05 05:34:00 919190 |  TRACE   |  - No improvement: 0.16706 (current best: 0.16673)
2023/05/05 05:34:00 919966 |  TRACE   |  - Patience: 7/10
2023/05/05 05:34:00 920837 |  TRACE   | 

2023/05/05 05:34:00 921529 |  TRACE   | Training for 300 steps (current step: 16500)
2023/05/05 05:46:40 693241 |  TRACE   | Learning rate scheduler:
2023/05/05 05:46:40 694914 |  TRACE   |  - Validation did not improve (0.16673 -> 0.16684), increasing plateau patience (2/3)
2023/05/05 05:46:40 695804 |  TRACE   | Early stopping:
2023/05/05 05:46:40 696905 |  TRACE   |  - No improvement: 0.16684 (current best: 0.16673)
2023/05/05 05:46:40 697842 |  TRACE   |  - Patience: 8/10
2023/05/05 05:46:40 699123 |  TRACE   | 

2023/05/05 05:46:40 700208 |  TRACE   | Training for 300 steps (current step: 16800)
2023/05/05 05:58:59 705970 |  TRACE   | Learning rate scheduler:
2023/05/05 05:58:59 707484 |  TRACE   |  - Validation improved (0.16673 -> 0.16664), resetting plateau patience
2023/05/05 05:58:59 708298 |  TRACE   | Early stopping:
2023/05/05 05:58:59 709006 |  TRACE   |  - New best loss: 0.16664
2023/05/05 05:58:59 709787 |  TRACE   |  - Improvement over previous best: -0.00009
2023/05/05 05:58:59 710555 |  TRACE   |  - Best model updated!
2023/05/05 05:59:07 961142 |  TRACE   | 

2023/05/05 05:59:07 962360 |  TRACE   | Training for 300 steps (current step: 17100)
2023/05/05 06:12:02 228300 |  TRACE   | Learning rate scheduler:
2023/05/05 06:12:02 229913 |  TRACE   |  - Validation did not improve (0.16664 -> 0.16675), increasing plateau patience (1/3)
2023/05/05 06:12:02 230809 |  TRACE   | Early stopping:
2023/05/05 06:12:02 231639 |  TRACE   |  - No improvement: 0.16675 (current best: 0.16664)
2023/05/05 06:12:02 232407 |  TRACE   |  - Patience: 1/10
2023/05/05 06:12:02 233202 |  TRACE   | 

2023/05/05 06:12:02 233962 |  TRACE   | Training for 300 steps (current step: 17400)
2023/05/05 06:24:23 418914 |  TRACE   | Learning rate scheduler:
2023/05/05 06:24:23 440154 |  TRACE   |  - Validation did not improve (0.16664 -> 0.16682), increasing plateau patience (2/3)
2023/05/05 06:24:23 441120 |  TRACE   | Early stopping:
2023/05/05 06:24:23 442083 |  TRACE   |  - No improvement: 0.16682 (current best: 0.16664)
2023/05/05 06:24:23 442836 |  TRACE   |  - Patience: 2/10
2023/05/05 06:24:23 443834 |  TRACE   | 

2023/05/05 06:24:23 444670 |  TRACE   | Training for 300 steps (current step: 17700)
2023/05/05 06:36:27 011693 |  TRACE   | Learning rate scheduler:
2023/05/05 06:36:27 013049 |  TRACE   |  - Validation did not improve (0.16664 -> 0.16668), increasing plateau patience (3/3)
2023/05/05 06:36:27 014030 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/05 06:36:27 014954 |  TRACE   |  - New learning rates: lr_min=1e-06 - lr_max=1e-06
2023/05/05 06:36:27 015921 |  TRACE   | Early stopping:
2023/05/05 06:36:27 016738 |  TRACE   |  - No improvement: 0.16668 (current best: 0.16664)
2023/05/05 06:36:27 017889 |  TRACE   |  - Patience: 3/10
2023/05/05 06:36:27 018772 |  TRACE   | 

2023/05/05 06:36:27 019701 |  TRACE   | Training for 300 steps (current step: 18000)
2023/05/05 06:48:35 223446 |  TRACE   | Learning rate scheduler:
2023/05/05 06:48:35 241010 |  TRACE   |  - Validation improved (0.16664 -> 0.16634), resetting plateau patience
2023/05/05 06:48:35 241768 |  TRACE   | Early stopping:
2023/05/05 06:48:35 242635 |  TRACE   |  - New best loss: 0.16634
2023/05/05 06:48:35 243430 |  TRACE   |  - Improvement over previous best: -0.00030
2023/05/05 06:48:35 244336 |  TRACE   |  - Best model updated!
2023/05/05 06:48:43 365749 |  TRACE   | 

2023/05/05 06:48:43 367051 |  TRACE   | Training for 300 steps (current step: 18300)
2023/05/05 07:01:12 749036 |  TRACE   | Learning rate scheduler:
2023/05/05 07:01:12 750261 |  TRACE   |  - Validation did not improve (0.16634 -> 0.16661), increasing plateau patience (1/3)
2023/05/05 07:01:12 751093 |  TRACE   | Early stopping:
2023/05/05 07:01:12 751855 |  TRACE   |  - No improvement: 0.16661 (current best: 0.16634)
2023/05/05 07:01:12 752785 |  TRACE   |  - Patience: 1/10
2023/05/05 07:01:12 753508 |  TRACE   | 

2023/05/05 07:01:12 754391 |  TRACE   | Training for 300 steps (current step: 18600)
2023/05/05 07:13:49 504913 |  TRACE   | Learning rate scheduler:
2023/05/05 07:13:49 506297 |  TRACE   |  - Validation did not improve (0.16634 -> 0.16690), increasing plateau patience (2/3)
2023/05/05 07:13:49 507163 |  TRACE   | Early stopping:
2023/05/05 07:13:49 507882 |  TRACE   |  - No improvement: 0.16690 (current best: 0.16634)
2023/05/05 07:13:49 508733 |  TRACE   |  - Patience: 2/10
2023/05/05 07:13:49 509550 |  TRACE   | 

2023/05/05 07:13:49 510442 |  TRACE   | Training for 300 steps (current step: 18900)
2023/05/05 07:26:58 310362 |  TRACE   | Learning rate scheduler:
2023/05/05 07:26:58 312008 |  TRACE   |  - Validation did not improve (0.16634 -> 0.16643), increasing plateau patience (3/3)
2023/05/05 07:26:58 312832 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/05 07:26:58 313634 |  TRACE   |  - New learning rates: lr_min=1e-06 - lr_max=1e-06
2023/05/05 07:26:58 314337 |  TRACE   | Early stopping:
2023/05/05 07:26:58 315042 |  TRACE   |  - No improvement: 0.16643 (current best: 0.16634)
2023/05/05 07:26:58 315855 |  TRACE   |  - Patience: 3/10
2023/05/05 07:26:58 317081 |  TRACE   | 

2023/05/05 07:26:58 317819 |  TRACE   | Training for 300 steps (current step: 19200)
2023/05/05 07:39:02 090201 |  TRACE   | Learning rate scheduler:
2023/05/05 07:39:02 179356 |  TRACE   |  - Validation did not improve (0.16634 -> 0.16703), increasing plateau patience (1/3)
2023/05/05 07:39:02 180202 |  TRACE   | Early stopping:
2023/05/05 07:39:02 180976 |  TRACE   |  - No improvement: 0.16703 (current best: 0.16634)
2023/05/05 07:39:02 181723 |  TRACE   |  - Patience: 4/10
2023/05/05 07:39:02 182473 |  TRACE   | 

2023/05/05 07:39:02 183191 |  TRACE   | Training for 300 steps (current step: 19500)
2023/05/05 07:51:32 333657 |  TRACE   | Learning rate scheduler:
2023/05/05 07:51:32 334956 |  TRACE   |  - Validation did not improve (0.16634 -> 0.16642), increasing plateau patience (2/3)
2023/05/05 07:51:32 335848 |  TRACE   | Early stopping:
2023/05/05 07:51:32 336692 |  TRACE   |  - No improvement: 0.16642 (current best: 0.16634)
2023/05/05 07:51:32 337790 |  TRACE   |  - Patience: 5/10
2023/05/05 07:51:32 338841 |  TRACE   | 

2023/05/05 07:51:32 339850 |  TRACE   | Training for 300 steps (current step: 19800)
2023/05/05 08:03:49 356374 |  TRACE   | Learning rate scheduler:
2023/05/05 08:03:49 375309 |  TRACE   |  - Validation did not improve (0.16634 -> 0.16644), increasing plateau patience (3/3)
2023/05/05 08:03:49 376176 |  TRACE   |  - Plateau patience reached, reducing learning rates by 0.5
2023/05/05 08:03:49 376949 |  TRACE   |  - New learning rates: lr_min=1e-06 - lr_max=1e-06
2023/05/05 08:03:49 377773 |  TRACE   | Early stopping:
2023/05/05 08:03:49 378441 |  TRACE   |  - No improvement: 0.16644 (current best: 0.16634)
2023/05/05 08:03:49 379257 |  TRACE   |  - Patience: 6/10
2023/05/05 08:03:49 380171 |  TRACE   | 

2023/05/05 08:03:49 380860 |  TRACE   | Training for 300 steps (current step: 20100)
2023/05/05 08:16:43 537883 |  TRACE   | Learning rate scheduler:
2023/05/05 08:16:43 539443 |  TRACE   |  - Validation did not improve (0.16634 -> 0.16710), increasing plateau patience (1/3)
2023/05/05 08:16:43 540330 |  TRACE   | Early stopping:
2023/05/05 08:16:43 541033 |  TRACE   |  - No improvement: 0.16710 (current best: 0.16634)
2023/05/05 08:16:43 541789 |  TRACE   |  - Patience: 7/10
2023/05/05 08:16:43 542585 |  TRACE   | 

2023/05/05 08:16:43 543387 |  TRACE   | Training for 300 steps (current step: 20400)
2023/05/05 08:28:34 540269 |  TRACE   | Learning rate scheduler:
2023/05/05 08:28:34 541563 |  TRACE   |  - Validation improved (0.16634 -> 0.16629), resetting plateau patience
2023/05/05 08:28:34 542461 |  TRACE   | Early stopping:
2023/05/05 08:28:34 543215 |  TRACE   |  - New best loss: 0.16629
2023/05/05 08:28:34 543930 |  TRACE   |  - Improvement over previous best: -0.00005
2023/05/05 08:28:34 544660 |  TRACE   |  - Best model updated!
2023/05/05 08:28:42 773780 |  TRACE   | 

2023/05/05 08:28:42 774983 |  TRACE   | Training for 300 steps (current step: 20700)
2023/05/05 08:41:10 944409 |  TRACE   | Learning rate scheduler:
2023/05/05 08:41:10 945838 |  TRACE   |  - Validation did not improve (0.16629 -> 0.16665), increasing plateau patience (1/3)
2023/05/05 08:41:10 946661 |  TRACE   | Early stopping:
2023/05/05 08:41:10 947552 |  TRACE   |  - No improvement: 0.16665 (current best: 0.16629)
2023/05/05 08:41:10 948427 |  TRACE   |  - Patience: 1/10
2023/05/05 08:41:10 949100 |  TRACE   | 

2023/05/05 08:41:10 949810 |  TRACE   | Training for 300 steps (current step: 21000)
2023/05/05 08:52:32 339059 |  TRACE   | Learning rate scheduler:
2023/05/05 08:52:32 358572 |  TRACE   |  - Validation improved (0.16629 -> 0.16622), resetting plateau patience
2023/05/05 08:52:32 359455 |  TRACE   | Early stopping:
2023/05/05 08:52:32 360339 |  TRACE   |  - New best loss: 0.16622
2023/05/05 08:52:32 361041 |  TRACE   |  - Improvement over previous best: -0.00006
2023/05/05 08:52:32 361934 |  TRACE   |  - Best model updated!
2023/05/05 08:52:40 571197 |  TRACE   | 

2023/05/05 08:52:40 572332 |  TRACE   | Training for 300 steps (current step: 21300)
2023/05/05 09:05:06 521039 |  TRACE   | Learning rate scheduler:
2023/05/05 09:05:06 624366 |  TRACE   |  - Validation did not improve (0.16622 -> 0.16641), increasing plateau patience (1/3)
2023/05/05 09:05:06 625225 |  TRACE   | Early stopping:
2023/05/05 09:05:06 625958 |  TRACE   |  - No improvement: 0.16641 (current best: 0.16622)
2023/05/05 09:05:06 626697 |  TRACE   |  - Patience: 1/10
2023/05/05 09:05:06 627383 |  TRACE   | 

2023/05/05 09:05:06 628107 |  TRACE   | Training for 300 steps (current step: 21600)
