2023/02/11 14:57:42 205127 |  TRACE   |  Model configs:
2023/02/11 14:57:42 205395 |  TRACE   |  - Model: 20230211T145741-Unet+Eff-b0+ws10+ts24
2023/02/11 14:57:42 205544 |  TRACE   |  - Parameters: 6_257_517
2023/02/11 14:57:42 205677 |  TRACE   |  - Device: cuda
2023/02/11 14:57:42 205810 |  TRACE   |  - Input bands: ['s2_ndvi']
2023/02/11 14:57:42 205935 |  TRACE   |  - N ts: 24
 
2023/02/11 14:57:42 206057 |  TRACE   |  Steps configs:
2023/02/11 14:57:42 206179 |  TRACE   |  - Maximum steps: None
2023/02/11 14:57:42 206299 |  TRACE   |  - Validation steps: 200
2023/02/11 14:57:42 206418 |  TRACE   |  - Batch-size: 16
 
2023/02/11 14:57:42 206538 |  TRACE   |  Optimizer configs:
2023/02/11 14:57:42 206659 |  TRACE   |  - Optimizer: Adam
2023/02/11 14:57:42 206779 |  TRACE   |  - Early-stopping tolerance: 10
2023/02/11 14:57:42 206903 |  TRACE   |  - Early-stopping improvement: 0.0001
2023/02/11 14:57:42 207024 |  TRACE   |  - Early-stopping metric: loss
2023/02/11 14:57:42 207143 |  TRACE   |  - Scheduler: ReduceLROnPlateau
2023/02/11 14:57:42 207262 |  TRACE   |  - Learninr rate tolerance: 3
 
2023/02/11 14:57:42 207380 |  TRACE   |  Loss configs:
2023/02/11 14:57:42 207503 |  TRACE   |  - Tag: DiceLoss
2023/02/11 14:57:42 207623 |  TRACE   |  - Loss: dice_loss
2023/02/11 14:57:42 207741 |  TRACE   |  - Weight / multiplier: None
2023/02/11 14:57:42 207859 |  TRACE   |  - LC coefficient: 0
2023/02/11 14:57:42 207976 |  TRACE   |  ---
2023/02/11 14:57:42 208095 |  TRACE   |  - Tag: DiceLoss_ws
2023/02/11 14:57:42 208214 |  TRACE   |  - Loss: dice_loss
2023/02/11 14:57:42 208335 |  TRACE   |  - Weight / multiplier: {'watersheds': 10}
2023/02/11 14:57:42 208483 |  TRACE   |  - LC coefficient: 1
2023/02/11 14:57:42 208602 |  TRACE   |  ---
2023/02/11 14:57:42 208721 |  TRACE   |  - Tag: DiceLoss_d
2023/02/11 14:57:42 208839 |  TRACE   |  - Loss: dice_loss
2023/02/11 14:57:42 208959 |  TRACE   |  - Weight / multiplier: {'distance': 1}
2023/02/11 14:57:42 209078 |  TRACE   |  - LC coefficient: 0
2023/02/11 14:57:42 209196 |  TRACE   | 

2023/02/11 14:58:09 983566 |  TRACE   | Writing results to TensorBoard:
2023/02/11 14:58:09 984147 |  TRACE   |  - Start loss computation
2023/02/11 14:58:09 984611 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:00:57 529345 |  TRACE   | 

2023/02/11 15:00:57 529804 |  TRACE   | Training for 200 steps (current step: 0)
2023/02/11 15:05:41 085324 |  TRACE   | Validating
2023/02/11 15:06:09 118353 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:06:09 118891 |  TRACE   |  - Start loss computation
2023/02/11 15:06:09 119345 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:06:52 553824 |  TRACE   |  - New best loss: 0.48114
2023/02/11 15:06:52 554457 |  TRACE   |  - Improvement over previous best: -0.22969
2023/02/11 15:06:52 695568 |  TRACE   |  - Best model updated!
 
2023/02/11 15:06:52 695936 |  TRACE   | Training for 200 steps (current step: 200)
2023/02/11 15:11:11 694244 |  TRACE   | Validating
2023/02/11 15:11:39 410190 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:11:39 410722 |  TRACE   |  - Start loss computation
2023/02/11 15:11:39 411179 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:12:41 443383 |  TRACE   |  - New best loss: 0.36144
2023/02/11 15:12:41 443918 |  TRACE   |  - Improvement over previous best: -0.11970
2023/02/11 15:12:41 571741 |  TRACE   |  - Best model updated!
 
2023/02/11 15:12:41 572141 |  TRACE   | Training for 200 steps (current step: 400)
2023/02/11 15:17:04 100614 |  TRACE   | Validating
2023/02/11 15:17:31 732166 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:17:31 732737 |  TRACE   |  - Start loss computation
2023/02/11 15:17:31 733198 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:18:41 989207 |  TRACE   |  - No improvement for loss: 0.37284 (current best: 0.36144)
2023/02/11 15:18:41 989682 |  TRACE   | Training for 200 steps (current step: 600)
2023/02/11 15:23:01 084174 |  TRACE   | Validating
2023/02/11 15:23:28 930540 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:23:28 931098 |  TRACE   |  - Start loss computation
2023/02/11 15:23:28 931545 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:24:31 623386 |  TRACE   |  - New best loss: 0.33942
2023/02/11 15:24:31 623933 |  TRACE   |  - Improvement over previous best: -0.02202
2023/02/11 15:24:31 757560 |  TRACE   |  - Best model updated!
 
2023/02/11 15:24:31 757912 |  TRACE   | Training for 200 steps (current step: 800)
2023/02/11 15:28:50 369196 |  TRACE   | Validating
2023/02/11 15:29:18 154524 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:29:18 155099 |  TRACE   |  - Start loss computation
2023/02/11 15:29:18 155657 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:30:31 251359 |  TRACE   |  - New best loss: 0.33270
2023/02/11 15:30:31 251908 |  TRACE   |  - Improvement over previous best: -0.00672
2023/02/11 15:30:31 380659 |  TRACE   |  - Best model updated!
 
2023/02/11 15:30:31 381074 |  TRACE   | Training for 200 steps (current step: 1000)
2023/02/11 15:34:58 146125 |  TRACE   | Validating
2023/02/11 15:35:25 815867 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:35:25 816467 |  TRACE   |  - Start loss computation
2023/02/11 15:35:25 816935 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:36:37 432420 |  TRACE   |  - New best loss: 0.33226
2023/02/11 15:36:37 432963 |  TRACE   |  - Improvement over previous best: -0.00044
2023/02/11 15:36:37 559945 |  TRACE   |  - Best model updated!
 
2023/02/11 15:36:37 560309 |  TRACE   | Training for 200 steps (current step: 1200)
2023/02/11 15:40:56 933633 |  TRACE   | Validating
2023/02/11 15:41:24 660018 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:41:24 660577 |  TRACE   |  - Start loss computation
2023/02/11 15:41:24 661009 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:42:39 126289 |  TRACE   |  - No improvement for loss: 0.34245 (current best: 0.33226)
2023/02/11 15:42:39 126782 |  TRACE   | Training for 200 steps (current step: 1400)
2023/02/11 15:46:56 667055 |  TRACE   | Validating
2023/02/11 15:47:24 462859 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:47:24 463442 |  TRACE   |  - Start loss computation
2023/02/11 15:47:24 463872 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:48:31 502081 |  TRACE   |  - No improvement for loss: 0.34428 (current best: 0.33226)
2023/02/11 15:48:31 502557 |  TRACE   | Training for 200 steps (current step: 1600)
2023/02/11 15:52:51 587545 |  TRACE   | Validating
2023/02/11 15:53:19 197774 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:53:19 198285 |  TRACE   |  - Start loss computation
2023/02/11 15:53:19 198788 |  TRACE   |  - Start image generation for 100 images
2023/02/11 15:54:37 204245 |  TRACE   |  - New best loss: 0.32812
2023/02/11 15:54:37 204820 |  TRACE   |  - Improvement over previous best: -0.00414
2023/02/11 15:54:37 353552 |  TRACE   |  - Best model updated!
 
2023/02/11 15:54:37 353938 |  TRACE   | Training for 200 steps (current step: 1800)
2023/02/11 15:58:57 849286 |  TRACE   | Validating
2023/02/11 15:59:25 580230 |  TRACE   | Writing results to TensorBoard:
2023/02/11 15:59:25 580842 |  TRACE   |  - Start loss computation
2023/02/11 15:59:25 581293 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:00:48 681641 |  TRACE   |  - No improvement for loss: 0.35770 (current best: 0.32812)
2023/02/11 16:00:48 682102 |  TRACE   | Training for 200 steps (current step: 2000)
2023/02/11 16:05:12 813951 |  TRACE   | Validating
2023/02/11 16:05:40 506605 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:05:40 507161 |  TRACE   |  - Start loss computation
2023/02/11 16:05:40 507596 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:06:58 781929 |  TRACE   |  - New best loss: 0.30201
2023/02/11 16:06:58 782468 |  TRACE   |  - Improvement over previous best: -0.02611
2023/02/11 16:06:58 921027 |  TRACE   |  - Best model updated!
 
2023/02/11 16:06:58 921420 |  TRACE   | Training for 200 steps (current step: 2200)
2023/02/11 16:11:22 348593 |  TRACE   | Validating
2023/02/11 16:11:50 119993 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:11:50 120581 |  TRACE   |  - Start loss computation
2023/02/11 16:11:50 121018 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:13:06 702262 |  TRACE   |  - New best loss: 0.30105
2023/02/11 16:13:06 702811 |  TRACE   |  - Improvement over previous best: -0.00096
2023/02/11 16:13:06 830327 |  TRACE   |  - Best model updated!
 
2023/02/11 16:13:06 830761 |  TRACE   | Training for 200 steps (current step: 2400)
2023/02/11 16:17:26 208181 |  TRACE   | Validating
2023/02/11 16:17:53 943428 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:17:53 944006 |  TRACE   |  - Start loss computation
2023/02/11 16:17:53 944478 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:19:07 833595 |  TRACE   |  - No improvement for loss: 0.30318 (current best: 0.30105)
2023/02/11 16:19:07 833988 |  TRACE   | Training for 200 steps (current step: 2600)
2023/02/11 16:23:33 254431 |  TRACE   | Validating
2023/02/11 16:24:00 910756 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:24:00 911267 |  TRACE   |  - Start loss computation
2023/02/11 16:24:00 911700 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:25:19 692681 |  TRACE   |  - New best loss: 0.29770
2023/02/11 16:25:19 693215 |  TRACE   |  - Improvement over previous best: -0.00335
2023/02/11 16:25:19 819803 |  TRACE   |  - Best model updated!
 
2023/02/11 16:25:19 820177 |  TRACE   | Training for 200 steps (current step: 2800)
2023/02/11 16:29:41 230780 |  TRACE   | Validating
2023/02/11 16:30:08 897300 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:30:08 897863 |  TRACE   |  - Start loss computation
2023/02/11 16:30:08 898309 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:31:23 772623 |  TRACE   |  - New best loss: 0.28515
2023/02/11 16:31:23 773160 |  TRACE   |  - Improvement over previous best: -0.01254
2023/02/11 16:31:23 902966 |  TRACE   |  - Best model updated!
 
2023/02/11 16:31:23 903365 |  TRACE   | Training for 200 steps (current step: 3000)
2023/02/11 16:35:46 882622 |  TRACE   | Validating
2023/02/11 16:36:14 520048 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:36:14 520584 |  TRACE   |  - Start loss computation
2023/02/11 16:36:14 521061 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:37:26 742292 |  TRACE   |  - No improvement for loss: 0.28954 (current best: 0.28515)
2023/02/11 16:37:26 742665 |  TRACE   | Training for 200 steps (current step: 3200)
2023/02/11 16:41:50 091931 |  TRACE   | Validating
2023/02/11 16:42:17 941981 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:42:17 942530 |  TRACE   |  - Start loss computation
2023/02/11 16:42:17 942959 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:43:27 547334 |  TRACE   |  - No improvement for loss: 0.29769 (current best: 0.28515)
2023/02/11 16:43:27 547780 |  TRACE   | Training for 200 steps (current step: 3400)
2023/02/11 16:47:50 834993 |  TRACE   | Validating
2023/02/11 16:48:18 614146 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:48:18 614756 |  TRACE   |  - Start loss computation
2023/02/11 16:48:18 615223 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:49:30 361703 |  TRACE   |  - No improvement for loss: 0.28955 (current best: 0.28515)
2023/02/11 16:49:30 362181 |  TRACE   | Training for 200 steps (current step: 3600)
2023/02/11 16:53:49 942207 |  TRACE   | Validating
2023/02/11 16:54:17 628669 |  TRACE   | Writing results to TensorBoard:
2023/02/11 16:54:17 629199 |  TRACE   |  - Start loss computation
2023/02/11 16:54:17 629633 |  TRACE   |  - Start image generation for 100 images
2023/02/11 16:55:39 201762 |  TRACE   |  - New best loss: 0.27731
2023/02/11 16:55:39 202305 |  TRACE   |  - Improvement over previous best: -0.00785
2023/02/11 16:55:39 331670 |  TRACE   |  - Best model updated!
 
2023/02/11 16:55:39 332082 |  TRACE   | Training for 200 steps (current step: 3800)
2023/02/11 17:00:05 437620 |  TRACE   | Validating
2023/02/11 17:00:33 068825 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:00:33 069366 |  TRACE   |  - Start loss computation
2023/02/11 17:00:33 069799 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:01:52 366235 |  TRACE   |  - No improvement for loss: 0.28647 (current best: 0.27731)
2023/02/11 17:01:52 366695 |  TRACE   | Training for 200 steps (current step: 4000)
2023/02/11 17:06:14 797338 |  TRACE   | Validating
2023/02/11 17:06:42 394464 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:06:42 394977 |  TRACE   |  - Start loss computation
2023/02/11 17:06:42 395475 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:08:06 602376 |  TRACE   |  - No improvement for loss: 0.29355 (current best: 0.27731)
2023/02/11 17:08:06 602858 |  TRACE   | Training for 200 steps (current step: 4200)
2023/02/11 17:12:28 110170 |  TRACE   | Validating
2023/02/11 17:12:55 746396 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:12:55 746947 |  TRACE   |  - Start loss computation
2023/02/11 17:12:55 747392 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:14:14 311229 |  TRACE   |  - New best loss: 0.27058
2023/02/11 17:14:14 311776 |  TRACE   |  - Improvement over previous best: -0.00673
2023/02/11 17:14:14 442373 |  TRACE   |  - Best model updated!
 
2023/02/11 17:14:14 442753 |  TRACE   | Training for 200 steps (current step: 4400)
2023/02/11 17:18:37 815766 |  TRACE   | Validating
2023/02/11 17:19:05 513192 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:19:05 513755 |  TRACE   |  - Start loss computation
2023/02/11 17:19:05 514189 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:20:19 948345 |  TRACE   |  - No improvement for loss: 0.28238 (current best: 0.27058)
2023/02/11 17:20:19 948842 |  TRACE   | Training for 200 steps (current step: 4600)
2023/02/11 17:24:44 543641 |  TRACE   | Validating
2023/02/11 17:25:12 189767 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:25:12 190321 |  TRACE   |  - Start loss computation
2023/02/11 17:25:12 190751 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:26:29 122394 |  TRACE   |  - No improvement for loss: 0.27480 (current best: 0.27058)
2023/02/11 17:26:29 122834 |  TRACE   | Training for 200 steps (current step: 4800)
2023/02/11 17:30:45 453112 |  TRACE   | Validating
2023/02/11 17:31:13 132352 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:31:13 132944 |  TRACE   |  - Start loss computation
2023/02/11 17:31:13 133377 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:32:35 114588 |  TRACE   |  - New best loss: 0.26828
2023/02/11 17:32:35 115116 |  TRACE   |  - Improvement over previous best: -0.00230
2023/02/11 17:32:35 246664 |  TRACE   |  - Best model updated!
 
2023/02/11 17:32:35 247044 |  TRACE   | Training for 200 steps (current step: 5000)
2023/02/11 17:36:53 495145 |  TRACE   | Validating
2023/02/11 17:37:21 202834 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:37:21 203441 |  TRACE   |  - Start loss computation
2023/02/11 17:37:21 203901 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:38:30 389194 |  TRACE   |  - New best loss: 0.26698
2023/02/11 17:38:30 389743 |  TRACE   |  - Improvement over previous best: -0.00130
2023/02/11 17:38:30 809171 |  TRACE   |  - Best model updated!
 
2023/02/11 17:38:30 809624 |  TRACE   | Training for 200 steps (current step: 5200)
2023/02/11 17:42:51 234634 |  TRACE   | Validating
2023/02/11 17:43:18 904878 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:43:18 905459 |  TRACE   |  - Start loss computation
2023/02/11 17:43:18 905896 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:44:46 165244 |  TRACE   |  - No improvement for loss: 0.28767 (current best: 0.26698)
2023/02/11 17:44:46 165697 |  TRACE   | Training for 200 steps (current step: 5400)
2023/02/11 17:49:13 701830 |  TRACE   | Validating
2023/02/11 17:49:41 357353 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:49:41 357887 |  TRACE   |  - Start loss computation
2023/02/11 17:49:41 358316 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:50:51 065286 |  TRACE   |  - No improvement for loss: 0.27134 (current best: 0.26698)
2023/02/11 17:50:51 065737 |  TRACE   | Training for 200 steps (current step: 5600)
2023/02/11 17:55:12 585105 |  TRACE   | Validating
2023/02/11 17:55:40 126007 |  TRACE   | Writing results to TensorBoard:
2023/02/11 17:55:40 126598 |  TRACE   |  - Start loss computation
2023/02/11 17:55:40 127032 |  TRACE   |  - Start image generation for 100 images
2023/02/11 17:57:04 587599 |  TRACE   |  - No improvement for loss: 0.26903 (current best: 0.26698)
2023/02/11 17:57:04 588107 |  TRACE   | Training for 200 steps (current step: 5800)
2023/02/11 18:01:29 675276 |  TRACE   | Validating
2023/02/11 18:01:57 347377 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:01:57 348022 |  TRACE   |  - Start loss computation
2023/02/11 18:01:57 348527 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:03:31 959821 |  TRACE   |  - No improvement for loss: 0.28308 (current best: 0.26698)
2023/02/11 18:03:31 960278 |  TRACE   | Training for 200 steps (current step: 6000)
2023/02/11 18:07:55 296284 |  TRACE   | Validating
2023/02/11 18:08:23 059971 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:08:23 060535 |  TRACE   |  - Start loss computation
2023/02/11 18:08:23 060964 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:09:43 222769 |  TRACE   |  - New best loss: 0.25550
2023/02/11 18:09:43 223307 |  TRACE   |  - Improvement over previous best: -0.01149
2023/02/11 18:09:43 351529 |  TRACE   |  - Best model updated!
 
2023/02/11 18:09:43 351925 |  TRACE   | Training for 200 steps (current step: 6200)
2023/02/11 18:14:07 132620 |  TRACE   | Validating
2023/02/11 18:14:34 905022 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:14:34 905575 |  TRACE   |  - Start loss computation
2023/02/11 18:14:34 906006 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:15:52 334930 |  TRACE   |  - New best loss: 0.25510
2023/02/11 18:15:52 335466 |  TRACE   |  - Improvement over previous best: -0.00039
2023/02/11 18:15:52 472875 |  TRACE   |  - Best model updated!
 
2023/02/11 18:15:52 473284 |  TRACE   | Training for 200 steps (current step: 6400)
2023/02/11 18:20:11 432980 |  TRACE   | Validating
2023/02/11 18:20:39 172493 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:20:39 173006 |  TRACE   |  - Start loss computation
2023/02/11 18:20:39 173438 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:21:59 771595 |  TRACE   |  - New best loss: 0.25271
2023/02/11 18:21:59 772152 |  TRACE   |  - Improvement over previous best: -0.00239
2023/02/11 18:21:59 899640 |  TRACE   |  - Best model updated!
 
2023/02/11 18:21:59 900029 |  TRACE   | Training for 200 steps (current step: 6600)
2023/02/11 18:26:25 329823 |  TRACE   | Validating
2023/02/11 18:26:53 101151 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:26:53 101711 |  TRACE   |  - Start loss computation
2023/02/11 18:26:53 102175 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:28:12 574445 |  TRACE   |  - No improvement for loss: 0.25357 (current best: 0.25271)
2023/02/11 18:28:12 574902 |  TRACE   | Training for 200 steps (current step: 6800)
2023/02/11 18:32:35 908203 |  TRACE   | Validating
2023/02/11 18:33:03 552723 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:33:03 553264 |  TRACE   |  - Start loss computation
2023/02/11 18:33:03 553694 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:34:23 326793 |  TRACE   |  - No improvement for loss: 0.25286 (current best: 0.25271)
2023/02/11 18:34:23 327244 |  TRACE   | Training for 200 steps (current step: 7000)
2023/02/11 18:38:43 242394 |  TRACE   | Validating
2023/02/11 18:39:10 839305 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:39:10 839861 |  TRACE   |  - Start loss computation
2023/02/11 18:39:10 840289 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:40:28 862487 |  TRACE   |  - No improvement for loss: 0.25329 (current best: 0.25271)
2023/02/11 18:40:28 862947 |  TRACE   | Training for 200 steps (current step: 7200)
2023/02/11 18:44:48 976313 |  TRACE   | Validating
2023/02/11 18:45:16 802393 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:45:16 802945 |  TRACE   |  - Start loss computation
2023/02/11 18:45:16 803387 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:46:39 000733 |  TRACE   |  - New best loss: 0.25186
2023/02/11 18:46:39 001291 |  TRACE   |  - Improvement over previous best: -0.00085
2023/02/11 18:46:39 133264 |  TRACE   |  - Best model updated!
 
2023/02/11 18:46:39 133663 |  TRACE   | Training for 200 steps (current step: 7400)
2023/02/11 18:51:02 713541 |  TRACE   | Validating
2023/02/11 18:51:30 366285 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:51:30 366883 |  TRACE   |  - Start loss computation
2023/02/11 18:51:30 367316 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:52:50 704946 |  TRACE   |  - No improvement for loss: 0.25323 (current best: 0.25186)
2023/02/11 18:52:50 705397 |  TRACE   | Training for 200 steps (current step: 7600)
2023/02/11 18:57:05 518378 |  TRACE   | Validating
2023/02/11 18:57:33 260769 |  TRACE   | Writing results to TensorBoard:
2023/02/11 18:57:33 261329 |  TRACE   |  - Start loss computation
2023/02/11 18:57:33 261804 |  TRACE   |  - Start image generation for 100 images
2023/02/11 18:58:49 019376 |  TRACE   |  - No improvement for loss: 0.25240 (current best: 0.25186)
2023/02/11 18:58:49 019834 |  TRACE   | Training for 200 steps (current step: 7800)
2023/02/11 19:03:15 318743 |  TRACE   | Validating
2023/02/11 19:03:43 017846 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:03:43 018397 |  TRACE   |  - Start loss computation
2023/02/11 19:03:43 018872 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:05:02 789345 |  TRACE   |  - New best loss: 0.25154
2023/02/11 19:05:02 789892 |  TRACE   |  - Improvement over previous best: -0.00031
2023/02/11 19:05:02 917211 |  TRACE   |  - Best model updated!
 
2023/02/11 19:05:02 917585 |  TRACE   | Training for 200 steps (current step: 8000)
2023/02/11 19:09:20 910693 |  TRACE   | Validating
2023/02/11 19:09:48 578685 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:09:48 579247 |  TRACE   |  - Start loss computation
2023/02/11 19:09:48 579707 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:11:05 587392 |  TRACE   |  - No improvement for loss: 0.25243 (current best: 0.25154)
2023/02/11 19:11:05 587853 |  TRACE   | Training for 200 steps (current step: 8200)
2023/02/11 19:15:23 438757 |  TRACE   | Validating
2023/02/11 19:15:51 049279 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:15:51 049799 |  TRACE   |  - Start loss computation
2023/02/11 19:15:51 050300 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:17:11 092120 |  TRACE   |  - New best loss: 0.25111
2023/02/11 19:17:11 092710 |  TRACE   |  - Improvement over previous best: -0.00043
2023/02/11 19:17:11 236013 |  TRACE   |  - Best model updated!
 
2023/02/11 19:17:11 236448 |  TRACE   | Training for 200 steps (current step: 8400)
2023/02/11 19:21:34 518842 |  TRACE   | Validating
2023/02/11 19:22:02 159144 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:22:02 159715 |  TRACE   |  - Start loss computation
2023/02/11 19:22:02 160153 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:23:21 549124 |  TRACE   |  - New best loss: 0.25100
2023/02/11 19:23:21 549741 |  TRACE   |  - Improvement over previous best: -0.00011
2023/02/11 19:23:21 679029 |  TRACE   |  - Best model updated!
 
2023/02/11 19:23:21 679406 |  TRACE   | Training for 200 steps (current step: 8600)
2023/02/11 19:27:44 865679 |  TRACE   | Validating
2023/02/11 19:28:12 552401 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:28:12 552967 |  TRACE   |  - Start loss computation
2023/02/11 19:28:12 553400 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:29:33 488878 |  TRACE   |  - New best loss: 0.25083
2023/02/11 19:29:33 489459 |  TRACE   |  - Improvement over previous best: -0.00018
2023/02/11 19:29:33 616280 |  TRACE   |  - Best model updated!
 
2023/02/11 19:29:33 616676 |  TRACE   | Training for 200 steps (current step: 8800)
2023/02/11 19:33:53 339670 |  TRACE   | Validating
2023/02/11 19:34:20 963208 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:34:20 963757 |  TRACE   |  - Start loss computation
2023/02/11 19:34:20 964190 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:35:36 560643 |  TRACE   |  - No improvement for loss: 0.25156 (current best: 0.25083)
2023/02/11 19:35:36 561109 |  TRACE   | Training for 200 steps (current step: 9000)
2023/02/11 19:39:57 584522 |  TRACE   | Validating
2023/02/11 19:40:25 343385 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:40:25 343992 |  TRACE   |  - Start loss computation
2023/02/11 19:40:25 344506 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:41:49 668763 |  TRACE   |  - New best loss: 0.24968
2023/02/11 19:41:49 669331 |  TRACE   |  - Improvement over previous best: -0.00114
2023/02/11 19:41:49 799671 |  TRACE   |  - Best model updated!
 
2023/02/11 19:41:49 800057 |  TRACE   | Training for 200 steps (current step: 9200)
2023/02/11 19:46:16 938623 |  TRACE   | Validating
2023/02/11 19:46:44 506954 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:46:44 507490 |  TRACE   |  - Start loss computation
2023/02/11 19:46:44 507991 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:48:05 066155 |  TRACE   |  - No improvement for loss: 0.24980 (current best: 0.24968)
2023/02/11 19:48:05 066677 |  TRACE   | Training for 200 steps (current step: 9400)
2023/02/11 19:52:28 173066 |  TRACE   | Validating
2023/02/11 19:52:55 880995 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:52:55 881585 |  TRACE   |  - Start loss computation
2023/02/11 19:52:55 882010 |  TRACE   |  - Start image generation for 100 images
2023/02/11 19:54:15 080454 |  TRACE   |  - New best loss: 0.24888
2023/02/11 19:54:15 080973 |  TRACE   |  - Improvement over previous best: -0.00081
2023/02/11 19:54:15 213849 |  TRACE   |  - Best model updated!
 
2023/02/11 19:54:15 214250 |  TRACE   | Training for 200 steps (current step: 9600)
2023/02/11 19:58:42 350080 |  TRACE   | Validating
2023/02/11 19:59:10 110647 |  TRACE   | Writing results to TensorBoard:
2023/02/11 19:59:10 111210 |  TRACE   |  - Start loss computation
2023/02/11 19:59:10 111670 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:00:27 914455 |  TRACE   |  - No improvement for loss: 0.25044 (current best: 0.24888)
2023/02/11 20:00:27 914909 |  TRACE   | Training for 200 steps (current step: 9800)
2023/02/11 20:04:44 995239 |  TRACE   | Validating
2023/02/11 20:05:12 751549 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:05:12 752103 |  TRACE   |  - Start loss computation
2023/02/11 20:05:12 752564 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:06:31 946829 |  TRACE   |  - No improvement for loss: 0.25056 (current best: 0.24888)
2023/02/11 20:06:31 947274 |  TRACE   | Training for 200 steps (current step: 10000)
2023/02/11 20:10:51 869218 |  TRACE   | Validating
2023/02/11 20:11:19 535004 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:11:19 535565 |  TRACE   |  - Start loss computation
2023/02/11 20:11:19 536022 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:12:39 332156 |  TRACE   |  - No improvement for loss: 0.24952 (current best: 0.24888)
2023/02/11 20:12:39 332683 |  TRACE   | Training for 200 steps (current step: 10200)
2023/02/11 20:17:03 907178 |  TRACE   | Validating
2023/02/11 20:17:31 578923 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:17:31 579512 |  TRACE   |  - Start loss computation
2023/02/11 20:17:31 580007 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:18:48 958688 |  TRACE   |  - New best loss: 0.24877
2023/02/11 20:18:48 959281 |  TRACE   |  - Improvement over previous best: -0.00011
2023/02/11 20:18:49 096159 |  TRACE   |  - Best model updated!
 
2023/02/11 20:18:49 096558 |  TRACE   | Training for 200 steps (current step: 10400)
2023/02/11 20:23:15 529369 |  TRACE   | Validating
2023/02/11 20:23:43 144816 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:23:43 145354 |  TRACE   |  - Start loss computation
2023/02/11 20:23:43 145785 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:25:04 440700 |  TRACE   |  - No improvement for loss: 0.24932 (current best: 0.24877)
2023/02/11 20:25:04 441169 |  TRACE   | Training for 200 steps (current step: 10600)
2023/02/11 20:29:23 707886 |  TRACE   | Validating
2023/02/11 20:29:51 308200 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:29:51 308791 |  TRACE   |  - Start loss computation
2023/02/11 20:29:51 309221 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:31:10 713407 |  TRACE   |  - No improvement for loss: 0.24974 (current best: 0.24877)
2023/02/11 20:31:10 714004 |  TRACE   | Training for 200 steps (current step: 10800)
2023/02/11 20:35:32 246068 |  TRACE   | Validating
2023/02/11 20:35:59 858348 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:35:59 858920 |  TRACE   |  - Start loss computation
2023/02/11 20:35:59 859353 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:37:20 545978 |  TRACE   |  - No improvement for loss: 0.25046 (current best: 0.24877)
2023/02/11 20:37:20 546431 |  TRACE   | Training for 200 steps (current step: 11000)
2023/02/11 20:41:43 685020 |  TRACE   | Validating
2023/02/11 20:42:11 422301 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:42:11 422851 |  TRACE   |  - Start loss computation
2023/02/11 20:42:11 423297 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:43:32 233331 |  TRACE   |  - New best loss: 0.24842
2023/02/11 20:43:32 233889 |  TRACE   |  - Improvement over previous best: -0.00035
2023/02/11 20:43:32 363723 |  TRACE   |  - Best model updated!
 
2023/02/11 20:43:32 364140 |  TRACE   | Training for 200 steps (current step: 11200)
2023/02/11 20:47:55 628107 |  TRACE   | Validating
2023/02/11 20:48:23 169737 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:48:23 170236 |  TRACE   |  - Start loss computation
2023/02/11 20:48:23 170665 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:49:40 141544 |  TRACE   |  - New best loss: 0.24827
2023/02/11 20:49:40 142066 |  TRACE   |  - Improvement over previous best: -0.00015
2023/02/11 20:49:40 270442 |  TRACE   |  - Best model updated!
 
2023/02/11 20:49:40 270798 |  TRACE   | Training for 200 steps (current step: 11400)
2023/02/11 20:54:05 765041 |  TRACE   | Validating
2023/02/11 20:54:33 388302 |  TRACE   | Writing results to TensorBoard:
2023/02/11 20:54:33 388880 |  TRACE   |  - Start loss computation
2023/02/11 20:54:33 389309 |  TRACE   |  - Start image generation for 100 images
2023/02/11 20:55:52 602600 |  TRACE   |  - New best loss: 0.24777
2023/02/11 20:55:52 603144 |  TRACE   |  - Improvement over previous best: -0.00050
2023/02/11 20:55:52 729337 |  TRACE   |  - Best model updated!
 
2023/02/11 20:55:52 729794 |  TRACE   | Training for 200 steps (current step: 11600)
2023/02/11 21:00:14 528200 |  TRACE   | Validating
2023/02/11 21:00:41 989978 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:00:41 990497 |  TRACE   |  - Start loss computation
2023/02/11 21:00:41 990922 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:02:00 712737 |  TRACE   |  - No improvement for loss: 0.24856 (current best: 0.24777)
2023/02/11 21:02:00 713297 |  TRACE   | Training for 200 steps (current step: 11800)
2023/02/11 21:06:19 606149 |  TRACE   | Validating
2023/02/11 21:06:47 271042 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:06:47 271594 |  TRACE   |  - Start loss computation
2023/02/11 21:06:47 272054 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:08:06 970077 |  TRACE   |  - No improvement for loss: 0.25061 (current best: 0.24777)
2023/02/11 21:08:06 970661 |  TRACE   | Training for 200 steps (current step: 12000)
2023/02/11 21:12:29 387296 |  TRACE   | Validating
2023/02/11 21:12:56 989538 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:12:56 990169 |  TRACE   |  - Start loss computation
2023/02/11 21:12:56 990623 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:14:15 888058 |  TRACE   |  - No improvement for loss: 0.25112 (current best: 0.24777)
2023/02/11 21:14:15 888556 |  TRACE   | Training for 200 steps (current step: 12200)
2023/02/11 21:18:44 541118 |  TRACE   | Validating
2023/02/11 21:19:12 446781 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:19:12 447347 |  TRACE   |  - Start loss computation
2023/02/11 21:19:12 447784 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:20:33 238485 |  TRACE   |  - New best loss: 0.24696
2023/02/11 21:20:33 239095 |  TRACE   |  - Improvement over previous best: -0.00080
2023/02/11 21:20:33 376662 |  TRACE   |  - Best model updated!
 
2023/02/11 21:20:33 377101 |  TRACE   | Training for 200 steps (current step: 12400)
2023/02/11 21:24:57 251337 |  TRACE   | Validating
2023/02/11 21:25:25 083534 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:25:25 084071 |  TRACE   |  - Start loss computation
2023/02/11 21:25:25 084536 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:26:44 147952 |  TRACE   |  - No improvement for loss: 0.25335 (current best: 0.24696)
2023/02/11 21:26:44 148527 |  TRACE   | Training for 200 steps (current step: 12600)
2023/02/11 21:31:08 128743 |  TRACE   | Validating
2023/02/11 21:31:36 077354 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:31:36 077925 |  TRACE   |  - Start loss computation
2023/02/11 21:31:36 078362 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:32:53 763551 |  TRACE   |  - No improvement for loss: 0.24813 (current best: 0.24696)
2023/02/11 21:32:53 764089 |  TRACE   | Training for 200 steps (current step: 12800)
2023/02/11 21:37:20 725249 |  TRACE   | Validating
2023/02/11 21:37:48 558436 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:37:48 558963 |  TRACE   |  - Start loss computation
2023/02/11 21:37:48 559484 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:39:09 445166 |  TRACE   |  - No improvement for loss: 0.24902 (current best: 0.24696)
2023/02/11 21:39:09 445728 |  TRACE   | Training for 200 steps (current step: 13000)
2023/02/11 21:43:30 837774 |  TRACE   | Validating
2023/02/11 21:43:58 719279 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:43:58 719875 |  TRACE   |  - Start loss computation
2023/02/11 21:43:58 720324 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:45:17 250960 |  TRACE   |  - No improvement for loss: 0.24837 (current best: 0.24696)
2023/02/11 21:45:17 251486 |  TRACE   | Training for 200 steps (current step: 13200)
2023/02/11 21:49:46 722382 |  TRACE   | Validating
2023/02/11 21:50:14 486769 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:50:14 487299 |  TRACE   |  - Start loss computation
2023/02/11 21:50:14 487796 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:51:35 294022 |  TRACE   |  - No improvement for loss: 0.24745 (current best: 0.24696)
2023/02/11 21:51:35 294557 |  TRACE   | Training for 200 steps (current step: 13400)
2023/02/11 21:55:58 806400 |  TRACE   | Validating
2023/02/11 21:56:26 709915 |  TRACE   | Writing results to TensorBoard:
2023/02/11 21:56:26 710500 |  TRACE   |  - Start loss computation
2023/02/11 21:56:26 710966 |  TRACE   |  - Start image generation for 100 images
2023/02/11 21:57:47 681900 |  TRACE   |  - No improvement for loss: 0.24741 (current best: 0.24696)
2023/02/11 21:57:47 682429 |  TRACE   | Training for 200 steps (current step: 13600)
2023/02/11 22:02:13 339095 |  TRACE   | Validating
2023/02/11 22:02:41 294409 |  TRACE   | Writing results to TensorBoard:
2023/02/11 22:02:41 294999 |  TRACE   |  - Start loss computation
2023/02/11 22:02:41 295439 |  TRACE   |  - Start image generation for 100 images
2023/02/11 22:04:00 049141 |  TRACE   |  - No improvement for loss: 0.24748 (current best: 0.24696)
2023/02/11 22:04:00 049725 |  TRACE   | Training for 200 steps (current step: 13800)
2023/02/11 22:08:21 512036 |  TRACE   | Validating
2023/02/11 22:08:49 325719 |  TRACE   | Writing results to TensorBoard:
2023/02/11 22:08:49 326299 |  TRACE   |  - Start loss computation
2023/02/11 22:08:49 326749 |  TRACE   |  - Start image generation for 100 images
2023/02/11 22:10:13 863687 |  TRACE   |  - No improvement for loss: 0.24729 (current best: 0.24696)
2023/02/11 22:10:13 864217 |  TRACE   | Training for 200 steps (current step: 14000)
2023/02/11 22:14:43 833945 |  TRACE   | Validating
2023/02/11 22:15:11 703973 |  TRACE   | Writing results to TensorBoard:
2023/02/11 22:15:11 704569 |  TRACE   |  - Start loss computation
2023/02/11 22:15:11 705011 |  TRACE   |  - Start image generation for 100 images
2023/02/11 22:16:32 327350 |  TRACE   |  - No improvement for loss: 0.24766 (current best: 0.24696)
2023/02/11 22:16:32 327886 |  TRACE   | Training for 200 steps (current step: 14200)
2023/02/11 22:20:55 477482 |  TRACE   | Validating
2023/02/11 22:21:23 366137 |  TRACE   | Writing results to TensorBoard:
2023/02/11 22:21:23 366681 |  TRACE   |  - Start loss computation
2023/02/11 22:21:23 367119 |  TRACE   |  - Start image generation for 100 images
2023/02/11 22:22:42 221199 |  TRACE   |  - No improvement for loss: 0.24774 (current best: 0.24696)
2023/02/11 22:22:42 221739 |  TRACE   |  - Early Stopping tolerance reached. Stopping training!
 
2023/02/11 22:22:42 938446 |  TRACE   | Start model evaluation
2023/02/14 11:29:48 483138 |  TRACE   | Start model evaluation
2023/02/14 11:32:35 203085 |  TRACE   | Evaluation results:
2023/02/14 11:32:35 203450 |  TRACE   |  - Results for field size: global
2023/02/14 11:32:35 203914 |  TRACE   |    - Precision: 45.29%
2023/02/14 11:32:35 204176 |  TRACE   |    - Recall: 50.35%
2023/02/14 11:32:35 204361 |  TRACE   |  - Results for field size: small
2023/02/14 11:32:35 204580 |  TRACE   |    - Precision: 26.33%
2023/02/14 11:32:35 204799 |  TRACE   |    - Recall: 30.78%
2023/02/14 11:32:35 204973 |  TRACE   |  - Results for field size: medium
2023/02/14 11:32:35 205184 |  TRACE   |    - Precision: 59.83%
2023/02/14 11:32:35 205403 |  TRACE   |    - Recall: 54.51%
2023/02/14 11:32:35 205577 |  TRACE   |  - Results for field size: large
2023/02/14 11:32:35 205793 |  TRACE   |    - Precision: 70.85%
2023/02/14 11:32:35 206009 |  TRACE   |    - Recall: 65.64%
2023/02/14 11:32:35 206244 |  TRACE   |  - Results for field size: extra large
2023/02/14 11:32:35 206480 |  TRACE   |    - Precision: 72.37%
2023/02/14 11:32:35 206703 |  TRACE   |    - Recall: 71.08%
