2023/06/22 16:47:44 105575 |  TRACE   |  Model configs:
2023/06/22 16:47:44 105905 |  TRACE   |  - Model: 20230622T164738-MultiHeadResUnet3D_ts24_noAugm
2023/06/22 16:47:44 106038 |  TRACE   |  - Parameters: 1_810_258
2023/06/22 16:47:44 106143 |  TRACE   |  - Device: cuda
2023/06/22 16:47:44 106249 |  TRACE   |  - Input bands: ['s2_ndvi', 's2_b02', 's2_b03', 's2_b04']
2023/06/22 16:47:44 106351 |  TRACE   |  - N ts: 24
 
2023/06/22 16:47:44 106452 |  TRACE   |  Steps configs:
2023/06/22 16:47:44 106555 |  TRACE   |  - Maximum steps: None
2023/06/22 16:47:44 106656 |  TRACE   |  - Validation steps: 250
2023/06/22 16:47:44 106756 |  TRACE   |  - Batch-size: 16
 
2023/06/22 16:47:44 106856 |  TRACE   |  Optimizer configs:
2023/06/22 16:47:44 106956 |  TRACE   |  - Optimizer: Adam
2023/06/22 16:47:44 107057 |  TRACE   |  - Early-stopping tolerance: 10
2023/06/22 16:47:44 107164 |  TRACE   |  - Early-stopping improvement: 0.0001
2023/06/22 16:47:44 107263 |  TRACE   |  - Early-stopping metric: loss
2023/06/22 16:47:44 107367 |  TRACE   |  - Scheduler: ReduceLROnPlateau
2023/06/22 16:47:44 107492 |  TRACE   |  - Learning rate tolerance: 3
 
2023/06/22 16:47:44 107597 |  TRACE   |  Loss configs:
2023/06/22 16:47:44 107698 |  TRACE   |  - Tag: Loss_ext
2023/06/22 16:47:44 107797 |  TRACE   |  - Loss: dice_loss
2023/06/22 16:47:44 107899 |  TRACE   |  - Weight / multiplier: {'watersheds': 10}
2023/06/22 16:47:44 107998 |  TRACE   |  - Target mask: extent
2023/06/22 16:47:44 108095 |  TRACE   |  - LC coefficient: 1
2023/06/22 16:47:44 108192 |  TRACE   |  ---
2023/06/22 16:47:44 108290 |  TRACE   |  - Tag: Loss_ws
2023/06/22 16:47:44 108390 |  TRACE   |  - Loss: binary_cross_entropy
2023/06/22 16:47:44 108505 |  TRACE   |  - Weight / multiplier: {'borders': 4}
2023/06/22 16:47:44 108604 |  TRACE   |  - Target mask: borders
2023/06/22 16:47:44 108701 |  TRACE   |  - LC coefficient: 1
2023/06/22 16:47:44 108797 |  TRACE   | 

2023/06/22 16:49:07 272430 |  TRACE   | Writing results to TensorBoard:
2023/06/22 16:49:07 272795 |  TRACE   |  - Start loss computation
2023/06/22 16:49:07 273092 |  TRACE   |  - Start image generation for 100 images
2023/06/22 16:49:10 832017 |  TRACE   | 

2023/06/22 16:49:10 832357 |  TRACE   | Training for 250 steps (current step: 0)
2023/06/22 16:59:26 241566 |  TRACE   | Validating
2023/06/22 17:00:41 606464 |  TRACE   | Writing results to TensorBoard:
2023/06/22 17:00:41 606817 |  TRACE   |  - Start loss computation
2023/06/22 17:00:41 607077 |  TRACE   |  - Start image generation for 100 images
2023/06/22 17:01:48 129138 |  TRACE   |  - New best loss: 1.23070
2023/06/22 17:01:48 129525 |  TRACE   |  - Improvement over previous best: -0.56291
2023/06/22 17:01:48 178779 |  TRACE   |  - Best model updated!
 
2023/06/22 17:01:48 179019 |  TRACE   | Training for 250 steps (current step: 250)
2023/06/22 17:11:21 377938 |  TRACE   | Validating
2023/06/22 17:12:36 770010 |  TRACE   | Writing results to TensorBoard:
2023/06/22 17:12:36 770338 |  TRACE   |  - Start loss computation
2023/06/22 17:12:36 770583 |  TRACE   |  - Start image generation for 100 images
2023/06/22 17:13:36 170112 |  TRACE   |  - New best loss: 1.12306
2023/06/22 17:13:36 170501 |  TRACE   |  - Improvement over previous best: -0.10764
2023/06/22 17:13:36 212106 |  TRACE   |  - Best model updated!
 
2023/06/22 17:13:36 212317 |  TRACE   | Training for 250 steps (current step: 500)
2023/06/22 17:23:22 448610 |  TRACE   | Validating
2023/06/22 17:24:39 250208 |  TRACE   | Writing results to TensorBoard:
2023/06/22 17:24:39 250534 |  TRACE   |  - Start loss computation
2023/06/22 17:24:39 250778 |  TRACE   |  - Start image generation for 100 images
2023/06/22 17:25:40 197087 |  TRACE   |  - New best loss: 1.10201
2023/06/22 17:25:40 197508 |  TRACE   |  - Improvement over previous best: -0.02105
2023/06/22 17:25:40 239736 |  TRACE   |  - Best model updated!
 
2023/06/22 17:25:40 239989 |  TRACE   | Training for 250 steps (current step: 750)
2023/06/22 17:35:06 474765 |  TRACE   | Validating
2023/06/22 17:36:19 921520 |  TRACE   | Writing results to TensorBoard:
2023/06/22 17:36:19 921842 |  TRACE   |  - Start loss computation
2023/06/22 17:36:19 922097 |  TRACE   |  - Start image generation for 100 images
2023/06/22 17:37:16 363519 |  TRACE   |  - No improvement for loss: 1.10398 (current best: 1.10201)
2023/06/22 17:37:16 363867 |  TRACE   | Training for 250 steps (current step: 1000)
2023/06/22 17:46:39 650422 |  TRACE   | Validating
2023/06/22 17:47:52 625248 |  TRACE   | Writing results to TensorBoard:
2023/06/22 17:47:52 625635 |  TRACE   |  - Start loss computation
2023/06/22 17:47:52 625895 |  TRACE   |  - Start image generation for 100 images
2023/06/22 17:48:59 791884 |  TRACE   |  - New best loss: 1.08683
2023/06/22 17:48:59 792285 |  TRACE   |  - Improvement over previous best: -0.01518
2023/06/22 17:48:59 836398 |  TRACE   |  - Best model updated!
 
2023/06/22 17:48:59 836624 |  TRACE   | Training for 250 steps (current step: 1250)
2023/06/22 17:58:03 626065 |  TRACE   | Validating
2023/06/22 17:59:14 731055 |  TRACE   | Writing results to TensorBoard:
2023/06/22 17:59:14 731415 |  TRACE   |  - Start loss computation
2023/06/22 17:59:14 731688 |  TRACE   |  - Start image generation for 100 images
2023/06/22 18:00:18 363947 |  TRACE   |  - New best loss: 1.07068
2023/06/22 18:00:18 364351 |  TRACE   |  - Improvement over previous best: -0.01615
2023/06/22 18:00:18 417079 |  TRACE   |  - Best model updated!
 
2023/06/22 18:00:18 417299 |  TRACE   | Training for 250 steps (current step: 1500)
2023/06/22 18:09:43 141313 |  TRACE   | Validating
2023/06/22 18:10:53 256160 |  TRACE   | Writing results to TensorBoard:
2023/06/22 18:10:53 256521 |  TRACE   |  - Start loss computation
2023/06/22 18:10:53 256771 |  TRACE   |  - Start image generation for 100 images
2023/06/22 18:11:52 346856 |  TRACE   |  - No improvement for loss: 1.07139 (current best: 1.07068)
2023/06/22 18:11:52 347207 |  TRACE   | Training for 250 steps (current step: 1750)
2023/06/22 18:20:58 406403 |  TRACE   | Validating
2023/06/22 18:22:09 022043 |  TRACE   | Writing results to TensorBoard:
2023/06/22 18:22:09 023052 |  TRACE   |  - Start loss computation
2023/06/22 18:22:09 023315 |  TRACE   |  - Start image generation for 100 images
2023/06/22 18:23:07 478204 |  TRACE   |  - New best loss: 1.07020
2023/06/22 18:23:07 478584 |  TRACE   |  - Improvement over previous best: -0.00048
2023/06/22 18:23:07 519156 |  TRACE   |  - Best model updated!
 
2023/06/22 18:23:07 519387 |  TRACE   | Training for 250 steps (current step: 2000)
2023/06/22 18:32:13 394686 |  TRACE   | Validating
2023/06/22 18:33:24 460993 |  TRACE   | Writing results to TensorBoard:
2023/06/22 18:33:24 461761 |  TRACE   |  - Start loss computation
2023/06/22 18:33:24 462016 |  TRACE   |  - Start image generation for 100 images
2023/06/22 18:34:23 508801 |  TRACE   |  - New best loss: 1.06668
2023/06/22 18:34:23 509195 |  TRACE   |  - Improvement over previous best: -0.00352
2023/06/22 18:34:23 550874 |  TRACE   |  - Best model updated!
 
2023/06/22 18:34:23 551082 |  TRACE   | Training for 250 steps (current step: 2250)
2023/06/22 18:43:30 701814 |  TRACE   | Validating
2023/06/22 18:44:41 663136 |  TRACE   | Writing results to TensorBoard:
2023/06/22 18:44:41 664189 |  TRACE   |  - Start loss computation
2023/06/22 18:44:41 664476 |  TRACE   |  - Start image generation for 100 images
2023/06/22 18:45:42 889661 |  TRACE   |  - No improvement for loss: 1.06685 (current best: 1.06668)
2023/06/22 18:45:42 890013 |  TRACE   | Training for 250 steps (current step: 2500)
2023/06/22 18:54:37 211407 |  TRACE   | Validating
2023/06/22 18:55:50 049788 |  TRACE   | Writing results to TensorBoard:
2023/06/22 18:55:50 050495 |  TRACE   |  - Start loss computation
2023/06/22 18:55:50 050759 |  TRACE   |  - Start image generation for 100 images
2023/06/22 18:56:53 548042 |  TRACE   |  - No improvement for loss: 1.06782 (current best: 1.06668)
2023/06/22 18:56:53 548451 |  TRACE   | Training for 250 steps (current step: 2750)
2023/06/22 19:05:56 154410 |  TRACE   | Validating
2023/06/22 19:07:06 699219 |  TRACE   | Writing results to TensorBoard:
2023/06/22 19:07:06 700228 |  TRACE   |  - Start loss computation
2023/06/22 19:07:06 700490 |  TRACE   |  - Start image generation for 100 images
2023/06/22 19:08:06 080116 |  TRACE   |  - New best loss: 1.04604
2023/06/22 19:08:06 080518 |  TRACE   |  - Improvement over previous best: -0.02063
2023/06/22 19:08:06 125466 |  TRACE   |  - Best model updated!
 
2023/06/22 19:08:06 125678 |  TRACE   | Training for 250 steps (current step: 3000)
2023/06/22 19:17:03 621363 |  TRACE   | Validating
2023/06/22 19:18:12 318164 |  TRACE   | Writing results to TensorBoard:
2023/06/22 19:18:12 319173 |  TRACE   |  - Start loss computation
2023/06/22 19:18:12 319436 |  TRACE   |  - Start image generation for 100 images
2023/06/22 19:19:09 873633 |  TRACE   |  - No improvement for loss: 1.04598 (current best: 1.04604)
2023/06/22 19:19:09 873968 |  TRACE   | Training for 250 steps (current step: 3250)
2023/06/22 19:28:09 712252 |  TRACE   | Validating
2023/06/22 19:29:17 893744 |  TRACE   | Writing results to TensorBoard:
2023/06/22 19:29:17 894091 |  TRACE   |  - Start loss computation
2023/06/22 19:29:17 894351 |  TRACE   |  - Start image generation for 100 images
2023/06/22 19:30:06 974927 |  TRACE   |  - No improvement for loss: 1.10605 (current best: 1.04604)
2023/06/22 19:30:06 975283 |  TRACE   | Training for 250 steps (current step: 3500)
2023/06/22 19:39:04 694889 |  TRACE   | Validating
2023/06/22 19:40:16 695246 |  TRACE   | Writing results to TensorBoard:
2023/06/22 19:40:16 695966 |  TRACE   |  - Start loss computation
2023/06/22 19:40:16 696232 |  TRACE   |  - Start image generation for 100 images
2023/06/22 19:41:26 873184 |  TRACE   |  - No improvement for loss: 1.14782 (current best: 1.04604)
2023/06/22 19:41:26 873657 |  TRACE   | Training for 250 steps (current step: 3750)
2023/06/22 19:50:27 373197 |  TRACE   | Validating
2023/06/22 19:51:37 717550 |  TRACE   | Writing results to TensorBoard:
2023/06/22 19:51:37 718307 |  TRACE   |  - Start loss computation
2023/06/22 19:51:37 718586 |  TRACE   |  - Start image generation for 100 images
2023/06/22 19:52:37 452910 |  TRACE   |  - No improvement for loss: 1.07157 (current best: 1.04604)
2023/06/22 19:52:37 453205 |  TRACE   | Training for 250 steps (current step: 4000)
2023/06/22 20:01:39 032879 |  TRACE   | Validating
2023/06/22 20:02:50 317422 |  TRACE   | Writing results to TensorBoard:
2023/06/22 20:02:50 318271 |  TRACE   |  - Start loss computation
2023/06/22 20:02:50 318539 |  TRACE   |  - Start image generation for 100 images
2023/06/22 20:03:47 122128 |  TRACE   |  - New best loss: 1.03274
2023/06/22 20:03:47 122525 |  TRACE   |  - Improvement over previous best: -0.01330
2023/06/22 20:03:47 166659 |  TRACE   |  - Best model updated!
 
2023/06/22 20:03:47 166886 |  TRACE   | Training for 250 steps (current step: 4250)
2023/06/22 20:12:38 725346 |  TRACE   | Validating
2023/06/22 20:13:47 388745 |  TRACE   | Writing results to TensorBoard:
2023/06/22 20:13:47 389064 |  TRACE   |  - Start loss computation
2023/06/22 20:13:47 389334 |  TRACE   |  - Start image generation for 100 images
2023/06/22 20:14:43 420571 |  TRACE   |  - New best loss: 1.03226
2023/06/22 20:14:43 421015 |  TRACE   |  - Improvement over previous best: -0.00048
2023/06/22 20:14:43 460988 |  TRACE   |  - Best model updated!
 
2023/06/22 20:14:43 461198 |  TRACE   | Training for 250 steps (current step: 4500)
2023/06/22 20:23:47 786092 |  TRACE   | Validating
2023/06/22 20:24:56 092009 |  TRACE   | Writing results to TensorBoard:
2023/06/22 20:24:56 092341 |  TRACE   |  - Start loss computation
2023/06/22 20:24:56 092636 |  TRACE   |  - Start image generation for 100 images
2023/06/22 20:25:53 882151 |  TRACE   |  - No improvement for loss: 1.03217 (current best: 1.03226)
2023/06/22 20:25:53 882506 |  TRACE   | Training for 250 steps (current step: 4750)
2023/06/22 20:34:52 456588 |  TRACE   | Validating
2023/06/22 20:36:00 564421 |  TRACE   | Writing results to TensorBoard:
2023/06/22 20:36:00 564750 |  TRACE   |  - Start loss computation
2023/06/22 20:36:00 564997 |  TRACE   |  - Start image generation for 100 images
2023/06/22 20:36:57 556081 |  TRACE   |  - New best loss: 1.03097
2023/06/22 20:36:57 556487 |  TRACE   |  - Improvement over previous best: -0.00129
2023/06/22 20:36:57 602500 |  TRACE   |  - Best model updated!
 
2023/06/22 20:36:57 602720 |  TRACE   | Training for 250 steps (current step: 5000)
2023/06/22 20:45:48 972028 |  TRACE   | Validating
2023/06/22 20:46:56 861534 |  TRACE   | Writing results to TensorBoard:
2023/06/22 20:46:56 862394 |  TRACE   |  - Start loss computation
2023/06/22 20:46:56 862655 |  TRACE   |  - Start image generation for 100 images
2023/06/22 20:47:51 579093 |  TRACE   |  - No improvement for loss: 1.03184 (current best: 1.03097)
2023/06/22 20:47:51 579442 |  TRACE   | Training for 250 steps (current step: 5250)
2023/06/22 20:56:35 040254 |  TRACE   | Validating
2023/06/22 20:57:44 710060 |  TRACE   | Writing results to TensorBoard:
2023/06/22 20:57:44 711156 |  TRACE   |  - Start loss computation
2023/06/22 20:57:44 711424 |  TRACE   |  - Start image generation for 100 images
2023/06/22 20:58:42 931459 |  TRACE   |  - New best loss: 1.03069
2023/06/22 20:58:42 931854 |  TRACE   |  - Improvement over previous best: -0.00028
2023/06/22 20:58:42 972318 |  TRACE   |  - Best model updated!
 
2023/06/22 20:58:42 972534 |  TRACE   | Training for 250 steps (current step: 5500)
2023/06/22 21:07:56 409397 |  TRACE   | Validating
2023/06/22 21:09:07 078274 |  TRACE   | Writing results to TensorBoard:
2023/06/22 21:09:07 079259 |  TRACE   |  - Start loss computation
2023/06/22 21:09:07 079536 |  TRACE   |  - Start image generation for 100 images
2023/06/22 21:10:02 616614 |  TRACE   |  - No improvement for loss: 1.03150 (current best: 1.03069)
2023/06/22 21:10:02 616953 |  TRACE   | Training for 250 steps (current step: 5750)
2023/06/22 21:18:58 975854 |  TRACE   | Validating
2023/06/22 21:20:08 242703 |  TRACE   | Writing results to TensorBoard:
2023/06/22 21:20:08 243025 |  TRACE   |  - Start loss computation
2023/06/22 21:20:08 243275 |  TRACE   |  - Start image generation for 100 images
2023/06/22 21:21:06 948812 |  TRACE   |  - New best loss: 1.03056
2023/06/22 21:21:06 949195 |  TRACE   |  - Improvement over previous best: -0.00013
2023/06/22 21:21:06 994072 |  TRACE   |  - Best model updated!
 
2023/06/22 21:21:06 994347 |  TRACE   | Training for 250 steps (current step: 6000)
2023/06/22 21:30:03 155043 |  TRACE   | Validating
2023/06/22 21:31:12 644719 |  TRACE   | Writing results to TensorBoard:
2023/06/22 21:31:12 645401 |  TRACE   |  - Start loss computation
2023/06/22 21:31:12 645760 |  TRACE   |  - Start image generation for 100 images
2023/06/22 21:32:10 076509 |  TRACE   |  - New best loss: 1.02919
2023/06/22 21:32:10 076914 |  TRACE   |  - Improvement over previous best: -0.00137
2023/06/22 21:32:10 123902 |  TRACE   |  - Best model updated!
 
2023/06/22 21:32:10 124120 |  TRACE   | Training for 250 steps (current step: 6250)
2023/06/22 21:41:09 059534 |  TRACE   | Validating
2023/06/22 21:42:16 678029 |  TRACE   | Writing results to TensorBoard:
2023/06/22 21:42:16 679058 |  TRACE   |  - Start loss computation
2023/06/22 21:42:16 679386 |  TRACE   |  - Start image generation for 100 images
2023/06/22 21:43:13 893211 |  TRACE   |  - New best loss: 1.02883
2023/06/22 21:43:13 893700 |  TRACE   |  - Improvement over previous best: -0.00035
2023/06/22 21:43:13 934132 |  TRACE   |  - Best model updated!
 
2023/06/22 21:43:13 934343 |  TRACE   | Training for 250 steps (current step: 6500)
2023/06/22 21:52:07 435158 |  TRACE   | Validating
2023/06/22 21:53:15 675312 |  TRACE   | Writing results to TensorBoard:
2023/06/22 21:53:15 675631 |  TRACE   |  - Start loss computation
2023/06/22 21:53:15 675883 |  TRACE   |  - Start image generation for 100 images
2023/06/22 21:54:13 305931 |  TRACE   |  - No improvement for loss: 1.02888 (current best: 1.02883)
2023/06/22 21:54:13 306286 |  TRACE   | Training for 250 steps (current step: 6750)
2023/06/22 22:03:17 647184 |  TRACE   | Validating
2023/06/22 22:04:28 075455 |  TRACE   | Writing results to TensorBoard:
2023/06/22 22:04:28 076211 |  TRACE   |  - Start loss computation
2023/06/22 22:04:28 076531 |  TRACE   |  - Start image generation for 100 images
2023/06/22 22:05:26 525787 |  TRACE   |  - No improvement for loss: 1.02956 (current best: 1.02883)
2023/06/22 22:05:26 526156 |  TRACE   | Training for 250 steps (current step: 7000)
2023/06/22 22:14:33 867800 |  TRACE   | Validating
2023/06/22 22:15:45 016395 |  TRACE   | Writing results to TensorBoard:
2023/06/22 22:15:45 017452 |  TRACE   |  - Start loss computation
2023/06/22 22:15:45 017766 |  TRACE   |  - Start image generation for 100 images
2023/06/22 22:16:40 619983 |  TRACE   |  - No improvement for loss: 1.02875 (current best: 1.02883)
2023/06/22 22:16:40 620335 |  TRACE   | Training for 250 steps (current step: 7250)
2023/06/22 22:25:51 470777 |  TRACE   | Validating
2023/06/22 22:27:01 124013 |  TRACE   | Writing results to TensorBoard:
2023/06/22 22:27:01 124333 |  TRACE   |  - Start loss computation
2023/06/22 22:27:01 124576 |  TRACE   |  - Start image generation for 100 images
2023/06/22 22:27:58 869033 |  TRACE   |  - New best loss: 1.02859
2023/06/22 22:27:58 869566 |  TRACE   |  - Improvement over previous best: -0.00024
2023/06/22 22:27:58 912269 |  TRACE   |  - Best model updated!
 
2023/06/22 22:27:58 912483 |  TRACE   | Training for 250 steps (current step: 7500)
2023/06/22 22:37:02 903365 |  TRACE   | Validating
2023/06/22 22:38:14 832208 |  TRACE   | Writing results to TensorBoard:
2023/06/22 22:38:14 833316 |  TRACE   |  - Start loss computation
2023/06/22 22:38:14 833695 |  TRACE   |  - Start image generation for 100 images
2023/06/22 22:39:10 487537 |  TRACE   |  - New best loss: 1.02822
2023/06/22 22:39:10 487947 |  TRACE   |  - Improvement over previous best: -0.00037
2023/06/22 22:39:10 533150 |  TRACE   |  - Best model updated!
 
2023/06/22 22:39:10 533411 |  TRACE   | Training for 250 steps (current step: 7750)
2023/06/22 22:48:07 751427 |  TRACE   | Validating
2023/06/22 22:49:18 566512 |  TRACE   | Writing results to TensorBoard:
2023/06/22 22:49:18 567545 |  TRACE   |  - Start loss computation
2023/06/22 22:49:18 567811 |  TRACE   |  - Start image generation for 100 images
2023/06/22 22:50:16 000493 |  TRACE   |  - New best loss: 1.02804
2023/06/22 22:50:16 000994 |  TRACE   |  - Improvement over previous best: -0.00018
2023/06/22 22:50:16 047337 |  TRACE   |  - Best model updated!
 
2023/06/22 22:50:16 047590 |  TRACE   | Training for 250 steps (current step: 8000)
2023/06/22 22:59:15 963503 |  TRACE   | Validating
2023/06/22 23:00:25 245706 |  TRACE   | Writing results to TensorBoard:
2023/06/22 23:00:25 246588 |  TRACE   |  - Start loss computation
2023/06/22 23:00:25 246911 |  TRACE   |  - Start image generation for 100 images
2023/06/22 23:01:22 914416 |  TRACE   |  - No improvement for loss: 1.02884 (current best: 1.02804)
2023/06/22 23:01:22 914767 |  TRACE   | Training for 250 steps (current step: 8250)
2023/06/22 23:10:23 260946 |  TRACE   | Validating
2023/06/22 23:11:32 976630 |  TRACE   | Writing results to TensorBoard:
2023/06/22 23:11:32 977615 |  TRACE   |  - Start loss computation
2023/06/22 23:11:32 977963 |  TRACE   |  - Start image generation for 100 images
2023/06/22 23:12:30 308018 |  TRACE   |  - New best loss: 1.02772
2023/06/22 23:12:30 308423 |  TRACE   |  - Improvement over previous best: -0.00032
2023/06/22 23:12:30 359114 |  TRACE   |  - Best model updated!
 
2023/06/22 23:12:30 359345 |  TRACE   | Training for 250 steps (current step: 8500)
2023/06/22 23:21:27 287299 |  TRACE   | Validating
2023/06/22 23:22:38 590736 |  TRACE   | Writing results to TensorBoard:
2023/06/22 23:22:38 591695 |  TRACE   |  - Start loss computation
2023/06/22 23:22:38 592019 |  TRACE   |  - Start image generation for 100 images
2023/06/22 23:23:35 204142 |  TRACE   |  - New best loss: 1.02721
2023/06/22 23:23:35 204529 |  TRACE   |  - Improvement over previous best: -0.00052
2023/06/22 23:23:35 249778 |  TRACE   |  - Best model updated!
 
2023/06/22 23:23:35 249998 |  TRACE   | Training for 250 steps (current step: 8750)
2023/06/22 23:32:26 836759 |  TRACE   | Validating
2023/06/22 23:33:37 556988 |  TRACE   | Writing results to TensorBoard:
2023/06/22 23:33:37 557900 |  TRACE   |  - Start loss computation
2023/06/22 23:33:37 558176 |  TRACE   |  - Start image generation for 100 images
2023/06/22 23:34:32 605092 |  TRACE   |  - No improvement for loss: 1.02815 (current best: 1.02721)
2023/06/22 23:34:32 605525 |  TRACE   | Training for 250 steps (current step: 9000)
2023/06/22 23:43:40 057462 |  TRACE   | Validating
2023/06/22 23:44:50 471646 |  TRACE   | Writing results to TensorBoard:
2023/06/22 23:44:50 472351 |  TRACE   |  - Start loss computation
2023/06/22 23:44:50 472642 |  TRACE   |  - Start image generation for 100 images
2023/06/22 23:45:46 262451 |  TRACE   |  - No improvement for loss: 1.02798 (current best: 1.02721)
2023/06/22 23:45:46 262813 |  TRACE   | Training for 250 steps (current step: 9250)
2023/06/22 23:54:37 216041 |  TRACE   | Validating
2023/06/22 23:55:45 383483 |  TRACE   | Writing results to TensorBoard:
2023/06/22 23:55:45 384169 |  TRACE   |  - Start loss computation
2023/06/22 23:55:45 384434 |  TRACE   |  - Start image generation for 100 images
2023/06/22 23:56:42 110990 |  TRACE   |  - New best loss: 1.02615
2023/06/22 23:56:42 111392 |  TRACE   |  - Improvement over previous best: -0.00106
2023/06/22 23:56:42 158213 |  TRACE   |  - Best model updated!
 
2023/06/22 23:56:42 158477 |  TRACE   | Training for 250 steps (current step: 9500)
2023/06/23 00:05:53 912982 |  TRACE   | Validating
2023/06/23 00:07:03 112381 |  TRACE   | Writing results to TensorBoard:
2023/06/23 00:07:03 113292 |  TRACE   |  - Start loss computation
2023/06/23 00:07:03 113618 |  TRACE   |  - Start image generation for 100 images
2023/06/23 00:07:59 375278 |  TRACE   |  - No improvement for loss: 1.02692 (current best: 1.02615)
2023/06/23 00:07:59 375618 |  TRACE   | Training for 250 steps (current step: 9750)
2023/06/23 00:16:51 532079 |  TRACE   | Validating
2023/06/23 00:18:00 788144 |  TRACE   | Writing results to TensorBoard:
2023/06/23 00:18:00 788881 |  TRACE   |  - Start loss computation
2023/06/23 00:18:00 789163 |  TRACE   |  - Start image generation for 100 images
2023/06/23 00:18:56 070008 |  TRACE   |  - No improvement for loss: 1.02888 (current best: 1.02615)
2023/06/23 00:18:56 070369 |  TRACE   | Training for 250 steps (current step: 10000)
2023/06/23 00:27:50 302528 |  TRACE   | Validating
2023/06/23 00:28:58 737548 |  TRACE   | Writing results to TensorBoard:
2023/06/23 00:28:58 738389 |  TRACE   |  - Start loss computation
2023/06/23 00:28:58 738683 |  TRACE   |  - Start image generation for 100 images
2023/06/23 00:29:56 109764 |  TRACE   |  - No improvement for loss: 1.02685 (current best: 1.02615)
2023/06/23 00:29:56 110119 |  TRACE   | Training for 250 steps (current step: 10250)
2023/06/23 00:39:00 480196 |  TRACE   | Validating
2023/06/23 00:40:08 969887 |  TRACE   | Writing results to TensorBoard:
2023/06/23 00:40:08 970235 |  TRACE   |  - Start loss computation
2023/06/23 00:40:08 970481 |  TRACE   |  - Start image generation for 100 images
2023/06/23 00:41:04 241994 |  TRACE   |  - No improvement for loss: 1.02689 (current best: 1.02615)
2023/06/23 00:41:04 242298 |  TRACE   | Training for 250 steps (current step: 10500)
2023/06/23 00:50:05 288996 |  TRACE   | Validating
2023/06/23 00:51:15 087619 |  TRACE   | Writing results to TensorBoard:
2023/06/23 00:51:15 088496 |  TRACE   |  - Start loss computation
2023/06/23 00:51:15 088759 |  TRACE   |  - Start image generation for 100 images
2023/06/23 00:52:11 551219 |  TRACE   |  - New best loss: 1.02507
2023/06/23 00:52:11 551622 |  TRACE   |  - Improvement over previous best: -0.00108
2023/06/23 00:52:11 601411 |  TRACE   |  - Best model updated!
 
2023/06/23 00:52:11 601674 |  TRACE   | Training for 250 steps (current step: 10750)
2023/06/23 01:01:02 552524 |  TRACE   | Validating
2023/06/23 01:02:14 252811 |  TRACE   | Writing results to TensorBoard:
2023/06/23 01:02:14 253326 |  TRACE   |  - Start loss computation
2023/06/23 01:02:14 253590 |  TRACE   |  - Start image generation for 100 images
2023/06/23 01:03:10 491191 |  TRACE   |  - New best loss: 1.02494
2023/06/23 01:03:10 491582 |  TRACE   |  - Improvement over previous best: -0.00013
2023/06/23 01:03:10 540752 |  TRACE   |  - Best model updated!
 
2023/06/23 01:03:10 541072 |  TRACE   | Training for 250 steps (current step: 11000)
2023/06/23 01:12:07 108232 |  TRACE   | Validating
2023/06/23 01:13:15 332579 |  TRACE   | Writing results to TensorBoard:
2023/06/23 01:13:15 333321 |  TRACE   |  - Start loss computation
2023/06/23 01:13:15 333661 |  TRACE   |  - Start image generation for 100 images
2023/06/23 01:14:12 145744 |  TRACE   |  - No improvement for loss: 1.02485 (current best: 1.02494)
2023/06/23 01:14:12 146131 |  TRACE   | Training for 250 steps (current step: 11250)
2023/06/23 01:23:01 586847 |  TRACE   | Validating
2023/06/23 01:24:11 771920 |  TRACE   | Writing results to TensorBoard:
2023/06/23 01:24:11 772248 |  TRACE   |  - Start loss computation
2023/06/23 01:24:11 772525 |  TRACE   |  - Start image generation for 100 images
2023/06/23 01:25:08 726493 |  TRACE   |  - No improvement for loss: 1.02494 (current best: 1.02494)
2023/06/23 01:25:08 726860 |  TRACE   | Training for 250 steps (current step: 11500)
2023/06/23 01:34:02 638378 |  TRACE   | Validating
2023/06/23 01:35:10 066199 |  TRACE   | Writing results to TensorBoard:
2023/06/23 01:35:10 066952 |  TRACE   |  - Start loss computation
2023/06/23 01:35:10 067220 |  TRACE   |  - Start image generation for 100 images
2023/06/23 01:36:06 226834 |  TRACE   |  - No improvement for loss: 1.02512 (current best: 1.02494)
2023/06/23 01:36:06 227189 |  TRACE   | Training for 250 steps (current step: 11750)
2023/06/23 01:44:47 761547 |  TRACE   | Validating
2023/06/23 01:45:56 636851 |  TRACE   | Writing results to TensorBoard:
2023/06/23 01:45:56 637725 |  TRACE   |  - Start loss computation
2023/06/23 01:45:56 638002 |  TRACE   |  - Start image generation for 100 images
2023/06/23 01:46:54 241530 |  TRACE   |  - New best loss: 1.02473
2023/06/23 01:46:54 241979 |  TRACE   |  - Improvement over previous best: -0.00021
2023/06/23 01:46:54 282554 |  TRACE   |  - Best model updated!
 
2023/06/23 01:46:54 282772 |  TRACE   | Training for 250 steps (current step: 12000)
2023/06/23 01:55:40 750402 |  TRACE   | Validating
2023/06/23 01:56:47 946760 |  TRACE   | Writing results to TensorBoard:
2023/06/23 01:56:47 947087 |  TRACE   |  - Start loss computation
2023/06/23 01:56:47 947330 |  TRACE   |  - Start image generation for 100 images
2023/06/23 01:57:43 554394 |  TRACE   |  - No improvement for loss: 1.02471 (current best: 1.02473)
2023/06/23 01:57:43 554744 |  TRACE   | Training for 250 steps (current step: 12250)
2023/06/23 02:06:40 077187 |  TRACE   | Validating
2023/06/23 02:07:49 389796 |  TRACE   | Writing results to TensorBoard:
2023/06/23 02:07:49 390127 |  TRACE   |  - Start loss computation
2023/06/23 02:07:49 390381 |  TRACE   |  - Start image generation for 100 images
2023/06/23 02:08:45 577979 |  TRACE   |  - No improvement for loss: 1.02502 (current best: 1.02473)
2023/06/23 02:08:45 578352 |  TRACE   | Training for 250 steps (current step: 12500)
2023/06/23 02:17:42 948485 |  TRACE   | Validating
2023/06/23 02:18:52 323232 |  TRACE   | Writing results to TensorBoard:
2023/06/23 02:18:52 323552 |  TRACE   |  - Start loss computation
2023/06/23 02:18:52 323797 |  TRACE   |  - Start image generation for 100 images
2023/06/23 02:19:48 520508 |  TRACE   |  - New best loss: 1.02459
2023/06/23 02:19:48 520971 |  TRACE   |  - Improvement over previous best: -0.00014
2023/06/23 02:19:48 563019 |  TRACE   |  - Best model updated!
 
2023/06/23 02:19:48 563240 |  TRACE   | Training for 250 steps (current step: 12750)
2023/06/23 02:28:32 461794 |  TRACE   | Validating
2023/06/23 02:29:41 451793 |  TRACE   | Writing results to TensorBoard:
2023/06/23 02:29:41 452114 |  TRACE   |  - Start loss computation
2023/06/23 02:29:41 452357 |  TRACE   |  - Start image generation for 100 images
2023/06/23 02:30:38 242190 |  TRACE   |  - No improvement for loss: 1.02492 (current best: 1.02459)
2023/06/23 02:30:38 242584 |  TRACE   | Training for 250 steps (current step: 13000)
2023/06/23 02:39:40 455522 |  TRACE   | Validating
2023/06/23 02:40:49 181065 |  TRACE   | Writing results to TensorBoard:
2023/06/23 02:40:49 182237 |  TRACE   |  - Start loss computation
2023/06/23 02:40:49 182501 |  TRACE   |  - Start image generation for 100 images
2023/06/23 02:41:46 009626 |  TRACE   |  - No improvement for loss: 1.02459 (current best: 1.02459)
2023/06/23 02:41:46 010026 |  TRACE   | Training for 250 steps (current step: 13250)
2023/06/23 02:50:35 758734 |  TRACE   | Validating
2023/06/23 02:51:47 637078 |  TRACE   | Writing results to TensorBoard:
2023/06/23 02:51:47 637813 |  TRACE   |  - Start loss computation
2023/06/23 02:51:47 638080 |  TRACE   |  - Start image generation for 100 images
2023/06/23 02:52:44 248454 |  TRACE   |  - No improvement for loss: 1.02481 (current best: 1.02459)
2023/06/23 02:52:44 248812 |  TRACE   | Training for 250 steps (current step: 13500)
2023/06/23 03:01:47 622781 |  TRACE   | Validating
2023/06/23 03:02:57 862204 |  TRACE   | Writing results to TensorBoard:
2023/06/23 03:02:57 863114 |  TRACE   |  - Start loss computation
2023/06/23 03:02:57 863377 |  TRACE   |  - Start image generation for 100 images
2023/06/23 03:03:53 904103 |  TRACE   |  - No improvement for loss: 1.02484 (current best: 1.02459)
2023/06/23 03:03:53 904404 |  TRACE   | Training for 250 steps (current step: 13750)
2023/06/23 03:12:50 458614 |  TRACE   | Validating
2023/06/23 03:14:01 020821 |  TRACE   | Writing results to TensorBoard:
2023/06/23 03:14:01 021578 |  TRACE   |  - Start loss computation
2023/06/23 03:14:01 021866 |  TRACE   |  - Start image generation for 100 images
2023/06/23 03:14:57 601388 |  TRACE   |  - No improvement for loss: 1.02457 (current best: 1.02459)
2023/06/23 03:14:57 601808 |  TRACE   | Training for 250 steps (current step: 14000)
2023/06/23 03:24:03 661369 |  TRACE   | Validating
2023/06/23 03:25:13 085899 |  TRACE   | Writing results to TensorBoard:
2023/06/23 03:25:13 086749 |  TRACE   |  - Start loss computation
2023/06/23 03:25:13 087034 |  TRACE   |  - Start image generation for 100 images
2023/06/23 03:26:09 124168 |  TRACE   |  - No improvement for loss: 1.02476 (current best: 1.02459)
2023/06/23 03:26:09 124532 |  TRACE   | Training for 250 steps (current step: 14250)
2023/06/23 03:34:57 936942 |  TRACE   | Validating
2023/06/23 03:36:07 570171 |  TRACE   | Writing results to TensorBoard:
2023/06/23 03:36:07 571172 |  TRACE   |  - Start loss computation
2023/06/23 03:36:07 571442 |  TRACE   |  - Start image generation for 100 images
2023/06/23 03:37:03 933978 |  TRACE   |  - No improvement for loss: 1.02460 (current best: 1.02459)
2023/06/23 03:37:03 934336 |  TRACE   | Training for 250 steps (current step: 14500)
2023/06/23 03:46:10 397234 |  TRACE   | Validating
2023/06/23 03:47:18 088951 |  TRACE   | Writing results to TensorBoard:
2023/06/23 03:47:18 090284 |  TRACE   |  - Start loss computation
2023/06/23 03:47:18 090551 |  TRACE   |  - Start image generation for 100 images
2023/06/23 03:48:14 031892 |  TRACE   |  - No improvement for loss: 1.02460 (current best: 1.02459)
2023/06/23 03:48:14 032237 |  TRACE   | Training for 250 steps (current step: 14750)
2023/06/23 03:57:06 563278 |  TRACE   | Validating
2023/06/23 03:58:15 555665 |  TRACE   | Writing results to TensorBoard:
2023/06/23 03:58:15 556387 |  TRACE   |  - Start loss computation
2023/06/23 03:58:15 556659 |  TRACE   |  - Start image generation for 100 images
2023/06/23 03:59:11 076123 |  TRACE   |  - No improvement for loss: 1.02524 (current best: 1.02459)
2023/06/23 03:59:11 076456 |  TRACE   | Training for 250 steps (current step: 15000)
2023/06/23 04:08:13 958317 |  TRACE   | Validating
2023/06/23 04:09:24 485655 |  TRACE   | Writing results to TensorBoard:
2023/06/23 04:09:24 486409 |  TRACE   |  - Start loss computation
2023/06/23 04:09:24 486704 |  TRACE   |  - Start image generation for 100 images
2023/06/23 04:10:20 775087 |  TRACE   |  - New best loss: 1.02447
2023/06/23 04:10:20 775500 |  TRACE   |  - Improvement over previous best: -0.00012
2023/06/23 04:10:20 848951 |  TRACE   |  - Best model updated!
 
2023/06/23 04:10:20 849198 |  TRACE   | Training for 250 steps (current step: 15250)
2023/06/23 04:19:17 466471 |  TRACE   | Validating
2023/06/23 04:20:28 162466 |  TRACE   | Writing results to TensorBoard:
2023/06/23 04:20:28 163344 |  TRACE   |  - Start loss computation
2023/06/23 04:20:28 163605 |  TRACE   |  - Start image generation for 100 images
2023/06/23 04:21:24 128389 |  TRACE   |  - No improvement for loss: 1.02459 (current best: 1.02447)
2023/06/23 04:21:24 128743 |  TRACE   | Training for 250 steps (current step: 15500)
2023/06/23 04:30:21 784103 |  TRACE   | Validating
2023/06/23 04:31:29 788561 |  TRACE   | Writing results to TensorBoard:
2023/06/23 04:31:29 789240 |  TRACE   |  - Start loss computation
2023/06/23 04:31:29 789478 |  TRACE   |  - Start image generation for 100 images
2023/06/23 04:32:26 725951 |  TRACE   |  - No improvement for loss: 1.02463 (current best: 1.02447)
2023/06/23 04:32:26 726303 |  TRACE   | Training for 250 steps (current step: 15750)
2023/06/23 04:41:25 141586 |  TRACE   | Validating
2023/06/23 04:42:35 990641 |  TRACE   | Writing results to TensorBoard:
2023/06/23 04:42:35 991828 |  TRACE   |  - Start loss computation
2023/06/23 04:42:35 992227 |  TRACE   |  - Start image generation for 100 images
2023/06/23 04:43:32 203990 |  TRACE   |  - No improvement for loss: 1.02465 (current best: 1.02447)
2023/06/23 04:43:32 204348 |  TRACE   | Training for 250 steps (current step: 16000)
2023/06/23 04:52:38 792044 |  TRACE   | Validating
2023/06/23 04:53:47 535975 |  TRACE   | Writing results to TensorBoard:
2023/06/23 04:53:47 536838 |  TRACE   |  - Start loss computation
2023/06/23 04:53:47 537097 |  TRACE   |  - Start image generation for 100 images
2023/06/23 04:54:43 949352 |  TRACE   |  - No improvement for loss: 1.02469 (current best: 1.02447)
2023/06/23 04:54:43 949677 |  TRACE   | Training for 250 steps (current step: 16250)
2023/06/23 05:03:31 810481 |  TRACE   | Validating
2023/06/23 05:04:42 216914 |  TRACE   | Writing results to TensorBoard:
2023/06/23 05:04:42 218029 |  TRACE   |  - Start loss computation
2023/06/23 05:04:42 218301 |  TRACE   |  - Start image generation for 100 images
2023/06/23 05:05:39 151138 |  TRACE   |  - No improvement for loss: 1.02454 (current best: 1.02447)
2023/06/23 05:05:39 151502 |  TRACE   | Training for 250 steps (current step: 16500)
2023/06/23 05:14:33 417445 |  TRACE   | Validating
2023/06/23 05:15:42 137272 |  TRACE   | Writing results to TensorBoard:
2023/06/23 05:15:42 137602 |  TRACE   |  - Start loss computation
2023/06/23 05:15:42 137851 |  TRACE   |  - Start image generation for 100 images
2023/06/23 05:16:38 171667 |  TRACE   |  - No improvement for loss: 1.02474 (current best: 1.02447)
2023/06/23 05:16:38 172001 |  TRACE   | Training for 250 steps (current step: 16750)
2023/06/23 05:25:28 380911 |  TRACE   | Validating
2023/06/23 05:26:38 794002 |  TRACE   | Writing results to TensorBoard:
2023/06/23 05:26:38 794312 |  TRACE   |  - Start loss computation
2023/06/23 05:26:38 794567 |  TRACE   |  - Start image generation for 100 images
2023/06/23 05:27:35 132723 |  TRACE   |  - No improvement for loss: 1.02446 (current best: 1.02447)
2023/06/23 05:27:35 133118 |  TRACE   | Training for 250 steps (current step: 17000)
2023/06/23 05:36:36 551704 |  TRACE   | Validating
2023/06/23 05:37:47 372790 |  TRACE   | Writing results to TensorBoard:
2023/06/23 05:37:47 373114 |  TRACE   |  - Start loss computation
2023/06/23 05:37:47 373368 |  TRACE   |  - Start image generation for 100 images
2023/06/23 05:38:43 865592 |  TRACE   |  - No improvement for loss: 1.02458 (current best: 1.02447)
2023/06/23 05:38:43 865944 |  TRACE   | Training for 250 steps (current step: 17250)
2023/06/23 05:47:33 039205 |  TRACE   | Validating
2023/06/23 05:48:41 046040 |  TRACE   | Writing results to TensorBoard:
2023/06/23 05:48:41 046421 |  TRACE   |  - Start loss computation
2023/06/23 05:48:41 046668 |  TRACE   |  - Start image generation for 100 images
2023/06/23 05:49:38 057638 |  TRACE   |  - No improvement for loss: 1.02453 (current best: 1.02447)
2023/06/23 05:49:38 057970 |  TRACE   | Training for 250 steps (current step: 17500)
2023/06/23 05:58:43 021790 |  TRACE   | Validating
2023/06/23 05:59:56 710719 |  TRACE   | Writing results to TensorBoard:
2023/06/23 05:59:56 711092 |  TRACE   |  - Start loss computation
2023/06/23 05:59:56 711349 |  TRACE   |  - Start image generation for 100 images
2023/06/23 06:00:52 958973 |  TRACE   |  - No improvement for loss: 1.02453 (current best: 1.02447)
2023/06/23 06:00:52 959342 |  TRACE   |  - Early Stopping tolerance reached. Stopping training!

2023/06/29 17:24:38 594497 |  TRACE   | Evaluation results:
2023/06/29 17:24:38 594866 |  TRACE   |  - Results for field size: global
2023/06/29 17:24:38 595128 |  TRACE   |    - Precision: 60.66%
2023/06/29 17:24:38 595385 |  TRACE   |    - Recall: 51.01%
2023/06/29 17:24:38 595520 |  TRACE   |  - Results for field size: small
2023/06/29 17:24:38 595672 |  TRACE   |    - Precision: 42.84%
2023/06/29 17:24:38 595847 |  TRACE   |    - Recall: 18.76%
2023/06/29 17:24:38 595973 |  TRACE   |  - Results for field size: medium
2023/06/29 17:24:38 596131 |  TRACE   |    - Precision: 55.85%
2023/06/29 17:24:38 596283 |  TRACE   |    - Recall: 55.10%
2023/06/29 17:24:38 596413 |  TRACE   |  - Results for field size: large
2023/06/29 17:24:38 596564 |  TRACE   |    - Precision: 69.37%
2023/06/29 17:24:38 596732 |  TRACE   |    - Recall: 67.19%
2023/06/29 17:24:38 596857 |  TRACE   |  - Results for field size: extra large
2023/06/29 17:24:38 597002 |  TRACE   |    - Precision: 70.35%
2023/06/29 17:24:38 597155 |  TRACE   |    - Recall: 73.54%

2023/07/03 08:26:46 720649 |  TRACE   | Start model evaluation
2023/07/03 15:59:48 318801 |  TRACE   | Evaluation results:
2023/07/03 15:59:48 319213 |  TRACE   |  - Results for field size: global
2023/07/03 15:59:48 319535 |  TRACE   |    - Precision: 60.69%
2023/07/03 15:59:48 319808 |  TRACE   |    - Recall: 50.97%
2023/07/03 15:59:48 319946 |  TRACE   |  - Results for field size: small
2023/07/03 15:59:48 320072 |  TRACE   |    - Precision: 43.01%
2023/07/03 15:59:48 320201 |  TRACE   |    - Recall: 18.65%
2023/07/03 15:59:48 320295 |  TRACE   |  - Results for field size: medium
2023/07/03 15:59:48 320418 |  TRACE   |    - Precision: 55.82%
2023/07/03 15:59:48 320555 |  TRACE   |    - Recall: 55.08%
2023/07/03 15:59:48 320650 |  TRACE   |  - Results for field size: large
2023/07/03 15:59:48 320769 |  TRACE   |    - Precision: 69.36%
2023/07/03 15:59:48 320896 |  TRACE   |    - Recall: 67.18%
2023/07/03 15:59:48 320985 |  TRACE   |  - Results for field size: extra large
2023/07/03 15:59:48 321104 |  TRACE   |    - Precision: 70.33%
2023/07/03 15:59:48 321233 |  TRACE   |    - Recall: 73.53%
