2023/05/24 14:52:01 319112 |  TRACE   |  Model configs:
2023/05/24 14:52:01 319330 |  TRACE   |  - Model: 20230524T145156-EnanchedResUnet3D_ts16
2023/05/24 14:52:01 319443 |  TRACE   |  - Parameters: 1_491_425
2023/05/24 14:52:01 319541 |  TRACE   |  - Device: cuda
2023/05/24 14:52:01 319655 |  TRACE   |  - Input bands: ['s2_ndvi', 's2_b02', 's2_b03', 's2_b04']
2023/05/24 14:52:01 319763 |  TRACE   |  - N ts: 16
 
2023/05/24 14:52:01 319884 |  TRACE   |  Steps configs:
2023/05/24 14:52:01 319994 |  TRACE   |  - Maximum steps: None
2023/05/24 14:52:01 320100 |  TRACE   |  - Validation steps: 250
2023/05/24 14:52:01 320198 |  TRACE   |  - Batch-size: 16
 
2023/05/24 14:52:01 320317 |  TRACE   |  Optimizer configs:
2023/05/24 14:52:01 320435 |  TRACE   |  - Optimizer: Adam
2023/05/24 14:52:01 320536 |  TRACE   |  - Early-stopping tolerance: 10
2023/05/24 14:52:01 320649 |  TRACE   |  - Early-stopping improvement: 0.0001
2023/05/24 14:52:01 320751 |  TRACE   |  - Early-stopping metric: loss
2023/05/24 14:52:01 320852 |  TRACE   |  - Scheduler: ReduceLROnPlateau
2023/05/24 14:52:01 320952 |  TRACE   |  - Learninr rate tolerance: 3
 
2023/05/24 14:52:01 321092 |  TRACE   |  Loss configs:
2023/05/24 14:52:01 321218 |  TRACE   |  - Tag: DiceLoss_ws
2023/05/24 14:52:01 321339 |  TRACE   |  - Loss: dice_loss
2023/05/24 14:52:01 321471 |  TRACE   |  - Weight / multiplier: {'watersheds': 10}
2023/05/24 14:52:01 321595 |  TRACE   |  - LC coefficient: 1
2023/05/24 14:52:01 321736 |  TRACE   | 

2023/05/24 14:52:14 397986 |  TRACE   | Writing results to TensorBoard:
2023/05/24 14:52:14 398373 |  TRACE   |  - Start loss computation
2023/05/24 14:52:14 398721 |  TRACE   |  - Start image generation for 100 images
2023/05/24 14:52:18 528488 |  TRACE   | 

2023/05/24 14:52:18 528830 |  TRACE   | Training for 250 steps (current step: 0)
2023/05/24 14:57:21 091013 |  TRACE   | Validating
2023/05/24 14:57:47 055595 |  TRACE   | Writing results to TensorBoard:
2023/05/24 14:57:47 055964 |  TRACE   |  - Start loss computation
2023/05/24 14:57:47 056306 |  TRACE   |  - Start image generation for 100 images
2023/05/24 14:58:29 247123 |  TRACE   |  - New best loss: 0.39934
2023/05/24 14:58:29 247532 |  TRACE   |  - Improvement over previous best: -0.32513
2023/05/24 14:58:29 287834 |  TRACE   |  - Best model updated!
 
2023/05/24 14:58:29 288076 |  TRACE   | Training for 250 steps (current step: 250)
2023/05/24 15:03:35 022705 |  TRACE   | Validating
2023/05/24 15:04:05 939620 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:04:05 940175 |  TRACE   |  - Start loss computation
2023/05/24 15:04:05 940460 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:05:02 439790 |  TRACE   |  - New best loss: 0.32629
2023/05/24 15:05:02 440198 |  TRACE   |  - Improvement over previous best: -0.07305
2023/05/24 15:05:02 481518 |  TRACE   |  - Best model updated!
 
2023/05/24 15:05:02 481782 |  TRACE   | Training for 250 steps (current step: 500)
2023/05/24 15:10:10 543020 |  TRACE   | Validating
2023/05/24 15:10:48 767115 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:10:48 767473 |  TRACE   |  - Start loss computation
2023/05/24 15:10:48 767710 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:11:42 967213 |  TRACE   |  - New best loss: 0.32117
2023/05/24 15:11:42 967622 |  TRACE   |  - Improvement over previous best: -0.00512
2023/05/24 15:11:43 004279 |  TRACE   |  - Best model updated!
 
2023/05/24 15:11:43 004500 |  TRACE   | Training for 250 steps (current step: 750)
2023/05/24 15:16:38 737692 |  TRACE   | Validating
2023/05/24 15:17:17 654779 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:17:17 655125 |  TRACE   |  - Start loss computation
2023/05/24 15:17:17 655390 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:18:13 612444 |  TRACE   |  - New best loss: 0.30062
2023/05/24 15:18:13 612850 |  TRACE   |  - Improvement over previous best: -0.02055
2023/05/24 15:18:13 650181 |  TRACE   |  - Best model updated!
 
2023/05/24 15:18:13 650415 |  TRACE   | Training for 250 steps (current step: 1000)
2023/05/24 15:23:25 519782 |  TRACE   | Validating
2023/05/24 15:24:03 856479 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:24:03 856844 |  TRACE   |  - Start loss computation
2023/05/24 15:24:03 857126 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:24:59 919632 |  TRACE   |  - New best loss: 0.29347
2023/05/24 15:24:59 920090 |  TRACE   |  - Improvement over previous best: -0.00715
2023/05/24 15:24:59 955028 |  TRACE   |  - Best model updated!
 
2023/05/24 15:24:59 955245 |  TRACE   | Training for 250 steps (current step: 1250)
2023/05/24 15:30:03 330357 |  TRACE   | Validating
2023/05/24 15:30:42 535576 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:30:42 535909 |  TRACE   |  - Start loss computation
2023/05/24 15:30:42 536122 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:31:38 363164 |  TRACE   |  - No improvement for loss: 0.29482 (current best: 0.29347)
2023/05/24 15:31:38 363554 |  TRACE   | Training for 250 steps (current step: 1500)
2023/05/24 15:36:50 061452 |  TRACE   | Validating
2023/05/24 15:37:29 059972 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:37:29 060314 |  TRACE   |  - Start loss computation
2023/05/24 15:37:29 060545 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:38:26 806403 |  TRACE   |  - New best loss: 0.29212
2023/05/24 15:38:26 806821 |  TRACE   |  - Improvement over previous best: -0.00135
2023/05/24 15:38:26 842921 |  TRACE   |  - Best model updated!
 
2023/05/24 15:38:26 843139 |  TRACE   | Training for 250 steps (current step: 1750)
2023/05/24 15:43:24 447189 |  TRACE   | Validating
2023/05/24 15:44:03 834307 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:44:03 834696 |  TRACE   |  - Start loss computation
2023/05/24 15:44:03 834995 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:44:59 183691 |  TRACE   |  - New best loss: 0.28874
2023/05/24 15:44:59 184150 |  TRACE   |  - Improvement over previous best: -0.00338
2023/05/24 15:44:59 219756 |  TRACE   |  - Best model updated!
 
2023/05/24 15:44:59 219985 |  TRACE   | Training for 250 steps (current step: 2000)
2023/05/24 15:50:06 840503 |  TRACE   | Validating
2023/05/24 15:50:46 054125 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:50:46 054449 |  TRACE   |  - Start loss computation
2023/05/24 15:50:46 054682 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:51:41 402327 |  TRACE   |  - No improvement for loss: 0.29116 (current best: 0.28874)
2023/05/24 15:51:41 402701 |  TRACE   | Training for 250 steps (current step: 2250)
2023/05/24 15:56:41 923634 |  TRACE   | Validating
2023/05/24 15:57:20 890820 |  TRACE   | Writing results to TensorBoard:
2023/05/24 15:57:20 891165 |  TRACE   |  - Start loss computation
2023/05/24 15:57:20 891471 |  TRACE   |  - Start image generation for 100 images
2023/05/24 15:58:16 274826 |  TRACE   |  - New best loss: 0.28498
2023/05/24 15:58:16 275266 |  TRACE   |  - Improvement over previous best: -0.00376
2023/05/24 15:58:16 313617 |  TRACE   |  - Best model updated!
 
2023/05/24 15:58:16 313856 |  TRACE   | Training for 250 steps (current step: 2500)
2023/05/24 16:03:37 922108 |  TRACE   | Validating
2023/05/24 16:04:16 496065 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:04:16 496431 |  TRACE   |  - Start loss computation
2023/05/24 16:04:16 496741 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:05:17 272326 |  TRACE   |  - No improvement for loss: 0.29128 (current best: 0.28498)
2023/05/24 16:05:17 272641 |  TRACE   | Training for 250 steps (current step: 2750)
2023/05/24 16:10:17 390542 |  TRACE   | Validating
2023/05/24 16:10:56 449475 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:10:56 449928 |  TRACE   |  - Start loss computation
2023/05/24 16:10:56 450177 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:11:53 894257 |  TRACE   |  - New best loss: 0.28163
2023/05/24 16:11:53 894641 |  TRACE   |  - Improvement over previous best: -0.00335
2023/05/24 16:11:53 929351 |  TRACE   |  - Best model updated!
 
2023/05/24 16:11:53 929543 |  TRACE   | Training for 250 steps (current step: 3000)
2023/05/24 16:17:05 427657 |  TRACE   | Validating
2023/05/24 16:17:44 427198 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:17:44 427586 |  TRACE   |  - Start loss computation
2023/05/24 16:17:44 427830 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:18:43 695317 |  TRACE   |  - No improvement for loss: 0.29100 (current best: 0.28163)
2023/05/24 16:18:43 695647 |  TRACE   | Training for 250 steps (current step: 3250)
2023/05/24 16:23:43 474506 |  TRACE   | Validating
2023/05/24 16:24:22 672513 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:24:22 672822 |  TRACE   |  - Start loss computation
2023/05/24 16:24:22 673066 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:25:18 622656 |  TRACE   |  - No improvement for loss: 0.28345 (current best: 0.28163)
2023/05/24 16:25:18 623014 |  TRACE   | Training for 250 steps (current step: 3500)
2023/05/24 16:30:27 684082 |  TRACE   | Validating
2023/05/24 16:31:06 704726 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:31:06 705094 |  TRACE   |  - Start loss computation
2023/05/24 16:31:06 705356 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:32:02 971114 |  TRACE   |  - New best loss: 0.27664
2023/05/24 16:32:02 971485 |  TRACE   |  - Improvement over previous best: -0.00499
2023/05/24 16:32:03 035692 |  TRACE   |  - Best model updated!
 
2023/05/24 16:32:03 035886 |  TRACE   | Training for 250 steps (current step: 3750)
2023/05/24 16:37:02 697670 |  TRACE   | Validating
2023/05/24 16:37:41 361508 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:37:41 361877 |  TRACE   |  - Start loss computation
2023/05/24 16:37:41 362139 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:38:44 570129 |  TRACE   |  - No improvement for loss: 0.27997 (current best: 0.27664)
2023/05/24 16:38:44 570456 |  TRACE   | Training for 250 steps (current step: 4000)
2023/05/24 16:43:52 649897 |  TRACE   | Validating
2023/05/24 16:44:32 090000 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:44:32 090336 |  TRACE   |  - Start loss computation
2023/05/24 16:44:32 090620 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:45:29 046470 |  TRACE   |  - New best loss: 0.27294
2023/05/24 16:45:29 046857 |  TRACE   |  - Improvement over previous best: -0.00370
2023/05/24 16:45:29 085084 |  TRACE   |  - Best model updated!
 
2023/05/24 16:45:29 085277 |  TRACE   | Training for 250 steps (current step: 4250)
2023/05/24 16:50:24 890682 |  TRACE   | Validating
2023/05/24 16:51:03 907397 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:51:03 907652 |  TRACE   |  - Start loss computation
2023/05/24 16:51:03 908048 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:52:00 850165 |  TRACE   |  - No improvement for loss: 0.27749 (current best: 0.27294)
2023/05/24 16:52:00 850514 |  TRACE   | Training for 250 steps (current step: 4500)
2023/05/24 16:57:00 907187 |  TRACE   | Validating
2023/05/24 16:57:39 909884 |  TRACE   | Writing results to TensorBoard:
2023/05/24 16:57:39 910263 |  TRACE   |  - Start loss computation
2023/05/24 16:57:39 910524 |  TRACE   |  - Start image generation for 100 images
2023/05/24 16:58:40 395660 |  TRACE   |  - New best loss: 0.27088
2023/05/24 16:58:40 396040 |  TRACE   |  - Improvement over previous best: -0.00207
2023/05/24 16:58:40 434335 |  TRACE   |  - Best model updated!
 
2023/05/24 16:58:40 434520 |  TRACE   | Training for 250 steps (current step: 4750)
2023/05/24 17:03:40 381656 |  TRACE   | Validating
2023/05/24 17:04:19 789685 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:04:19 790037 |  TRACE   |  - Start loss computation
2023/05/24 17:04:19 790329 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:05:15 609815 |  TRACE   |  - No improvement for loss: 0.27449 (current best: 0.27088)
2023/05/24 17:05:15 610162 |  TRACE   | Training for 250 steps (current step: 5000)
2023/05/24 17:10:19 874534 |  TRACE   | Validating
2023/05/24 17:10:58 891280 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:10:58 891699 |  TRACE   |  - Start loss computation
2023/05/24 17:10:58 891965 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:11:56 152718 |  TRACE   |  - New best loss: 0.26934
2023/05/24 17:11:56 153166 |  TRACE   |  - Improvement over previous best: -0.00154
2023/05/24 17:11:56 187383 |  TRACE   |  - Best model updated!
 
2023/05/24 17:11:56 187624 |  TRACE   | Training for 250 steps (current step: 5250)
2023/05/24 17:17:01 993658 |  TRACE   | Validating
2023/05/24 17:17:40 571968 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:17:40 572298 |  TRACE   |  - Start loss computation
2023/05/24 17:17:40 572600 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:18:39 702902 |  TRACE   |  - New best loss: 0.26728
2023/05/24 17:18:39 703299 |  TRACE   |  - Improvement over previous best: -0.00206
2023/05/24 17:18:39 740781 |  TRACE   |  - Best model updated!
 
2023/05/24 17:18:39 741040 |  TRACE   | Training for 250 steps (current step: 5500)
2023/05/24 17:23:49 290477 |  TRACE   | Validating
2023/05/24 17:24:27 820572 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:24:27 820922 |  TRACE   |  - Start loss computation
2023/05/24 17:24:27 821310 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:25:24 214788 |  TRACE   |  - No improvement for loss: 0.27441 (current best: 0.26728)
2023/05/24 17:25:24 215128 |  TRACE   | Training for 250 steps (current step: 5750)
2023/05/24 17:30:20 864830 |  TRACE   | Validating
2023/05/24 17:31:00 244136 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:31:00 244475 |  TRACE   |  - Start loss computation
2023/05/24 17:31:00 244749 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:32:00 852313 |  TRACE   |  - New best loss: 0.26567
2023/05/24 17:32:00 852691 |  TRACE   |  - Improvement over previous best: -0.00161
2023/05/24 17:32:00 888173 |  TRACE   |  - Best model updated!
 
2023/05/24 17:32:00 888397 |  TRACE   | Training for 250 steps (current step: 6000)
2023/05/24 17:37:09 547405 |  TRACE   | Validating
2023/05/24 17:37:48 556963 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:37:48 557339 |  TRACE   |  - Start loss computation
2023/05/24 17:37:48 557699 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:38:48 092394 |  TRACE   |  - New best loss: 0.26198
2023/05/24 17:38:48 092813 |  TRACE   |  - Improvement over previous best: -0.00370
2023/05/24 17:38:48 128463 |  TRACE   |  - Best model updated!
 
2023/05/24 17:38:48 128698 |  TRACE   | Training for 250 steps (current step: 6250)
2023/05/24 17:43:46 450330 |  TRACE   | Validating
2023/05/24 17:44:25 803686 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:44:25 804021 |  TRACE   |  - Start loss computation
2023/05/24 17:44:25 804253 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:45:23 606789 |  TRACE   |  - New best loss: 0.25846
2023/05/24 17:45:23 607195 |  TRACE   |  - Improvement over previous best: -0.00352
2023/05/24 17:45:23 644088 |  TRACE   |  - Best model updated!
 
2023/05/24 17:45:23 644319 |  TRACE   | Training for 250 steps (current step: 6500)
2023/05/24 17:50:38 253934 |  TRACE   | Validating
2023/05/24 17:51:16 633066 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:51:16 633522 |  TRACE   |  - Start loss computation
2023/05/24 17:51:16 633814 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:52:15 051715 |  TRACE   |  - No improvement for loss: 0.26210 (current best: 0.25846)
2023/05/24 17:52:15 052085 |  TRACE   | Training for 250 steps (current step: 6750)
2023/05/24 17:57:19 956859 |  TRACE   | Validating
2023/05/24 17:57:58 899284 |  TRACE   | Writing results to TensorBoard:
2023/05/24 17:57:58 899621 |  TRACE   |  - Start loss computation
2023/05/24 17:57:58 899897 |  TRACE   |  - Start image generation for 100 images
2023/05/24 17:58:57 049548 |  TRACE   |  - No improvement for loss: 0.27265 (current best: 0.25846)
2023/05/24 17:58:57 049914 |  TRACE   | Training for 250 steps (current step: 7000)
2023/05/24 18:04:01 056869 |  TRACE   | Validating
2023/05/24 18:04:39 640134 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:04:39 640478 |  TRACE   |  - Start loss computation
2023/05/24 18:04:39 640777 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:05:36 482746 |  TRACE   |  - No improvement for loss: 0.27303 (current best: 0.25846)
2023/05/24 18:05:36 483078 |  TRACE   | Training for 250 steps (current step: 7250)
2023/05/24 18:10:34 811588 |  TRACE   | Validating
2023/05/24 18:11:14 195772 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:11:14 196137 |  TRACE   |  - Start loss computation
2023/05/24 18:11:14 196496 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:12:12 668288 |  TRACE   |  - No improvement for loss: 0.26554 (current best: 0.25846)
2023/05/24 18:12:12 668638 |  TRACE   | Training for 250 steps (current step: 7500)
2023/05/24 18:17:24 044530 |  TRACE   | Validating
2023/05/24 18:18:02 944772 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:18:02 945119 |  TRACE   |  - Start loss computation
2023/05/24 18:18:02 945418 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:19:03 435427 |  TRACE   |  - No improvement for loss: 0.25989 (current best: 0.25846)
2023/05/24 18:19:03 435804 |  TRACE   | Training for 250 steps (current step: 7750)
2023/05/24 18:24:11 373891 |  TRACE   | Validating
2023/05/24 18:24:50 371213 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:24:50 371577 |  TRACE   |  - Start loss computation
2023/05/24 18:24:50 371889 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:25:50 474704 |  TRACE   |  - No improvement for loss: 0.25880 (current best: 0.25846)
2023/05/24 18:25:50 475086 |  TRACE   | Training for 250 steps (current step: 8000)
2023/05/24 18:31:07 601451 |  TRACE   | Validating
2023/05/24 18:31:46 191121 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:31:46 191448 |  TRACE   |  - Start loss computation
2023/05/24 18:31:46 191678 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:32:46 330724 |  TRACE   |  - No improvement for loss: 0.25864 (current best: 0.25846)
2023/05/24 18:32:46 331102 |  TRACE   | Training for 250 steps (current step: 8250)
2023/05/24 18:37:49 592063 |  TRACE   | Validating
2023/05/24 18:38:28 650767 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:38:28 651119 |  TRACE   |  - Start loss computation
2023/05/24 18:38:28 651356 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:39:28 236604 |  TRACE   |  - New best loss: 0.25690
2023/05/24 18:39:28 237041 |  TRACE   |  - Improvement over previous best: -0.00156
2023/05/24 18:39:28 297266 |  TRACE   |  - Best model updated!
 
2023/05/24 18:39:28 297508 |  TRACE   | Training for 250 steps (current step: 8500)
2023/05/24 18:44:38 229422 |  TRACE   | Validating
2023/05/24 18:45:17 230609 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:45:17 230975 |  TRACE   |  - Start loss computation
2023/05/24 18:45:17 231344 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:46:17 805988 |  TRACE   |  - No improvement for loss: 0.25969 (current best: 0.25690)
2023/05/24 18:46:17 806384 |  TRACE   | Training for 250 steps (current step: 8750)
2023/05/24 18:51:16 485834 |  TRACE   | Validating
2023/05/24 18:51:55 950258 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:51:55 950637 |  TRACE   |  - Start loss computation
2023/05/24 18:51:55 950899 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:52:56 674197 |  TRACE   |  - No improvement for loss: 0.25766 (current best: 0.25690)
2023/05/24 18:52:56 674574 |  TRACE   | Training for 250 steps (current step: 9000)
2023/05/24 18:58:09 285443 |  TRACE   | Validating
2023/05/24 18:58:48 287680 |  TRACE   | Writing results to TensorBoard:
2023/05/24 18:58:48 288051 |  TRACE   |  - Start loss computation
2023/05/24 18:58:48 288429 |  TRACE   |  - Start image generation for 100 images
2023/05/24 18:59:49 092308 |  TRACE   |  - New best loss: 0.25584
2023/05/24 18:59:49 092745 |  TRACE   |  - Improvement over previous best: -0.00106
2023/05/24 18:59:49 131786 |  TRACE   |  - Best model updated!
 
2023/05/24 18:59:49 132021 |  TRACE   | Training for 250 steps (current step: 9250)
2023/05/24 19:04:48 003290 |  TRACE   | Validating
2023/05/24 19:05:27 002196 |  TRACE   | Writing results to TensorBoard:
2023/05/24 19:05:27 002564 |  TRACE   |  - Start loss computation
2023/05/24 19:05:27 002901 |  TRACE   |  - Start image generation for 100 images
2023/05/24 19:06:26 134284 |  TRACE   |  - No improvement for loss: 0.25587 (current best: 0.25584)
2023/05/24 19:06:26 134648 |  TRACE   | Training for 250 steps (current step: 9500)
2023/05/24 19:11:45 837196 |  TRACE   | Validating
2023/05/24 19:12:24 088518 |  TRACE   | Writing results to TensorBoard:
2023/05/24 19:12:24 088873 |  TRACE   |  - Start loss computation
2023/05/24 19:12:24 089213 |  TRACE   |  - Start image generation for 100 images
2023/05/24 19:13:23 256333 |  TRACE   |  - No improvement for loss: 0.25866 (current best: 0.25584)
2023/05/24 19:13:23 256683 |  TRACE   | Training for 250 steps (current step: 9750)
2023/05/24 19:18:22 987059 |  TRACE   | Validating
2023/05/24 19:19:02 360735 |  TRACE   | Writing results to TensorBoard:
2023/05/24 19:19:02 361142 |  TRACE   |  - Start loss computation
2023/05/24 19:19:02 361483 |  TRACE   |  - Start image generation for 100 images
2023/05/24 19:20:02 144009 |  TRACE   |  - No improvement for loss: 0.25921 (current best: 0.25584)
2023/05/24 19:20:02 144394 |  TRACE   | Training for 250 steps (current step: 10000)
2023/05/24 19:25:11 950804 |  TRACE   | Validating
2023/05/24 19:25:50 946203 |  TRACE   | Writing results to TensorBoard:
2023/05/24 19:25:50 946578 |  TRACE   |  - Start loss computation
2023/05/24 19:25:50 946871 |  TRACE   |  - Start image generation for 100 images
2023/05/24 19:26:50 733053 |  TRACE   |  - No improvement for loss: 0.25845 (current best: 0.25584)
2023/05/24 19:26:50 733398 |  TRACE   | Training for 250 steps (current step: 10250)
2023/05/24 19:31:54 114818 |  TRACE   | Validating
2023/05/24 19:32:33 365597 |  TRACE   | Writing results to TensorBoard:
2023/05/24 19:32:33 365938 |  TRACE   |  - Start loss computation
2023/05/24 19:32:33 366212 |  TRACE   |  - Start image generation for 100 images
2023/05/24 19:33:33 665084 |  TRACE   |  - No improvement for loss: 0.25752 (current best: 0.25584)
2023/05/24 19:33:33 665453 |  TRACE   | Training for 250 steps (current step: 10500)
2023/05/24 19:38:52 755135 |  TRACE   | Validating
2023/05/24 19:39:31 761612 |  TRACE   | Writing results to TensorBoard:
2023/05/24 19:39:31 761984 |  TRACE   |  - Start loss computation
2023/05/24 19:39:31 762381 |  TRACE   |  - Start image generation for 100 images
2023/05/24 19:40:31 664513 |  TRACE   |  - No improvement for loss: 0.25649 (current best: 0.25584)
2023/05/24 19:40:31 664895 |  TRACE   | Training for 250 steps (current step: 10750)
2023/05/26 08:50:52 972231 |  TRACE   | Start model evaluation
2023/05/26 08:57:43 760787 |  TRACE   | Avg prediction time per sample:  0.0183s
2023/05/26 08:57:43 835446 |  TRACE   | Evaluation results:
2023/05/26 08:57:43 835808 |  TRACE   |  - Results for field size: global
2023/05/26 08:57:43 836066 |  TRACE   |    - Precision: 59.85%
2023/05/26 08:57:43 836239 |  TRACE   |    - Recall: 49.39%
2023/05/26 08:57:43 836349 |  TRACE   |  - Results for field size: small
2023/05/26 08:57:43 836505 |  TRACE   |    - Precision: 43.79%
2023/05/26 08:57:43 836668 |  TRACE   |    - Recall: 16.68%
2023/05/26 08:57:43 836768 |  TRACE   |  - Results for field size: medium
2023/05/26 08:57:43 836939 |  TRACE   |    - Precision: 57.20%
2023/05/26 08:57:43 837116 |  TRACE   |    - Recall: 53.83%
2023/05/26 08:57:43 837260 |  TRACE   |  - Results for field size: large
2023/05/26 08:57:43 837439 |  TRACE   |    - Precision: 67.24%
2023/05/26 08:57:43 837640 |  TRACE   |    - Recall: 65.16%
2023/05/26 08:57:43 837772 |  TRACE   |  - Results for field size: extra large
2023/05/26 08:57:43 837975 |  TRACE   |    - Precision: 63.68%
2023/05/26 08:57:43 838155 |  TRACE   |    - Recall: 69.82%
