2023/02/26 22:37:54 054324 |  TRACE   |  Model configs:
2023/02/26 22:37:54 054824 |  TRACE   |  - Model: 20230226T223753-baseline+ts36+rgb+interp
2023/02/26 22:37:54 054994 |  TRACE   |  - Parameters: 6_292_077
2023/02/26 22:37:54 055137 |  TRACE   |  - Device: cuda
2023/02/26 22:37:54 055281 |  TRACE   |  - Input bands: ['s2_ndvi', 's2_b02', 's2_b03', 's2_b04']
2023/02/26 22:37:54 055421 |  TRACE   |  - N ts: 36
 
2023/02/26 22:37:54 055555 |  TRACE   |  Steps configs:
2023/02/26 22:37:54 055688 |  TRACE   |  - Maximum steps: None
2023/02/26 22:37:54 055820 |  TRACE   |  - Validation steps: 250
2023/02/26 22:37:54 055950 |  TRACE   |  - Batch-size: 32
 
2023/02/26 22:37:54 056079 |  TRACE   |  Optimizer configs:
2023/02/26 22:37:54 056207 |  TRACE   |  - Optimizer: Adam
2023/02/26 22:37:54 056337 |  TRACE   |  - Early-stopping tolerance: 10
2023/02/26 22:37:54 056474 |  TRACE   |  - Early-stopping improvement: 0.0001
2023/02/26 22:37:54 056610 |  TRACE   |  - Early-stopping metric: loss
2023/02/26 22:37:54 056739 |  TRACE   |  - Scheduler: ReduceLROnPlateau
2023/02/26 22:37:54 056870 |  TRACE   |  - Learninr rate tolerance: 3
 
2023/02/26 22:37:54 056997 |  TRACE   |  Loss configs:
2023/02/26 22:37:54 057129 |  TRACE   |  - Tag: DiceLoss
2023/02/26 22:37:54 057266 |  TRACE   |  - Loss: dice_loss
2023/02/26 22:37:54 057396 |  TRACE   |  - Weight / multiplier: None
2023/02/26 22:37:54 057526 |  TRACE   |  - LC coefficient: 0
2023/02/26 22:37:54 057654 |  TRACE   |  ---
2023/02/26 22:37:54 057783 |  TRACE   |  - Tag: DiceLoss_ws
2023/02/26 22:37:54 057911 |  TRACE   |  - Loss: dice_loss
2023/02/26 22:37:54 058038 |  TRACE   |  - Weight / multiplier: {'watersheds': 10}
2023/02/26 22:37:54 058172 |  TRACE   |  - LC coefficient: 1
2023/02/26 22:37:54 058299 |  TRACE   |  ---
2023/02/26 22:37:54 058428 |  TRACE   |  - Tag: DiceLoss_d
2023/02/26 22:37:54 058557 |  TRACE   |  - Loss: dice_loss
2023/02/26 22:37:54 058683 |  TRACE   |  - Weight / multiplier: {'distance': 1}
2023/02/26 22:37:54 058837 |  TRACE   |  - LC coefficient: 0
2023/02/26 22:37:54 058969 |  TRACE   | 

2023/02/26 22:38:23 046725 |  TRACE   | Writing results to TensorBoard:
2023/02/26 22:38:23 047403 |  TRACE   |  - Start loss computation
2023/02/26 22:38:23 047905 |  TRACE   |  - Start image generation for 100 images
2023/02/26 22:38:28 701529 |  TRACE   | 

2023/02/26 22:38:28 702154 |  TRACE   | Training for 250 steps (current step: 0)
2023/02/26 22:50:45 485666 |  TRACE   | Validating
2023/02/26 22:51:14 488175 |  TRACE   | Writing results to TensorBoard:
2023/02/26 22:51:14 488876 |  TRACE   |  - Start loss computation
2023/02/26 22:51:14 489300 |  TRACE   |  - Start image generation for 100 images
2023/02/26 22:51:19 099394 |  TRACE   |  - No improvement for loss: 0.99362 (current best: 0.64824)
2023/02/26 22:51:19 100038 |  TRACE   | Training for 250 steps (current step: 250)
2023/02/26 23:03:36 169538 |  TRACE   | Validating
2023/02/26 23:04:05 102440 |  TRACE   | Writing results to TensorBoard:
2023/02/26 23:04:05 103167 |  TRACE   |  - Start loss computation
2023/02/26 23:04:05 103619 |  TRACE   |  - Start image generation for 100 images
2023/02/26 23:05:14 531304 |  TRACE   |  - New best loss: 0.30451
2023/02/26 23:05:14 532120 |  TRACE   |  - Improvement over previous best: -0.34373
2023/02/26 23:05:14 675876 |  TRACE   |  - Best model updated!
 
2023/02/26 23:05:14 676445 |  TRACE   | Training for 250 steps (current step: 500)
2023/02/26 23:17:44 048397 |  TRACE   | Validating
2023/02/26 23:18:13 397151 |  TRACE   | Writing results to TensorBoard:
2023/02/26 23:18:13 397786 |  TRACE   |  - Start loss computation
2023/02/26 23:18:13 398257 |  TRACE   |  - Start image generation for 100 images
2023/02/26 23:19:31 317618 |  TRACE   |  - No improvement for loss: 0.32716 (current best: 0.30451)
2023/02/26 23:19:31 318235 |  TRACE   | Training for 250 steps (current step: 750)
2023/02/26 23:31:45 783856 |  TRACE   | Validating
2023/02/26 23:32:13 475209 |  TRACE   | Writing results to TensorBoard:
2023/02/26 23:32:13 475921 |  TRACE   |  - Start loss computation
2023/02/26 23:32:13 476424 |  TRACE   |  - Start image generation for 100 images
2023/02/26 23:33:26 555374 |  TRACE   |  - No improvement for loss: 0.32787 (current best: 0.30451)
2023/02/26 23:33:26 556048 |  TRACE   | Training for 250 steps (current step: 1000)
2023/02/26 23:45:59 773078 |  TRACE   | Validating
2023/02/26 23:46:28 435762 |  TRACE   | Writing results to TensorBoard:
2023/02/26 23:46:28 436420 |  TRACE   |  - Start loss computation
2023/02/26 23:46:28 436862 |  TRACE   |  - Start image generation for 100 images
2023/02/26 23:47:39 018767 |  TRACE   |  - New best loss: 0.30186
2023/02/26 23:47:39 019466 |  TRACE   |  - Improvement over previous best: -0.00264
2023/02/26 23:47:39 156253 |  TRACE   |  - Best model updated!
 
2023/02/26 23:47:39 156815 |  TRACE   | Training for 250 steps (current step: 1250)
2023/02/26 23:59:47 364054 |  TRACE   | Validating
2023/02/27 00:00:14 713017 |  TRACE   | Writing results to TensorBoard:
2023/02/27 00:00:14 713666 |  TRACE   |  - Start loss computation
2023/02/27 00:00:14 714237 |  TRACE   |  - Start image generation for 100 images
2023/02/27 00:01:30 530234 |  TRACE   |  - New best loss: 0.26876
2023/02/27 00:01:30 530972 |  TRACE   |  - Improvement over previous best: -0.03311
2023/02/27 00:01:30 696536 |  TRACE   |  - Best model updated!
 
2023/02/27 00:01:30 697086 |  TRACE   | Training for 250 steps (current step: 1500)
2023/02/27 00:14:01 329473 |  TRACE   | Validating
2023/02/27 00:14:29 786380 |  TRACE   | Writing results to TensorBoard:
2023/02/27 00:14:29 787054 |  TRACE   |  - Start loss computation
2023/02/27 00:14:29 787518 |  TRACE   |  - Start image generation for 100 images
2023/02/27 00:15:59 803301 |  TRACE   |  - No improvement for loss: 0.27320 (current best: 0.26876)
2023/02/27 00:15:59 803962 |  TRACE   | Training for 250 steps (current step: 1750)
2023/02/27 00:28:17 856192 |  TRACE   | Validating
2023/02/27 00:28:45 438691 |  TRACE   | Writing results to TensorBoard:
2023/02/27 00:28:45 439380 |  TRACE   |  - Start loss computation
2023/02/27 00:28:45 439811 |  TRACE   |  - Start image generation for 100 images
2023/02/27 00:30:13 110100 |  TRACE   |  - No improvement for loss: 0.28414 (current best: 0.26876)
2023/02/27 00:30:13 110719 |  TRACE   | Training for 250 steps (current step: 2000)
2023/02/27 00:42:58 909671 |  TRACE   | Validating
2023/02/27 00:43:27 895375 |  TRACE   | Writing results to TensorBoard:
2023/02/27 00:43:27 896033 |  TRACE   |  - Start loss computation
2023/02/27 00:43:27 896486 |  TRACE   |  - Start image generation for 100 images
2023/02/27 00:44:51 754280 |  TRACE   |  - New best loss: 0.26016
2023/02/27 00:44:51 754991 |  TRACE   |  - Improvement over previous best: -0.00860
2023/02/27 00:44:51 891756 |  TRACE   |  - Best model updated!
 
2023/02/27 00:44:51 892339 |  TRACE   | Training for 250 steps (current step: 2250)
2023/02/27 00:56:56 110527 |  TRACE   | Validating
2023/02/27 00:57:24 934084 |  TRACE   | Writing results to TensorBoard:
2023/02/27 00:57:24 934834 |  TRACE   |  - Start loss computation
2023/02/27 00:57:24 935311 |  TRACE   |  - Start image generation for 100 images
2023/02/27 00:58:57 401165 |  TRACE   |  - No improvement for loss: 0.26781 (current best: 0.26016)
2023/02/27 00:58:57 401844 |  TRACE   | Training for 250 steps (current step: 2500)
2023/02/27 01:11:35 125914 |  TRACE   | Validating
2023/02/27 01:12:02 352705 |  TRACE   | Writing results to TensorBoard:
2023/02/27 01:12:02 353382 |  TRACE   |  - Start loss computation
2023/02/27 01:12:02 353841 |  TRACE   |  - Start image generation for 100 images
2023/02/27 01:13:26 896037 |  TRACE   |  - New best loss: 0.25107
2023/02/27 01:13:26 896893 |  TRACE   |  - Improvement over previous best: -0.00909
2023/02/27 01:13:27 033661 |  TRACE   |  - Best model updated!
 
2023/02/27 01:13:27 034272 |  TRACE   | Training for 250 steps (current step: 2750)
2023/02/27 01:25:39 231901 |  TRACE   | Validating
2023/02/27 01:26:07 876758 |  TRACE   | Writing results to TensorBoard:
2023/02/27 01:26:07 877428 |  TRACE   |  - Start loss computation
2023/02/27 01:26:07 877876 |  TRACE   |  - Start image generation for 100 images
2023/02/27 01:27:35 070823 |  TRACE   |  - No improvement for loss: 0.25364 (current best: 0.25107)
2023/02/27 01:27:35 071618 |  TRACE   | Training for 250 steps (current step: 3000)
2023/02/27 01:40:04 839682 |  TRACE   | Validating
2023/02/27 01:40:32 550017 |  TRACE   | Writing results to TensorBoard:
2023/02/27 01:40:32 550657 |  TRACE   |  - Start loss computation
2023/02/27 01:40:32 551123 |  TRACE   |  - Start image generation for 100 images
2023/02/27 01:42:01 510544 |  TRACE   |  - No improvement for loss: 0.26981 (current best: 0.25107)
2023/02/27 01:42:01 511093 |  TRACE   | Training for 250 steps (current step: 3250)
2023/02/27 01:54:16 650322 |  TRACE   | Validating
2023/02/27 01:54:43 767094 |  TRACE   | Writing results to TensorBoard:
2023/02/27 01:54:43 767690 |  TRACE   |  - Start loss computation
2023/02/27 01:54:43 768254 |  TRACE   |  - Start image generation for 100 images
2023/02/27 01:56:04 534111 |  TRACE   |  - New best loss: 0.24445
2023/02/27 01:56:04 534896 |  TRACE   |  - Improvement over previous best: -0.00662
2023/02/27 01:56:04 670068 |  TRACE   |  - Best model updated!
 
2023/02/27 01:56:04 670644 |  TRACE   | Training for 250 steps (current step: 3500)
2023/02/27 02:08:27 797250 |  TRACE   | Validating
2023/02/27 02:08:58 105980 |  TRACE   | Writing results to TensorBoard:
2023/02/27 02:08:58 106675 |  TRACE   |  - Start loss computation
2023/02/27 02:08:58 107120 |  TRACE   |  - Start image generation for 100 images
2023/02/27 02:10:14 468668 |  TRACE   |  - No improvement for loss: 0.25478 (current best: 0.24445)
2023/02/27 02:10:14 469470 |  TRACE   | Training for 250 steps (current step: 3750)
2023/02/27 02:22:45 071103 |  TRACE   | Validating
2023/02/27 02:23:13 061438 |  TRACE   | Writing results to TensorBoard:
2023/02/27 02:23:13 062135 |  TRACE   |  - Start loss computation
2023/02/27 02:23:13 062583 |  TRACE   |  - Start image generation for 100 images
2023/02/27 02:24:30 327342 |  TRACE   |  - No improvement for loss: 0.24639 (current best: 0.24445)
2023/02/27 02:24:30 327963 |  TRACE   | Training for 250 steps (current step: 4000)
2023/02/27 02:37:14 082770 |  TRACE   | Validating
2023/02/27 02:37:41 544867 |  TRACE   | Writing results to TensorBoard:
2023/02/27 02:37:41 545564 |  TRACE   |  - Start loss computation
2023/02/27 02:37:41 546108 |  TRACE   |  - Start image generation for 100 images
2023/02/27 02:38:53 191501 |  TRACE   |  - No improvement for loss: 0.25359 (current best: 0.24445)
2023/02/27 02:38:53 192181 |  TRACE   | Training for 250 steps (current step: 4250)
2023/02/27 02:51:07 291857 |  TRACE   | Validating
2023/02/27 02:51:36 483232 |  TRACE   | Writing results to TensorBoard:
2023/02/27 02:51:36 483934 |  TRACE   |  - Start loss computation
2023/02/27 02:51:36 484364 |  TRACE   |  - Start image generation for 100 images
2023/02/27 02:53:01 177049 |  TRACE   |  - No improvement for loss: 0.24683 (current best: 0.24445)
2023/02/27 02:53:01 177893 |  TRACE   | Training for 250 steps (current step: 4500)
2023/02/27 03:05:32 577696 |  TRACE   | Validating
2023/02/27 03:06:00 604681 |  TRACE   | Writing results to TensorBoard:
2023/02/27 03:06:00 605325 |  TRACE   |  - Start loss computation
2023/02/27 03:06:00 605754 |  TRACE   |  - Start image generation for 100 images
2023/02/27 03:07:19 357554 |  TRACE   |  - New best loss: 0.23335
2023/02/27 03:07:19 358203 |  TRACE   |  - Improvement over previous best: -0.01109
2023/02/27 03:07:19 495164 |  TRACE   |  - Best model updated!
 
2023/02/27 03:07:19 495691 |  TRACE   | Training for 250 steps (current step: 4750)
2023/02/27 03:19:40 387139 |  TRACE   | Validating
2023/02/27 03:20:07 993561 |  TRACE   | Writing results to TensorBoard:
2023/02/27 03:20:07 994272 |  TRACE   |  - Start loss computation
2023/02/27 03:20:07 994737 |  TRACE   |  - Start image generation for 100 images
2023/02/27 03:21:33 398713 |  TRACE   |  - New best loss: 0.23282
2023/02/27 03:21:33 399533 |  TRACE   |  - Improvement over previous best: -0.00053
2023/02/27 03:21:33 541414 |  TRACE   |  - Best model updated!
 
2023/02/27 03:21:33 542060 |  TRACE   | Training for 250 steps (current step: 5000)
2023/02/27 03:34:01 217707 |  TRACE   | Validating
2023/02/27 03:34:30 629939 |  TRACE   | Writing results to TensorBoard:
2023/02/27 03:34:30 630637 |  TRACE   |  - Start loss computation
2023/02/27 03:34:30 631139 |  TRACE   |  - Start image generation for 100 images
2023/02/27 03:35:54 647717 |  TRACE   |  - New best loss: 0.23218
2023/02/27 03:35:54 648475 |  TRACE   |  - Improvement over previous best: -0.00064
2023/02/27 03:35:54 793378 |  TRACE   |  - Best model updated!
 
2023/02/27 03:35:54 793959 |  TRACE   | Training for 250 steps (current step: 5250)
2023/02/27 03:48:01 372075 |  TRACE   | Validating
2023/02/27 03:48:29 993155 |  TRACE   | Writing results to TensorBoard:
2023/02/27 03:48:29 993794 |  TRACE   |  - Start loss computation
2023/02/27 03:48:29 994238 |  TRACE   |  - Start image generation for 100 images
2023/02/27 03:49:49 934449 |  TRACE   |  - New best loss: 0.23205
2023/02/27 03:49:49 935098 |  TRACE   |  - Improvement over previous best: -0.00013
2023/02/27 03:49:50 075452 |  TRACE   |  - Best model updated!
 
2023/02/27 03:49:50 075978 |  TRACE   | Training for 250 steps (current step: 5500)
2023/02/27 04:02:21 635121 |  TRACE   | Validating
2023/02/27 04:02:48 809904 |  TRACE   | Writing results to TensorBoard:
2023/02/27 04:02:48 810556 |  TRACE   |  - Start loss computation
2023/02/27 04:02:48 811164 |  TRACE   |  - Start image generation for 100 images
2023/02/27 04:04:10 066375 |  TRACE   |  - No improvement for loss: 0.23256 (current best: 0.23205)
2023/02/27 04:04:10 066891 |  TRACE   | Training for 250 steps (current step: 5750)
2023/02/27 04:16:12 127395 |  TRACE   | Validating
2023/02/27 04:16:40 155131 |  TRACE   | Writing results to TensorBoard:
2023/02/27 04:16:40 155778 |  TRACE   |  - Start loss computation
2023/02/27 04:16:40 156256 |  TRACE   |  - Start image generation for 100 images
2023/02/27 04:18:00 167875 |  TRACE   |  - New best loss: 0.23148
2023/02/27 04:18:00 168653 |  TRACE   |  - Improvement over previous best: -0.00057
2023/02/27 04:18:00 307140 |  TRACE   |  - Best model updated!
 
2023/02/27 04:18:00 307657 |  TRACE   | Training for 250 steps (current step: 6000)
2023/02/27 04:30:28 147349 |  TRACE   | Validating
2023/02/27 04:30:55 378764 |  TRACE   | Writing results to TensorBoard:
2023/02/27 04:30:55 379424 |  TRACE   |  - Start loss computation
2023/02/27 04:30:55 379838 |  TRACE   |  - Start image generation for 100 images
2023/02/27 04:32:14 057358 |  TRACE   |  - New best loss: 0.23085
2023/02/27 04:32:14 058021 |  TRACE   |  - Improvement over previous best: -0.00063
2023/02/27 04:32:14 189863 |  TRACE   |  - Best model updated!
 
2023/02/27 04:32:14 190303 |  TRACE   | Training for 250 steps (current step: 6250)
2023/02/27 04:44:22 018559 |  TRACE   | Validating
2023/02/27 04:44:49 727867 |  TRACE   | Writing results to TensorBoard:
2023/02/27 04:44:49 728531 |  TRACE   |  - Start loss computation
2023/02/27 04:44:49 728975 |  TRACE   |  - Start image generation for 100 images
2023/02/27 04:46:10 398999 |  TRACE   |  - No improvement for loss: 0.23135 (current best: 0.23085)
2023/02/27 04:46:10 399491 |  TRACE   | Training for 250 steps (current step: 6500)
2023/02/27 04:58:37 345077 |  TRACE   | Validating
2023/02/27 04:59:04 340816 |  TRACE   | Writing results to TensorBoard:
2023/02/27 04:59:04 341436 |  TRACE   |  - Start loss computation
2023/02/27 04:59:04 341852 |  TRACE   |  - Start image generation for 100 images
2023/02/27 05:00:23 436192 |  TRACE   |  - No improvement for loss: 0.23106 (current best: 0.23085)
2023/02/27 05:00:23 436666 |  TRACE   | Training for 250 steps (current step: 6750)
2023/02/27 05:12:30 359014 |  TRACE   | Validating
2023/02/27 05:12:57 843893 |  TRACE   | Writing results to TensorBoard:
2023/02/27 05:12:57 844574 |  TRACE   |  - Start loss computation
2023/02/27 05:12:57 845076 |  TRACE   |  - Start image generation for 100 images
2023/02/27 05:14:23 319716 |  TRACE   |  - New best loss: 0.23060
2023/02/27 05:14:23 320508 |  TRACE   |  - Improvement over previous best: -0.00025
2023/02/27 05:14:23 456290 |  TRACE   |  - Best model updated!
 
2023/02/27 05:14:23 456860 |  TRACE   | Training for 250 steps (current step: 7000)
2023/02/27 05:26:53 926258 |  TRACE   | Validating
2023/02/27 05:27:23 000156 |  TRACE   | Writing results to TensorBoard:
2023/02/27 05:27:23 000828 |  TRACE   |  - Start loss computation
2023/02/27 05:27:23 001283 |  TRACE   |  - Start image generation for 100 images
2023/02/27 05:28:42 202331 |  TRACE   |  - No improvement for loss: 0.23059 (current best: 0.23060)
2023/02/27 05:28:42 203028 |  TRACE   | Training for 250 steps (current step: 7250)
2023/02/27 05:41:04 119229 |  TRACE   | Validating
2023/02/27 05:41:31 669642 |  TRACE   | Writing results to TensorBoard:
2023/02/27 05:41:31 670320 |  TRACE   |  - Start loss computation
2023/02/27 05:41:31 670773 |  TRACE   |  - Start image generation for 100 images
2023/02/27 05:42:55 836300 |  TRACE   |  - No improvement for loss: 0.23051 (current best: 0.23060)
2023/02/27 05:42:55 836866 |  TRACE   | Training for 250 steps (current step: 7500)
2023/02/27 05:55:29 171678 |  TRACE   | Validating
2023/02/27 05:55:57 220120 |  TRACE   | Writing results to TensorBoard:
2023/02/27 05:55:57 220778 |  TRACE   |  - Start loss computation
2023/02/27 05:55:57 221187 |  TRACE   |  - Start image generation for 100 images
2023/02/27 05:57:17 119709 |  TRACE   |  - No improvement for loss: 0.23082 (current best: 0.23060)
2023/02/27 05:57:17 120349 |  TRACE   | Training for 250 steps (current step: 7750)
2023/02/27 06:09:10 165220 |  TRACE   | Validating
2023/02/27 06:09:37 446759 |  TRACE   | Writing results to TensorBoard:
2023/02/27 06:09:37 447471 |  TRACE   |  - Start loss computation
2023/02/27 06:09:37 447896 |  TRACE   |  - Start image generation for 100 images
2023/02/27 06:10:56 727726 |  TRACE   |  - No improvement for loss: 0.23054 (current best: 0.23060)
2023/02/27 06:10:56 728332 |  TRACE   | Training for 250 steps (current step: 8000)
2023/02/27 06:23:18 177791 |  TRACE   | Validating
2023/02/27 06:23:46 199983 |  TRACE   | Writing results to TensorBoard:
2023/02/27 06:23:46 200677 |  TRACE   |  - Start loss computation
2023/02/27 06:23:46 201101 |  TRACE   |  - Start image generation for 100 images
2023/02/27 06:25:05 833440 |  TRACE   |  - New best loss: 0.22969
2023/02/27 06:25:05 834104 |  TRACE   |  - Improvement over previous best: -0.00092
2023/02/27 06:25:05 988705 |  TRACE   |  - Best model updated!
 
2023/02/27 06:25:05 989200 |  TRACE   | Training for 250 steps (current step: 8250)
2023/02/27 06:37:11 466572 |  TRACE   | Validating
2023/02/27 06:37:38 618923 |  TRACE   | Writing results to TensorBoard:
2023/02/27 06:37:38 619558 |  TRACE   |  - Start loss computation
2023/02/27 06:37:38 620005 |  TRACE   |  - Start image generation for 100 images
2023/02/27 06:39:05 200411 |  TRACE   |  - No improvement for loss: 0.23118 (current best: 0.22969)
2023/02/27 06:39:05 200919 |  TRACE   | Training for 250 steps (current step: 8500)
2023/02/27 06:51:20 012516 |  TRACE   | Validating
2023/02/27 06:51:48 383322 |  TRACE   | Writing results to TensorBoard:
2023/02/27 06:51:48 384029 |  TRACE   |  - Start loss computation
2023/02/27 06:51:48 384467 |  TRACE   |  - Start image generation for 100 images
2023/02/27 06:53:09 111314 |  TRACE   |  - No improvement for loss: 0.23086 (current best: 0.22969)
2023/02/27 06:53:09 111938 |  TRACE   | Training for 250 steps (current step: 8750)
2023/02/27 07:05:02 742156 |  TRACE   | Validating
2023/02/27 07:05:31 047988 |  TRACE   | Writing results to TensorBoard:
2023/02/27 07:05:31 048653 |  TRACE   |  - Start loss computation
2023/02/27 07:05:31 049124 |  TRACE   |  - Start image generation for 100 images
2023/02/27 07:06:50 483986 |  TRACE   |  - No improvement for loss: 0.23052 (current best: 0.22969)
2023/02/27 07:06:50 484685 |  TRACE   | Training for 250 steps (current step: 9000)
2023/02/27 07:19:17 373514 |  TRACE   | Validating
2023/02/27 07:19:45 483907 |  TRACE   | Writing results to TensorBoard:
2023/02/27 07:19:45 484569 |  TRACE   |  - Start loss computation
2023/02/27 07:19:45 484984 |  TRACE   |  - Start image generation for 100 images
2023/02/27 07:21:04 083234 |  TRACE   |  - No improvement for loss: 0.23056 (current best: 0.22969)
2023/02/27 07:21:04 083766 |  TRACE   | Training for 250 steps (current step: 9250)
2023/02/27 07:33:10 840534 |  TRACE   | Validating
2023/02/27 07:33:38 020824 |  TRACE   | Writing results to TensorBoard:
2023/02/27 07:33:38 021449 |  TRACE   |  - Start loss computation
2023/02/27 07:33:38 021919 |  TRACE   |  - Start image generation for 100 images
2023/02/27 07:34:58 108533 |  TRACE   |  - New best loss: 0.22943
2023/02/27 07:34:58 109124 |  TRACE   |  - Improvement over previous best: -0.00025
2023/02/27 07:34:58 246945 |  TRACE   |  - Best model updated!
 
2023/02/27 07:34:58 247400 |  TRACE   | Training for 250 steps (current step: 9500)
2023/02/27 07:47:19 516997 |  TRACE   | Validating
2023/02/27 07:47:46 556048 |  TRACE   | Writing results to TensorBoard:
2023/02/27 07:47:46 556751 |  TRACE   |  - Start loss computation
2023/02/27 07:47:46 557208 |  TRACE   |  - Start image generation for 100 images
2023/02/27 07:49:12 074361 |  TRACE   |  - New best loss: 0.22929
2023/02/27 07:49:12 075146 |  TRACE   |  - Improvement over previous best: -0.00014
2023/02/27 07:49:12 591174 |  TRACE   |  - Best model updated!
 
2023/02/27 07:49:12 591685 |  TRACE   | Training for 250 steps (current step: 9750)
2023/02/27 08:01:12 597232 |  TRACE   | Validating
2023/02/27 08:01:40 619239 |  TRACE   | Writing results to TensorBoard:
2023/02/27 08:01:40 619861 |  TRACE   |  - Start loss computation
2023/02/27 08:01:40 620324 |  TRACE   |  - Start image generation for 100 images
2023/02/27 08:02:59 981479 |  TRACE   |  - New best loss: 0.22910
2023/02/27 08:02:59 982408 |  TRACE   |  - Improvement over previous best: -0.00019
2023/02/27 08:03:00 122725 |  TRACE   |  - Best model updated!
 
2023/02/27 08:03:00 123284 |  TRACE   | Training for 250 steps (current step: 10000)
2023/02/27 08:15:18 426514 |  TRACE   | Validating
2023/02/27 08:15:45 438862 |  TRACE   | Writing results to TensorBoard:
2023/02/27 08:15:45 439486 |  TRACE   |  - Start loss computation
2023/02/27 08:15:45 439930 |  TRACE   |  - Start image generation for 100 images
2023/02/27 08:17:03 933356 |  TRACE   |  - No improvement for loss: 0.22908 (current best: 0.22910)
2023/02/27 08:17:03 933899 |  TRACE   | Training for 250 steps (current step: 10250)
2023/02/27 08:29:07 988289 |  TRACE   | Validating
2023/02/27 08:29:36 763682 |  TRACE   | Writing results to TensorBoard:
2023/02/27 08:29:36 764398 |  TRACE   |  - Start loss computation
2023/02/27 08:29:36 765134 |  TRACE   |  - Start image generation for 100 images
2023/02/27 08:30:56 391512 |  TRACE   |  - New best loss: 0.22882
2023/02/27 08:30:56 392173 |  TRACE   |  - Improvement over previous best: -0.00028
2023/02/27 08:30:56 534294 |  TRACE   |  - Best model updated!
 
2023/02/27 08:30:56 534767 |  TRACE   | Training for 250 steps (current step: 10500)
2023/02/27 08:43:19 484934 |  TRACE   | Validating
2023/02/27 08:43:47 118552 |  TRACE   | Writing results to TensorBoard:
2023/02/27 08:43:47 119278 |  TRACE   |  - Start loss computation
2023/02/27 08:43:47 119806 |  TRACE   |  - Start image generation for 100 images
2023/02/27 08:45:09 865568 |  TRACE   |  - No improvement for loss: 0.22879 (current best: 0.22882)
2023/02/27 08:45:09 866219 |  TRACE   | Training for 250 steps (current step: 10750)
2023/02/27 08:57:15 356717 |  TRACE   | Validating
2023/02/27 08:57:42 781063 |  TRACE   | Writing results to TensorBoard:
2023/02/27 08:57:42 781660 |  TRACE   |  - Start loss computation
2023/02/27 08:57:42 782226 |  TRACE   |  - Start image generation for 100 images
2023/02/27 08:59:02 652024 |  TRACE   |  - No improvement for loss: 0.22882 (current best: 0.22882)
2023/02/27 08:59:02 652705 |  TRACE   | Training for 250 steps (current step: 11000)
2023/02/27 09:11:17 311924 |  TRACE   | Validating
2023/02/27 09:11:46 017965 |  TRACE   | Writing results to TensorBoard:
2023/02/27 09:11:46 018639 |  TRACE   |  - Start loss computation
2023/02/27 09:11:46 019203 |  TRACE   |  - Start image generation for 100 images
2023/02/27 09:13:10 630675 |  TRACE   |  - No improvement for loss: 0.22882 (current best: 0.22882)
2023/02/27 09:13:10 631378 |  TRACE   | Training for 250 steps (current step: 11250)
2023/02/27 09:25:08 915642 |  TRACE   | Validating
2023/02/27 09:25:37 399689 |  TRACE   | Writing results to TensorBoard:
2023/02/27 09:25:37 400370 |  TRACE   |  - Start loss computation
2023/02/27 09:25:37 400816 |  TRACE   |  - Start image generation for 100 images
2023/02/27 09:26:56 284011 |  TRACE   |  - No improvement for loss: 0.22899 (current best: 0.22882)
2023/02/27 09:26:56 284648 |  TRACE   | Training for 250 steps (current step: 11500)
2023/02/27 09:39:09 869949 |  TRACE   | Validating
2023/02/27 09:39:37 689131 |  TRACE   | Writing results to TensorBoard:
2023/02/27 09:39:37 689776 |  TRACE   |  - Start loss computation
2023/02/27 09:39:37 690262 |  TRACE   |  - Start image generation for 100 images
2023/02/27 09:40:56 818654 |  TRACE   |  - No improvement for loss: 0.22884 (current best: 0.22882)
2023/02/27 09:40:56 819274 |  TRACE   | Training for 250 steps (current step: 11750)
2023/02/27 09:53:03 289416 |  TRACE   | Validating
2023/02/27 09:53:31 078152 |  TRACE   | Writing results to TensorBoard:
2023/02/27 09:53:31 078760 |  TRACE   |  - Start loss computation
2023/02/27 09:53:31 079710 |  TRACE   |  - Start image generation for 100 images
2023/02/27 09:54:50 231852 |  TRACE   |  - No improvement for loss: 0.22885 (current best: 0.22882)
2023/02/27 09:54:50 232433 |  TRACE   | Training for 250 steps (current step: 12000)
2023/02/27 10:07:11 452080 |  TRACE   | Validating
2023/02/27 10:07:38 231908 |  TRACE   | Writing results to TensorBoard:
2023/02/27 10:07:38 232678 |  TRACE   |  - Start loss computation
2023/02/27 10:07:38 233195 |  TRACE   |  - Start image generation for 100 images
2023/02/27 10:08:58 181918 |  TRACE   |  - No improvement for loss: 0.22885 (current best: 0.22882)
2023/02/27 10:08:58 182488 |  TRACE   | Training for 250 steps (current step: 12250)
2023/02/27 10:21:02 569370 |  TRACE   | Validating
2023/02/27 10:21:29 837683 |  TRACE   | Writing results to TensorBoard:
2023/02/27 10:21:29 838318 |  TRACE   |  - Start loss computation
2023/02/27 10:21:29 838951 |  TRACE   |  - Start image generation for 100 images
2023/02/27 10:22:50 220008 |  TRACE   |  - No improvement for loss: 0.22875 (current best: 0.22882)
2023/02/27 10:22:50 220577 |  TRACE   | Training for 250 steps (current step: 12500)
2023/02/27 10:35:15 675013 |  TRACE   | Validating
2023/02/27 10:35:43 807152 |  TRACE   | Writing results to TensorBoard:
2023/02/27 10:35:43 807824 |  TRACE   |  - Start loss computation
2023/02/27 10:35:43 808310 |  TRACE   |  - Start image generation for 100 images
2023/02/27 10:37:03 222726 |  TRACE   |  - No improvement for loss: 0.22880 (current best: 0.22882)
2023/02/27 10:37:03 223463 |  TRACE   | Training for 250 steps (current step: 12750)
2023/02/27 10:49:08 060196 |  TRACE   | Validating
2023/02/27 10:49:36 601678 |  TRACE   | Writing results to TensorBoard:
2023/02/27 10:49:36 602406 |  TRACE   |  - Start loss computation
2023/02/27 10:49:36 602938 |  TRACE   |  - Start image generation for 100 images
2023/02/27 10:50:55 186164 |  TRACE   |  - No improvement for loss: 0.22878 (current best: 0.22882)
2023/02/27 10:50:55 186780 |  TRACE   |  - Early Stopping tolerance reached. Stopping training!
 
2023/02/27 10:50:55 502875 |  TRACE   | Start model evaluation
2023/02/27 15:06:02 572938 |  TRACE   | Start model evaluation
2023/02/27 15:10:00 277485 |  TRACE   | Evaluation results:
2023/02/27 15:10:00 278704 |  TRACE   |  - Results for field size: global
2023/02/27 15:10:00 279038 |  TRACE   |    - Precision: 47.37%
2023/02/27 15:10:00 279292 |  TRACE   |    - Recall: 52.41%
2023/02/27 15:10:00 279481 |  TRACE   |  - Results for field size: small
2023/02/27 15:10:00 279715 |  TRACE   |    - Precision: 23.06%
2023/02/27 15:10:00 279957 |  TRACE   |    - Recall: 26.80%
2023/02/27 15:10:00 280239 |  TRACE   |  - Results for field size: medium
2023/02/27 15:10:00 280480 |  TRACE   |    - Precision: 58.62%
2023/02/27 15:10:00 280717 |  TRACE   |    - Recall: 53.45%
2023/02/27 15:10:00 280904 |  TRACE   |  - Results for field size: large
2023/02/27 15:10:00 281132 |  TRACE   |    - Precision: 71.26%
2023/02/27 15:10:00 281369 |  TRACE   |    - Recall: 66.87%
2023/02/27 15:10:00 281556 |  TRACE   |  - Results for field size: extra large
2023/02/27 15:10:00 281781 |  TRACE   |    - Precision: 75.36%
2023/02/27 15:10:00 282009 |  TRACE   |    - Recall: 70.68%
