When I train the model use bavsic_pp backbone and REDS by given config file, always stop in the 5000th epoch. #1768

Libyte · 2023-04-18T03:25:36Z

Libyte
Apr 18, 2023

When I train the model use bavsic_pp backbone and REDS by given config file, always stop in the 5000th epoch. I check the config file and change some args, but still not work.

Answered by Z-Fran

Apr 19, 2023

Please use the new version https://github.com/open-mmlab/mmediting/tree/main . I run it successfully on MMEditing v1.0.0rc7. And you can use vis_backends = [dict(type='TensorboardVisBackend')] or vis_backends = [dict(type='WandbVisBackend')] in config to visualize loss

View full answer

Z-Fran · 2023-04-18T08:41:24Z

Z-Fran
Apr 18, 2023
Collaborator

Please upload your logs and error messages/

2 replies

Libyte Apr 18, 2023
Author

2023-03-23 13:58:17,669 - mmedit - INFO - Environment info:

sys.platform: linux
Python: 3.8.16 (default, Jan 17 2023, 23:13:24) [GCC 11.2.0]
CUDA available: True
GPU 0: NVIDIA GeForce GTX 1080 Ti
CUDA_HOME: /usr/local/cuda-11.6
NVCC: Cuda compilation tools, release 11.6, V11.6.55
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 1.10.2
PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.3
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
CuDNN 8.2
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.11.3
OpenCV: 4.7.0
MMCV: 1.7.1
MMCV Compiler: GCC 9.4
MMCV CUDA Compiler: 11.6
MMEditing: 0.16.0+8de2721

2023-03-23 13:58:17,686 - mmedit - INFO - Distributed training: False
2023-03-23 13:58:17,686 - mmedit - INFO - mmedit Version: 0.16.0
2023-03-23 13:58:17,686 - mmedit - INFO - Config:
/home/ici/mmediting/configs/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4.py
exp_name = 'basicvsr_plusplus_c64n7_8x1_600k_reds4'

model settings

model = dict(
type='BasicVSR',
generator=dict(
type='BasicVSRPlusPlus',
mid_channels=64,
num_blocks=7,
is_low_res_input=True,
spynet_pretrained='https://download.openmmlab.com/mmediting/restorers/'
'basicvsr/spynet_20210409-c6c1bd09.pth'),
pixel_loss=dict(type='CharbonnierLoss', loss_weight=1.0, reduction='mean'))

model training and testing settings

train_cfg = dict(fix_iter=5000)
test_cfg = dict(metrics=['PSNR'], crop_border=0)

dataset settings

train_dataset_type = 'SRREDSMultipleGTDataset'
val_dataset_type = 'SRREDSMultipleGTDataset'

train_pipeline = [
dict(type='GenerateSegmentIndices', interval_list=[1]),
dict(
type='LoadImageFromFileList',
io_backend='disk',
key='lq',
channel_order='rgb'),
dict(
type='LoadImageFromFileList',
io_backend='disk',
key='gt',
channel_order='rgb'),
dict(type='RescaleToZeroOne', keys=['lq', 'gt']),
dict(type='PairedRandomCrop', gt_patch_size=256),
dict(
type='Flip', keys=['lq', 'gt'], flip_ratio=0.5,
direction='horizontal'),
dict(type='Flip', keys=['lq', 'gt'], flip_ratio=0.5, direction='vertical'),
dict(type='RandomTransposeHW', keys=['lq', 'gt'], transpose_ratio=0.5),
dict(type='FramesToTensor', keys=['lq', 'gt']),
dict(type='Collect', keys=['lq', 'gt'], meta_keys=['lq_path', 'gt_path'])
]

test_pipeline = [
dict(type='GenerateSegmentIndices', interval_list=[1]),
dict(
type='LoadImageFromFileList',
io_backend='disk',
key='lq',
channel_order='rgb'),
dict(
type='LoadImageFromFileList',
io_backend='disk',
key='gt',
channel_order='rgb'),
dict(type='RescaleToZeroOne', keys=['lq', 'gt']),
dict(type='FramesToTensor', keys=['lq', 'gt']),
dict(
type='Collect',
keys=['lq', 'gt'],
meta_keys=['lq_path', 'gt_path', 'key'])
]

demo_pipeline = [
dict(type='GenerateSegmentIndices', interval_list=[1]),
dict(
type='LoadImageFromFileList',
io_backend='disk',
key='lq',
channel_order='rgb'),
dict(type='RescaleToZeroOne', keys=['lq']),
dict(type='FramesToTensor', keys=['lq']),
dict(type='Collect', keys=['lq'], meta_keys=['lq_path', 'key'])
]

data = dict(
workers_per_gpu=6,
train_dataloader=dict(samples_per_gpu=1, drop_last=True), # 8 gpus
val_dataloader=dict(samples_per_gpu=1),
test_dataloader=dict(samples_per_gpu=1, workers_per_gpu=1),

# train
train=dict(
    type='RepeatDataset',
    times=1000,
    dataset=dict(
        type=train_dataset_type,
        lq_folder='data/REDS/train_sharp_bicubic/X4',
        gt_folder='data/REDS/train_sharp',
        num_input_frames=30,
        pipeline=train_pipeline,
        scale=4,
        val_partition='REDS4',
        test_mode=False)),
# val
val=dict(
    type=val_dataset_type,
    lq_folder='data/REDS/train_sharp_bicubic/X4',
    gt_folder='data/REDS/train_sharp',
    num_input_frames=100,
    pipeline=test_pipeline,
    scale=4,
    val_partition='REDS4',
    repeat=2,
    test_mode=True),
# test
test=dict(
    type=val_dataset_type,
    lq_folder='data/REDS/train_sharp_bicubic/X4',
    gt_folder='data/REDS/train_sharp',
    num_input_frames=100,
    pipeline=test_pipeline,
    scale=4,
    val_partition='REDS4',
    test_mode=True),

)

optimizer

optimizers = dict(
generator=dict(
type='Adam',
lr=1e-4,
betas=(0.9, 0.99),
paramwise_cfg=dict(custom_keys={'spynet': dict(lr_mult=0.25)})))

learning policy

total_iters = 600000
lr_config = dict(
policy='CosineRestart',
by_epoch=False,
periods=[600000],
restart_weights=[1],
min_lr=1e-7)

checkpoint_config = dict(interval=5000, save_optimizer=True, by_epoch=False)

remove gpu_collect=True in non distributed training

evaluation = dict(interval=5000, save_image=False, gpu_collect=True)
log_config = dict(
interval=100,
hooks=[
dict(type='TextLoggerHook', by_epoch=False),
# dict(type='TensorboardLoggerHook'),
])
visual_config = None

runtime settings

dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = f'./work_dirs/{exp_name}'
load_from = None
resume_from = None
workflow = [('train', 1)]
find_unused_parameters = True

2023-03-23 13:58:17,686 - mmedit - INFO - Set random seed to 1015228841, deterministic: False
2023-03-23 13:58:17,710 - mmedit - INFO - load checkpoint from http path: https://download.openmmlab.com/mmediting/restorers/basicvsr/spynet_20210409-c6c1bd09.pth
2023-03-23 13:58:22,937 - mmedit - INFO - Start running, host: ici@1080, work_dir: /home/ici/mmediting/work_dirs/basicvsr_plusplus_c64n7_8x1_600k_reds4
2023-03-23 13:58:22,937 - mmedit - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) CosineRestartLrUpdaterHook
(NORMAL ) CheckpointHook
(VERY_LOW ) TextLoggerHook

before_train_epoch:
(VERY_HIGH ) CosineRestartLrUpdaterHook
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook

before_train_iter:
(VERY_HIGH ) CosineRestartLrUpdaterHook
(LOW ) IterTimerHook

after_train_iter:
(NORMAL ) CheckpointHook
(LOW ) IterTimerHook
(LOW ) EvalIterHook
(VERY_LOW ) TextLoggerHook

after_train_epoch:
(NORMAL ) CheckpointHook
(VERY_LOW ) TextLoggerHook

before_val_epoch:
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook

before_val_iter:
(LOW ) IterTimerHook

after_val_iter:
(LOW ) IterTimerHook

after_val_epoch:
(VERY_LOW ) TextLoggerHook

after_run:
(VERY_LOW ) TextLoggerHook

2023-03-23 13:58:22,938 - mmedit - INFO - workflow: [('train', 1)], max: 600000 iters
2023-03-23 13:58:22,938 - mmedit - INFO - Checkpoints will be saved to /home/ici/mmediting/work_dirs/basicvsr_plusplus_c64n7_8x1_600k_reds4 by HardDiskBackend.
2023-03-23 14:00:48,593 - mmedit - INFO - Iter [100/600000] lr_generator: 2.500e-05, eta: 10 days, 2:19:40, time: 1.454, data_time: 0.029, memory: 6669, loss_pix: 0.0331, loss: 0.0331
2023-03-23 14:03:12,303 - mmedit - INFO - Iter [200/600000] lr_generator: 2.500e-05, eta: 10 days, 0:52:46, time: 1.437, data_time: 0.006, memory: 6669, loss_pix: 0.0349, loss: 0.0349
2023-03-23 14:05:36,824 - mmedit - INFO - Iter [300/600000] lr_generator: 2.500e-05, eta: 10 days, 0:48:20, time: 1.445, data_time: 0.006, memory: 6669, loss_pix: 0.0314, loss: 0.0314
2023-03-23 14:08:01,738 - mmedit - INFO - Iter [400/600000] lr_generator: 2.500e-05, eta: 10 days, 0:54:57, time: 1.449, data_time: 0.006, memory: 6669, loss_pix: 0.0303, loss: 0.0303
2023-03-23 14:10:26,942 - mmedit - INFO - Iter [500/600000] lr_generator: 2.500e-05, eta: 10 days, 1:03:32, time: 1.452, data_time: 0.006, memory: 6669, loss_pix: 0.0290, loss: 0.0290
2023-03-23 14:12:53,515 - mmedit - INFO - Iter [600/600000] lr_generator: 2.500e-05, eta: 10 days, 1:31:17, time: 1.466, data_time: 0.005, memory: 6669, loss_pix: 0.0282, loss: 0.0282
2023-03-23 14:15:19,753 - mmedit - INFO - Iter [700/600000] lr_generator: 2.500e-05, eta: 10 days, 1:45:43, time: 1.462, data_time: 0.006, memory: 6669, loss_pix: 0.0279, loss: 0.0279
2023-03-23 14:17:47,296 - mmedit - INFO - Iter [800/600000] lr_generator: 2.500e-05, eta: 10 days, 2:12:07, time: 1.475, data_time: 0.006, memory: 6669, loss_pix: 0.0293, loss: 0.0293
2023-03-23 14:20:12,445 - mmedit - INFO - Iter [900/600000] lr_generator: 2.500e-05, eta: 10 days, 2:05:37, time: 1.451, data_time: 0.006, memory: 6669, loss_pix: 0.0267, loss: 0.0267
2023-03-23 14:22:38,020 - mmedit - INFO - Exp name: basicvsr_plusplus_c64n7_8x1_600k_reds4
2023-03-23 14:22:38,040 - mmedit - INFO - Iter [1000/600000] lr_generator: 2.500e-05, eta: 10 days, 2:04:10, time: 1.456, data_time: 0.006, memory: 6669, loss_pix: 0.0266, loss: 0.0266
2023-03-23 14:25:03,623 - mmedit - INFO - Iter [1100/600000] lr_generator: 2.500e-05, eta: 10 days, 2:02:49, time: 1.456, data_time: 0.006, memory: 6669, loss_pix: 0.0273, loss: 0.0273
2023-03-23 14:27:29,236 - mmedit - INFO - Iter [1200/600000] lr_generator: 2.500e-05, eta: 10 days, 2:01:22, time: 1.456, data_time: 0.006, memory: 6669, loss_pix: 0.0274, loss: 0.0274
2023-03-23 14:29:54,421 - mmedit - INFO - Iter [1300/600000] lr_generator: 2.500e-05, eta: 10 days, 1:56:28, time: 1.452, data_time: 0.006, memory: 6669, loss_pix: 0.0267, loss: 0.0267
2023-03-23 14:32:21,695 - mmedit - INFO - Iter [1400/600000] lr_generator: 2.500e-05, eta: 10 days, 2:06:49, time: 1.473, data_time: 0.006, memory: 6669, loss_pix: 0.0257, loss: 0.0257
2023-03-23 14:34:48,001 - mmedit - INFO - Iter [1500/600000] lr_generator: 2.500e-05, eta: 10 days, 2:09:01, time: 1.463, data_time: 0.006, memory: 6669, loss_pix: 0.0241, loss: 0.0241
2023-03-23 14:37:16,271 - mmedit - INFO - Iter [1600/600000] lr_generator: 2.500e-05, eta: 10 days, 2:22:53, time: 1.483, data_time: 0.006, memory: 6669, loss_pix: 0.0248, loss: 0.0248
2023-03-23 14:39:41,826 - mmedit - INFO - Iter [1700/600000] lr_generator: 2.500e-05, eta: 10 days, 2:18:54, time: 1.456, data_time: 0.006, memory: 6669, loss_pix: 0.0243, loss: 0.0243
2023-03-23 14:42:07,665 - mmedit - INFO - Iter [1800/600000] lr_generator: 2.500e-05, eta: 10 days, 2:16:40, time: 1.458, data_time: 0.006, memory: 6669, loss_pix: 0.0252, loss: 0.0252
2023-03-23 14:44:33,763 - mmedit - INFO - Iter [1900/600000] lr_generator: 2.500e-05, eta: 10 days, 2:15:45, time: 1.461, data_time: 0.006, memory: 6669, loss_pix: 0.0248, loss: 0.0248
2023-03-23 14:47:00,083 - mmedit - INFO - Exp name: basicvsr_plusplus_c64n7_8x1_600k_reds4
2023-03-23 14:47:00,100 - mmedit - INFO - Iter [2000/600000] lr_generator: 2.500e-05, eta: 10 days, 2:15:49, time: 1.463, data_time: 0.006, memory: 6669, loss_pix: 0.0283, loss: 0.0283
2023-03-23 14:49:26,376 - mmedit - INFO - Iter [2100/600000] lr_generator: 2.500e-05, eta: 10 days, 2:15:31, time: 1.463, data_time: 0.006, memory: 6669, loss_pix: 0.0235, loss: 0.0235
2023-03-23 14:51:52,432 - mmedit - INFO - Iter [2200/600000] lr_generator: 2.500e-05, eta: 10 days, 2:13:57, time: 1.461, data_time: 0.005, memory: 6669, loss_pix: 0.0237, loss: 0.0237
2023-03-23 14:54:19,391 - mmedit - INFO - Iter [2300/600000] lr_generator: 2.500e-05, eta: 10 days, 2:16:11, time: 1.470, data_time: 0.006, memory: 6669, loss_pix: 0.0248, loss: 0.0248
2023-03-23 14:56:46,144 - mmedit - INFO - Iter [2400/600000] lr_generator: 2.500e-05, eta: 10 days, 2:17:13, time: 1.468, data_time: 0.006, memory: 6669, loss_pix: 0.0234, loss: 0.0234
2023-03-23 14:59:12,845 - mmedit - INFO - Iter [2500/600000] lr_generator: 2.500e-05, eta: 10 days, 2:17:45, time: 1.467, data_time: 0.006, memory: 6669, loss_pix: 0.0252, loss: 0.0252
2023-03-23 15:01:39,345 - mmedit - INFO - Iter [2600/600000] lr_generator: 2.500e-05, eta: 10 days, 2:17:17, time: 1.465, data_time: 0.006, memory: 6669, loss_pix: 0.0256, loss: 0.0256
2023-03-23 15:04:06,310 - mmedit - INFO - Iter [2700/600000] lr_generator: 2.500e-05, eta: 10 days, 2:18:23, time: 1.470, data_time: 0.006, memory: 6669, loss_pix: 0.0250, loss: 0.0250
2023-03-23 15:06:33,146 - mmedit - INFO - Iter [2800/600000] lr_generator: 2.500e-05, eta: 10 days, 2:18:46, time: 1.468, data_time: 0.005, memory: 6669, loss_pix: 0.0242, loss: 0.0242
2023-03-23 15:09:00,508 - mmedit - INFO - Iter [2900/600000] lr_generator: 2.500e-05, eta: 10 days, 2:20:46, time: 1.474, data_time: 0.006, memory: 6669, loss_pix: 0.0240, loss: 0.0240
2023-03-23 15:11:29,045 - mmedit - INFO - Exp name: basicvsr_plusplus_c64n7_8x1_600k_reds4
2023-03-23 15:11:29,068 - mmedit - INFO - Iter [3000/600000] lr_generator: 2.500e-05, eta: 10 days, 2:26:20, time: 1.485, data_time: 0.006, memory: 6669, loss_pix: 0.0226, loss: 0.0226
2023-03-23 15:13:57,090 - mmedit - INFO - Iter [3100/600000] lr_generator: 2.500e-05, eta: 10 days, 2:29:52, time: 1.481, data_time: 0.007, memory: 6669, loss_pix: 0.0263, loss: 0.0263
2023-03-23 15:16:24,394 - mmedit - INFO - Iter [3200/600000] lr_generator: 2.500e-05, eta: 10 days, 2:30:41, time: 1.473, data_time: 0.006, memory: 6669, loss_pix: 0.0241, loss: 0.0241
2023-03-23 15:18:51,517 - mmedit - INFO - Iter [3300/600000] lr_generator: 2.500e-05, eta: 10 days, 2:30:46, time: 1.471, data_time: 0.006, memory: 6669, loss_pix: 0.0240, loss: 0.0240
2023-03-23 15:21:18,861 - mmedit - INFO - Iter [3400/600000] lr_generator: 2.500e-05, eta: 10 days, 2:31:20, time: 1.473, data_time: 0.006, memory: 6669, loss_pix: 0.0246, loss: 0.0246
2023-03-23 15:23:46,229 - mmedit - INFO - Iter [3500/600000] lr_generator: 2.500e-05, eta: 10 days, 2:31:49, time: 1.474, data_time: 0.006, memory: 6669, loss_pix: 0.0236, loss: 0.0236
2023-03-23 15:26:13,204 - mmedit - INFO - Iter [3600/600000] lr_generator: 2.500e-05, eta: 10 days, 2:31:01, time: 1.470, data_time: 0.006, memory: 6669, loss_pix: 0.0262, loss: 0.0262
2023-03-23 15:28:39,843 - mmedit - INFO - Iter [3700/600000] lr_generator: 2.500e-05, eta: 10 days, 2:29:15, time: 1.466, data_time: 0.006, memory: 6669, loss_pix: 0.0265, loss: 0.0265
2023-03-23 15:31:06,529 - mmedit - INFO - Iter [3800/600000] lr_generator: 2.500e-05, eta: 10 days, 2:27:35, time: 1.467, data_time: 0.006, memory: 6669, loss_pix: 0.0249, loss: 0.0249
2023-03-23 15:33:33,486 - mmedit - INFO - Iter [3900/600000] lr_generator: 2.500e-05, eta: 10 days, 2:26:32, time: 1.470, data_time: 0.006, memory: 6669, loss_pix: 0.0255, loss: 0.0255
2023-03-23 15:36:02,465 - mmedit - INFO - Exp name: basicvsr_plusplus_c64n7_8x1_600k_reds4
2023-03-23 15:36:02,489 - mmedit - INFO - Iter [4000/600000] lr_generator: 2.500e-05, eta: 10 days, 2:30:27, time: 1.490, data_time: 0.006, memory: 6669, loss_pix: 0.0239, loss: 0.0239
2023-03-23 15:38:32,986 - mmedit - INFO - Iter [4100/600000] lr_generator: 2.500e-05, eta: 10 days, 2:37:48, time: 1.505, data_time: 0.007, memory: 6669, loss_pix: 0.0230, loss: 0.0230
2023-03-23 15:41:00,222 - mmedit - INFO - Iter [4200/600000] lr_generator: 2.500e-05, eta: 10 days, 2:36:55, time: 1.472, data_time: 0.006, memory: 6669, loss_pix: 0.0225, loss: 0.0225
2023-03-23 15:43:28,183 - mmedit - INFO - Iter [4300/600000] lr_generator: 2.500e-05, eta: 10 days, 2:37:37, time: 1.480, data_time: 0.006, memory: 6669, loss_pix: 0.0224, loss: 0.0224
2023-03-23 15:45:55,969 - mmedit - INFO - Iter [4400/600000] lr_generator: 2.500e-05, eta: 10 days, 2:37:46, time: 1.478, data_time: 0.007, memory: 6669, loss_pix: 0.0242, loss: 0.0242
2023-03-23 15:48:25,827 - mmedit - INFO - Iter [4500/600000] lr_generator: 2.500e-05, eta: 10 days, 2:42:24, time: 1.499, data_time: 0.007, memory: 6669, loss_pix: 0.0245, loss: 0.0245
2023-03-23 15:50:54,335 - mmedit - INFO - Iter [4600/600000] lr_generator: 2.500e-05, eta: 10 days, 2:43:48, time: 1.485, data_time: 0.007, memory: 6669, loss_pix: 0.0249, loss: 0.0249
2023-03-23 15:53:21,242 - mmedit - INFO - Iter [4700/600000] lr_generator: 2.500e-05, eta: 10 days, 2:41:40, time: 1.469, data_time: 0.006, memory: 6669, loss_pix: 0.0251, loss: 0.0251
2023-03-23 15:55:49,088 - mmedit - INFO - Iter [4800/600000] lr_generator: 2.500e-05, eta: 10 days, 2:41:26, time: 1.478, data_time: 0.006, memory: 6669, loss_pix: 0.0229, loss: 0.0229
2023-03-23 15:58:16,949 - mmedit - INFO - Iter [4900/600000] lr_generator: 2.500e-05, eta: 10 days, 2:41:10, time: 1.479, data_time: 0.006, memory: 6669, loss_pix: 0.0221, loss: 0.0221
2023-03-23 16:00:47,677 - mmedit - INFO - Saving checkpoint at 5000 iterations

The server I am currently using has a problem and cannot be remotely connected. This was previously running on another server. I have modified any parameters related to 5000 on the new server, but it still only trains 5000 times like this.

Libyte Apr 18, 2023
Author

By the way, Is there a method for visualizing loss in the project? 🫣

Z-Fran · 2023-04-19T06:19:09Z

Z-Fran
Apr 19, 2023
Collaborator

Please use the new version https://github.com/open-mmlab/mmediting/tree/main . I run it successfully on MMEditing v1.0.0rc7. And you can use vis_backends = [dict(type='TensorboardVisBackend')] or vis_backends = [dict(type='WandbVisBackend')] in config to visualize loss

2 replies

Libyte Apr 24, 2023
Author

Do I need to do anything else besides logging in when using wandb?

Z-Fran Apr 24, 2023
Collaborator

You need to modify the config https://github.com/open-mmlab/mmediting/blob/main/configs/_base_/default_runtime.py#L32

Z-Fran · 2023-04-19T06:27:04Z

Z-Fran
Apr 19, 2023
Collaborator

And saving checkpoint needs two minutes.

04/19 14:20:09 - mmengine - INFO - Iter(train) [   300/600000]  lr: 2.5000e-05  eta: 6 days, 16:22:50  time: 0.9356  data_time: 0.0075  memory: 6661  loss: 0.0317
04/19 14:21:42 - mmengine - INFO - Iter(train) [   400/600000]  lr: 2.5000e-05  eta: 6 days, 15:00:56  time: 0.9306  data_time: 0.0072  memory: 6661  loss: 0.0305
04/19 14:23:16 - mmengine - INFO - Iter(train) [   500/600000]  lr: 2.5000e-05  eta: 6 days, 14:18:13  time: 0.9341  data_time: 0.0075  memory: 6661  loss: 0.0261
04/19 14:23:16 - mmengine - INFO - Saving checkpoint at 500 iterations
04/19 14:25:45 - mmengine - INFO - Iter(val) [4/4]    PSNR: 27.7785  SSIM: 0.7956  data_time: 1.4361  time: 37.0986
04/19 14:25:46 - mmengine - INFO - The best checkpoint with 27.7785 PSNR at 500 iter is saved to best_PSNR_iter_500.pth.

0 replies

Libyte · 2023-04-19T06:30:22Z

Libyte
Apr 19, 2023
Author

Thank you very much, I will have a try on this branch🥰

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When I train the model use bavsic_pp backbone and REDS by given config file, always stop in the 5000th epoch. #1768

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

When I train the model use bavsic_pp backbone and REDS by given config file, always stop in the 5000th epoch. #1768

Libyte Apr 18, 2023

Replies: 4 comments · 4 replies

Z-Fran Apr 18, 2023 Collaborator

Libyte Apr 18, 2023 Author

2023-03-23 13:58:17,669 - mmedit - INFO - Environment info:

TorchVision: 0.11.3 OpenCV: 4.7.0 MMCV: 1.7.1 MMCV Compiler: GCC 9.4 MMCV CUDA Compiler: 11.6 MMEditing: 0.16.0+8de2721

model settings

model training and testing settings

dataset settings

optimizer

learning policy

remove gpu_collect=True in non distributed training

runtime settings

before_train_epoch: (VERY_HIGH ) CosineRestartLrUpdaterHook (LOW ) IterTimerHook (VERY_LOW ) TextLoggerHook

before_train_iter: (VERY_HIGH ) CosineRestartLrUpdaterHook (LOW ) IterTimerHook

after_train_iter: (NORMAL ) CheckpointHook (LOW ) IterTimerHook (LOW ) EvalIterHook (VERY_LOW ) TextLoggerHook

after_train_epoch: (NORMAL ) CheckpointHook (VERY_LOW ) TextLoggerHook

before_val_epoch: (LOW ) IterTimerHook (VERY_LOW ) TextLoggerHook

before_val_iter: (LOW ) IterTimerHook

after_val_iter: (LOW ) IterTimerHook

after_val_epoch: (VERY_LOW ) TextLoggerHook

after_run: (VERY_LOW ) TextLoggerHook

Libyte Apr 18, 2023 Author

Z-Fran Apr 19, 2023 Collaborator

Libyte Apr 24, 2023 Author

Z-Fran Apr 24, 2023 Collaborator

Z-Fran Apr 19, 2023 Collaborator

Libyte Apr 19, 2023 Author

Libyte
Apr 18, 2023

Replies: 4 comments 4 replies

Z-Fran
Apr 18, 2023
Collaborator

Libyte Apr 18, 2023
Author

TorchVision: 0.11.3
OpenCV: 4.7.0
MMCV: 1.7.1
MMCV Compiler: GCC 9.4
MMCV CUDA Compiler: 11.6
MMEditing: 0.16.0+8de2721

before_train_epoch:
(VERY_HIGH ) CosineRestartLrUpdaterHook
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook

before_train_iter:
(VERY_HIGH ) CosineRestartLrUpdaterHook
(LOW ) IterTimerHook

after_train_iter:
(NORMAL ) CheckpointHook
(LOW ) IterTimerHook
(LOW ) EvalIterHook
(VERY_LOW ) TextLoggerHook

after_train_epoch:
(NORMAL ) CheckpointHook
(VERY_LOW ) TextLoggerHook

before_val_epoch:
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook

before_val_iter:
(LOW ) IterTimerHook

after_val_iter:
(LOW ) IterTimerHook

after_val_epoch:
(VERY_LOW ) TextLoggerHook

after_run:
(VERY_LOW ) TextLoggerHook

Libyte Apr 18, 2023
Author

Z-Fran
Apr 19, 2023
Collaborator

Libyte Apr 24, 2023
Author

Z-Fran Apr 24, 2023
Collaborator

Z-Fran
Apr 19, 2023
Collaborator

Libyte
Apr 19, 2023
Author