nohup: ignoring input grep: warning: GREP_OPTIONS is deprecated; please use an alias or script [2024-07-10 02:52:28,731] [ ERROR] check_version.py:39 - Error fetching version info Traceback (most recent call last): File "/root/anaconda3/envs/paddleocr/lib/python3.10/site-packages/albumentations/check_version.py", line 32, in fetch_version_info data = response.read() File "/root/anaconda3/envs/paddleocr/lib/python3.10/http/client.py", line 480, in read s = self._safe_read(self.length) File "/root/anaconda3/envs/paddleocr/lib/python3.10/http/client.py", line 629, in _safe_read data = self.fp.read(amt) File "/root/anaconda3/envs/paddleocr/lib/python3.10/socket.py", line 705, in readinto return self._sock.recv_into(b) File "/root/anaconda3/envs/paddleocr/lib/python3.10/ssl.py", line 1273, in recv_into return self.read(nbytes, buffer) File "/root/anaconda3/envs/paddleocr/lib/python3.10/ssl.py", line 1129, in read return self._sslobj.read(len, buffer) TimeoutError: The read operation timed out [2024/07/10 02:52:28] ppocr INFO: Architecture : [2024/07/10 02:52:28] ppocr INFO: Backbone : [2024/07/10 02:52:28] ppocr INFO: depth : 4 [2024/07/10 02:52:28] ppocr INFO: embed_dim : 256 [2024/07/10 02:52:28] ppocr INFO: img_size : [192, 672] [2024/07/10 02:52:28] ppocr INFO: input_channel : 1 [2024/07/10 02:52:28] ppocr INFO: name : HybridTransformer [2024/07/10 02:52:28] ppocr INFO: num_classes : 0 [2024/07/10 02:52:28] ppocr INFO: num_heads : 8 [2024/07/10 02:52:28] ppocr INFO: patch_size : 16 [2024/07/10 02:52:28] ppocr INFO: Head : [2024/07/10 02:52:28] ppocr INFO: decoder_args : [2024/07/10 02:52:28] ppocr INFO: attn_on_attn : True [2024/07/10 02:52:28] ppocr INFO: cross_attend : True [2024/07/10 02:52:28] ppocr INFO: ff_glu : True [2024/07/10 02:52:28] ppocr INFO: rel_pos_bias : False [2024/07/10 02:52:28] ppocr INFO: use_scalenorm : False [2024/07/10 02:52:28] ppocr INFO: ignore_index : -100 [2024/07/10 02:52:28] ppocr INFO: name : LaTeXOCRHead [2024/07/10 02:52:28] ppocr INFO: pad_value : 0 [2024/07/10 02:52:28] ppocr INFO: Transform : None [2024/07/10 02:52:28] ppocr INFO: algorithm : LaTeXOCR [2024/07/10 02:52:28] ppocr INFO: in_channels : 1 [2024/07/10 02:52:28] ppocr INFO: model_type : rec [2024/07/10 02:52:28] ppocr INFO: Eval : [2024/07/10 02:52:28] ppocr INFO: dataset : [2024/07/10 02:52:28] ppocr INFO: batch_size_per_pair : 10 [2024/07/10 02:52:28] ppocr INFO: data : /workspace/code/paddle_ocr/mycode/pickle_file_paddle/latexocr_val.pkl [2024/07/10 02:52:28] ppocr INFO: keep_smaller_batches : True [2024/07/10 02:52:28] ppocr INFO: max_dimensions : [672, 192] [2024/07/10 02:52:28] ppocr INFO: min_dimensions : [32, 32] [2024/07/10 02:52:28] ppocr INFO: name : LaTeXOCRDataSet [2024/07/10 02:52:28] ppocr INFO: transforms : [2024/07/10 02:52:28] ppocr INFO: DecodeImage : [2024/07/10 02:52:28] ppocr INFO: channel_first : False [2024/07/10 02:52:28] ppocr INFO: LatexTestTransform : None [2024/07/10 02:52:28] ppocr INFO: NormalizeImage : [2024/07/10 02:52:28] ppocr INFO: mean : [0.7931, 0.7931, 0.7931] [2024/07/10 02:52:28] ppocr INFO: order : hwc [2024/07/10 02:52:28] ppocr INFO: std : [0.1738, 0.1738, 0.1738] [2024/07/10 02:52:28] ppocr INFO: LatexImageFormat : None [2024/07/10 02:52:28] ppocr INFO: KeepKeys : [2024/07/10 02:52:28] ppocr INFO: keep_keys : ['image'] [2024/07/10 02:52:28] ppocr INFO: loader : [2024/07/10 02:52:28] ppocr INFO: batch_size_per_card : 1 [2024/07/10 02:52:28] ppocr INFO: collate_fn : LaTeXOCRCollator [2024/07/10 02:52:28] ppocr INFO: drop_last : False [2024/07/10 02:52:28] ppocr INFO: num_workers : 0 [2024/07/10 02:52:28] ppocr INFO: shuffle : False [2024/07/10 02:52:28] ppocr INFO: Global : [2024/07/10 02:52:28] ppocr INFO: cal_metric_during_train : True [2024/07/10 02:52:28] ppocr INFO: checkpoints : /workspace/code/paddle_ocr/PaddleOCR-main/output/rec/latex_ocr/iter_epoch_300 [2024/07/10 02:52:28] ppocr INFO: distributed : False [2024/07/10 02:52:28] ppocr INFO: epoch_num : 500 [2024/07/10 02:52:28] ppocr INFO: eval_batch_step : [0, 300000000000] [2024/07/10 02:52:28] ppocr INFO: fast_tokenizer_file : ppocr/utils/dict/latex_ocr_tokenizer.json [2024/07/10 02:52:28] ppocr INFO: infer_img : doc/datasets/crohme_demo/hme_00.jpg [2024/07/10 02:52:28] ppocr INFO: infer_mode : False [2024/07/10 02:52:28] ppocr INFO: log_smooth_window : 20 [2024/07/10 02:52:28] ppocr INFO: max_seq_len : 512 [2024/07/10 02:52:28] ppocr INFO: pretrained_model : None [2024/07/10 02:52:28] ppocr INFO: print_batch_step : 100 [2024/07/10 02:52:28] ppocr INFO: save_epoch_step : 5 [2024/07/10 02:52:28] ppocr INFO: save_inference_dir : None [2024/07/10 02:52:28] ppocr INFO: save_model_dir : ./output/rec/latex_ocr/ [2024/07/10 02:52:28] ppocr INFO: save_res_path : ./output/rec/predicts_latexocr.txt [2024/07/10 02:52:28] ppocr INFO: use_gpu : True [2024/07/10 02:52:28] ppocr INFO: use_space_char : False [2024/07/10 02:52:28] ppocr INFO: use_visualdl : False [2024/07/10 02:52:28] ppocr INFO: Loss : [2024/07/10 02:52:28] ppocr INFO: name : LaTeXOCRLoss [2024/07/10 02:52:28] ppocr INFO: Metric : [2024/07/10 02:52:28] ppocr INFO: cal_blue_score : False [2024/07/10 02:52:28] ppocr INFO: main_indicator : exp_rate [2024/07/10 02:52:28] ppocr INFO: name : LaTeXOCRMetric [2024/07/10 02:52:28] ppocr INFO: Optimizer : [2024/07/10 02:52:28] ppocr INFO: beta1 : 0.9 [2024/07/10 02:52:28] ppocr INFO: beta2 : 0.999 [2024/07/10 02:52:28] ppocr INFO: lr : [2024/07/10 02:52:28] ppocr INFO: learning_rate : 0.0001 [2024/07/10 02:52:28] ppocr INFO: name : Const [2024/07/10 02:52:28] ppocr INFO: name : AdamW [2024/07/10 02:52:28] ppocr INFO: PostProcess : [2024/07/10 02:52:28] ppocr INFO: fast_tokenizer_file : ppocr/utils/dict/latex_ocr_tokenizer.json [2024/07/10 02:52:28] ppocr INFO: name : LaTeXOCRDecode [2024/07/10 02:52:28] ppocr INFO: Train : [2024/07/10 02:52:28] ppocr INFO: dataset : [2024/07/10 02:52:28] ppocr INFO: batch_size_per_pair : 56 [2024/07/10 02:52:28] ppocr INFO: data : /workspace/code/paddle_ocr/mycode/pickle_file_paddle/latexocr_train.pkl [2024/07/10 02:52:28] ppocr INFO: keep_smaller_batches : False [2024/07/10 02:52:28] ppocr INFO: max_dimensions : [672, 192] [2024/07/10 02:52:28] ppocr INFO: min_dimensions : [32, 32] [2024/07/10 02:52:28] ppocr INFO: name : LaTeXOCRDataSet [2024/07/10 02:52:28] ppocr INFO: transforms : [2024/07/10 02:52:28] ppocr INFO: DecodeImage : [2024/07/10 02:52:28] ppocr INFO: channel_first : False [2024/07/10 02:52:28] ppocr INFO: LatexTrainTransform : [2024/07/10 02:52:28] ppocr INFO: bitmap_prob : 0.04 [2024/07/10 02:52:28] ppocr INFO: NormalizeImage : [2024/07/10 02:52:28] ppocr INFO: mean : [0.7931, 0.7931, 0.7931] [2024/07/10 02:52:28] ppocr INFO: order : hwc [2024/07/10 02:52:28] ppocr INFO: std : [0.1738, 0.1738, 0.1738] [2024/07/10 02:52:28] ppocr INFO: LatexImageFormat : None [2024/07/10 02:52:28] ppocr INFO: KeepKeys : [2024/07/10 02:52:28] ppocr INFO: keep_keys : ['image'] [2024/07/10 02:52:28] ppocr INFO: loader : [2024/07/10 02:52:28] ppocr INFO: batch_size_per_card : 1 [2024/07/10 02:52:28] ppocr INFO: collate_fn : LaTeXOCRCollator [2024/07/10 02:52:28] ppocr INFO: drop_last : False [2024/07/10 02:52:28] ppocr INFO: num_workers : 0 [2024/07/10 02:52:28] ppocr INFO: shuffle : True [2024/07/10 02:52:28] ppocr INFO: profiler_options : None [2024/07/10 02:52:28] ppocr INFO: train with paddle 2.6.1 and device Place(gpu:0) W0710 02:52:28.885243 4724 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 12.0, Runtime API Version: 11.7 W0710 02:52:28.893407 4724 gpu_resources.cc:164] device: 0, cuDNN Version: 8.5. load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range load_config {'shuffle': False, 'drop_last': False, 'batch_size_per_card': 1, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} out_channel 256 [2024/07/10 02:52:32] ppocr INFO: train dataloader has 2740 iters [2024/07/10 02:52:32] ppocr INFO: valid dataloader has 716 iters [2024/07/10 02:52:33] ppocr INFO: resume from /workspace/code/paddle_ocr/PaddleOCR-main/output/rec/latex_ocr/iter_epoch_300 [2024/07/10 02:52:33] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 300000000000 iterations load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 02:52:55] ppocr INFO: epoch: [301/500], global_step: 100, lr: 0.000100, edit distance: 0.001450, exp_rate: 0.930357, exp_rate<=1 : 0.973164, exp_rate<=2 : 0.991808, exp_rate<=3 : 0.995317, loss: 0.001580, avg_reader_cost: 0.00938 s, avg_batch_cost: 0.22429 s, avg_samples: 56.0, ips: 249.68207 samples/s, eta: 1 day, 10:08:05, max_mem_reserved: 25027 MB, max_mem_allocated: 18324 MB [2024/07/10 02:53:14] ppocr INFO: epoch: [301/500], global_step: 200, lr: 0.000100, edit distance: 0.001506, exp_rate: 0.930628, exp_rate<=1 : 0.973238, exp_rate<=2 : 0.992703, exp_rate<=3 : 0.996544, loss: 0.001449, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19509 s, avg_samples: 56.0, ips: 287.04932 samples/s, eta: 1 day, 7:54:26, max_mem_reserved: 29087 MB, max_mem_allocated: 18324 MB [2024/07/10 02:53:35] ppocr INFO: epoch: [301/500], global_step: 300, lr: 0.000100, edit distance: 0.001481, exp_rate: 0.932144, exp_rate<=1 : 0.974946, exp_rate<=2 : 0.993493, exp_rate<=3 : 0.996998, loss: 0.002201, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20028 s, avg_samples: 56.0, ips: 279.60776 samples/s, eta: 1 day, 7:25:28, max_mem_reserved: 29087 MB, max_mem_allocated: 18324 MB [2024/07/10 02:53:53] ppocr INFO: epoch: [301/500], global_step: 400, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.932898, exp_rate<=1 : 0.975041, exp_rate<=2 : 0.993347, exp_rate<=3 : 0.996662, loss: 0.001606, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18359 s, avg_samples: 56.0, ips: 305.02453 samples/s, eta: 1 day, 6:32:44, max_mem_reserved: 29087 MB, max_mem_allocated: 18324 MB [2024/07/10 02:54:12] ppocr INFO: epoch: [301/500], global_step: 500, lr: 0.000100, edit distance: 0.001539, exp_rate: 0.931517, exp_rate<=1 : 0.974366, exp_rate<=2 : 0.993529, exp_rate<=3 : 0.996905, loss: 0.002338, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19367 s, avg_samples: 56.0, ips: 289.15393 samples/s, eta: 1 day, 6:19:22, max_mem_reserved: 29087 MB, max_mem_allocated: 18324 MB [2024/07/10 02:54:33] ppocr INFO: epoch: [301/500], global_step: 600, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.930313, exp_rate<=1 : 0.973304, exp_rate<=2 : 0.993328, exp_rate<=3 : 0.997090, loss: 0.001398, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20572 s, avg_samples: 56.0, ips: 272.21116 samples/s, eta: 1 day, 6:28:40, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:54:52] ppocr INFO: epoch: [301/500], global_step: 700, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.931306, exp_rate<=1 : 0.973441, exp_rate<=2 : 0.993503, exp_rate<=3 : 0.997233, loss: 0.001778, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18652 s, avg_samples: 56.0, ips: 300.23498 samples/s, eta: 1 day, 6:10:12, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:55:11] ppocr INFO: epoch: [301/500], global_step: 800, lr: 0.000100, edit distance: 0.001550, exp_rate: 0.929619, exp_rate<=1 : 0.972720, exp_rate<=2 : 0.993367, exp_rate<=3 : 0.997143, loss: 0.001266, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19306 s, avg_samples: 56.0, ips: 290.06386 samples/s, eta: 1 day, 6:03:43, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:55:31] ppocr INFO: epoch: [301/500], global_step: 900, lr: 0.000100, edit distance: 0.001678, exp_rate: 0.925024, exp_rate<=1 : 0.970095, exp_rate<=2 : 0.992130, exp_rate<=3 : 0.996427, loss: 0.002934, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19601 s, avg_samples: 56.0, ips: 285.70304 samples/s, eta: 1 day, 6:01:36, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:55:49] ppocr INFO: epoch: [301/500], global_step: 1000, lr: 0.000100, edit distance: 0.001711, exp_rate: 0.922874, exp_rate<=1 : 0.968838, exp_rate<=2 : 0.991491, exp_rate<=3 : 0.996142, loss: 0.002824, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18714 s, avg_samples: 56.0, ips: 299.24664 samples/s, eta: 1 day, 5:51:45, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:56:09] ppocr INFO: epoch: [301/500], global_step: 1100, lr: 0.000100, edit distance: 0.001719, exp_rate: 0.921920, exp_rate<=1 : 0.968563, exp_rate<=2 : 0.991416, exp_rate<=3 : 0.996129, loss: 0.002475, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19484 s, avg_samples: 56.0, ips: 287.41448 samples/s, eta: 1 day, 5:50:01, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:56:26] ppocr INFO: epoch: [301/500], global_step: 1200, lr: 0.000100, edit distance: 0.001721, exp_rate: 0.923040, exp_rate<=1 : 0.969102, exp_rate<=2 : 0.991647, exp_rate<=3 : 0.996279, loss: 0.001748, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17128 s, avg_samples: 56.0, ips: 326.95635 samples/s, eta: 1 day, 5:30:37, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:56:46] ppocr INFO: epoch: [301/500], global_step: 1300, lr: 0.000100, edit distance: 0.001703, exp_rate: 0.923338, exp_rate<=1 : 0.969118, exp_rate<=2 : 0.991684, exp_rate<=3 : 0.996347, loss: 0.001672, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19774 s, avg_samples: 56.0, ips: 283.19821 samples/s, eta: 1 day, 5:32:43, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:57:04] ppocr INFO: epoch: [301/500], global_step: 1400, lr: 0.000100, edit distance: 0.001677, exp_rate: 0.924456, exp_rate<=1 : 0.969798, exp_rate<=2 : 0.991872, exp_rate<=3 : 0.996448, loss: 0.001762, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17886 s, avg_samples: 56.0, ips: 313.09969 samples/s, eta: 1 day, 5:22:11, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:57:22] ppocr INFO: epoch: [301/500], global_step: 1500, lr: 0.000100, edit distance: 0.001662, exp_rate: 0.924668, exp_rate<=1 : 0.969971, exp_rate<=2 : 0.992119, exp_rate<=3 : 0.996562, loss: 0.001965, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18518 s, avg_samples: 56.0, ips: 302.40330 samples/s, eta: 1 day, 5:16:51, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:57:42] ppocr INFO: epoch: [301/500], global_step: 1600, lr: 0.000100, edit distance: 0.001664, exp_rate: 0.924468, exp_rate<=1 : 0.969730, exp_rate<=2 : 0.992065, exp_rate<=3 : 0.996576, loss: 0.002270, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20016 s, avg_samples: 56.0, ips: 279.77635 samples/s, eta: 1 day, 5:20:40, max_mem_reserved: 30359 MB, max_mem_allocated: 23149 MB [2024/07/10 02:58:01] ppocr INFO: epoch: [301/500], global_step: 1700, lr: 0.000100, edit distance: 0.001683, exp_rate: 0.924164, exp_rate<=1 : 0.969490, exp_rate<=2 : 0.992034, exp_rate<=3 : 0.996556, loss: 0.002226, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19198 s, avg_samples: 56.0, ips: 291.69053 samples/s, eta: 1 day, 5:19:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 02:58:19] ppocr INFO: epoch: [301/500], global_step: 1800, lr: 0.000100, edit distance: 0.001680, exp_rate: 0.924572, exp_rate<=1 : 0.969596, exp_rate<=2 : 0.992089, exp_rate<=3 : 0.996624, loss: 0.001492, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17796 s, avg_samples: 56.0, ips: 314.68413 samples/s, eta: 1 day, 5:11:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 02:58:38] ppocr INFO: epoch: [301/500], global_step: 1900, lr: 0.000100, edit distance: 0.001673, exp_rate: 0.924915, exp_rate<=1 : 0.969539, exp_rate<=2 : 0.992063, exp_rate<=3 : 0.996612, loss: 0.001377, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18602 s, avg_samples: 56.0, ips: 301.03706 samples/s, eta: 1 day, 5:08:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 02:58:56] ppocr INFO: epoch: [301/500], global_step: 2000, lr: 0.000100, edit distance: 0.001679, exp_rate: 0.925063, exp_rate<=1 : 0.969672, exp_rate<=2 : 0.992111, exp_rate<=3 : 0.996704, loss: 0.002421, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17874 s, avg_samples: 56.0, ips: 313.29848 samples/s, eta: 1 day, 5:01:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 02:59:14] ppocr INFO: epoch: [301/500], global_step: 2100, lr: 0.000100, edit distance: 0.001683, exp_rate: 0.925175, exp_rate<=1 : 0.969824, exp_rate<=2 : 0.992170, exp_rate<=3 : 0.996754, loss: 0.001689, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18455 s, avg_samples: 56.0, ips: 303.44739 samples/s, eta: 1 day, 4:58:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 02:59:36] ppocr INFO: epoch: [301/500], global_step: 2200, lr: 0.000100, edit distance: 0.001686, exp_rate: 0.924405, exp_rate<=1 : 0.969251, exp_rate<=2 : 0.992000, exp_rate<=3 : 0.996659, loss: 0.002084, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21357 s, avg_samples: 56.0, ips: 262.21134 samples/s, eta: 1 day, 5:07:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 02:59:54] ppocr INFO: epoch: [301/500], global_step: 2300, lr: 0.000100, edit distance: 0.001674, exp_rate: 0.924640, exp_rate<=1 : 0.969479, exp_rate<=2 : 0.992126, exp_rate<=3 : 0.996745, loss: 0.001726, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18720 s, avg_samples: 56.0, ips: 299.14289 samples/s, eta: 1 day, 5:05:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:00:14] ppocr INFO: epoch: [301/500], global_step: 2400, lr: 0.000100, edit distance: 0.001669, exp_rate: 0.924712, exp_rate<=1 : 0.969453, exp_rate<=2 : 0.992094, exp_rate<=3 : 0.996678, loss: 0.002449, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19694 s, avg_samples: 56.0, ips: 284.35571 samples/s, eta: 1 day, 5:06:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:00:34] ppocr INFO: epoch: [301/500], global_step: 2500, lr: 0.000100, edit distance: 0.001676, exp_rate: 0.924259, exp_rate<=1 : 0.969356, exp_rate<=2 : 0.992063, exp_rate<=3 : 0.996681, loss: 0.002003, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20247 s, avg_samples: 56.0, ips: 276.58396 samples/s, eta: 1 day, 5:10:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:00:53] ppocr INFO: epoch: [301/500], global_step: 2600, lr: 0.000100, edit distance: 0.001682, exp_rate: 0.924130, exp_rate<=1 : 0.969346, exp_rate<=2 : 0.992149, exp_rate<=3 : 0.996757, loss: 0.001795, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18901 s, avg_samples: 56.0, ips: 296.27595 samples/s, eta: 1 day, 5:08:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:01:12] ppocr INFO: epoch: [301/500], global_step: 2700, lr: 0.000100, edit distance: 0.001681, exp_rate: 0.924243, exp_rate<=1 : 0.969320, exp_rate<=2 : 0.992206, exp_rate<=3 : 0.996813, loss: 0.001759, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19107 s, avg_samples: 56.0, ips: 293.08214 samples/s, eta: 1 day, 5:07:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:01:20] ppocr INFO: epoch: [301/500], global_step: 2740, lr: 0.000100, edit distance: 0.001686, exp_rate: 0.924072, exp_rate<=1 : 0.969293, exp_rate<=2 : 0.992248, exp_rate<=3 : 0.996853, loss: 0.002424, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.08104 s, avg_samples: 22.4, ips: 276.40089 samples/s, eta: 1 day, 5:09:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:01:23] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 03:01:36] ppocr INFO: epoch: [302/500], global_step: 2800, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.936566, exp_rate<=1 : 0.972240, exp_rate<=2 : 0.994328, exp_rate<=3 : 0.999215, loss: 0.001484, avg_reader_cost: 0.03018 s, avg_batch_cost: 0.15884 s, avg_samples: 33.6, ips: 211.52740 samples/s, eta: 1 day, 5:23:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:01:56] ppocr INFO: epoch: [302/500], global_step: 2900, lr: 0.000100, edit distance: 0.001511, exp_rate: 0.927694, exp_rate<=1 : 0.970357, exp_rate<=2 : 0.991813, exp_rate<=3 : 0.997917, loss: 0.001848, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20084 s, avg_samples: 56.0, ips: 278.83107 samples/s, eta: 1 day, 5:24:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:02:16] ppocr INFO: epoch: [302/500], global_step: 3000, lr: 0.000100, edit distance: 0.001555, exp_rate: 0.927784, exp_rate<=1 : 0.971238, exp_rate<=2 : 0.992226, exp_rate<=3 : 0.997692, loss: 0.002270, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19478 s, avg_samples: 56.0, ips: 287.49711 samples/s, eta: 1 day, 5:24:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:02:36] ppocr INFO: epoch: [302/500], global_step: 3100, lr: 0.000100, edit distance: 0.001614, exp_rate: 0.926287, exp_rate<=1 : 0.970584, exp_rate<=2 : 0.992106, exp_rate<=3 : 0.997395, loss: 0.002378, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19539 s, avg_samples: 56.0, ips: 286.61141 samples/s, eta: 1 day, 5:24:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:02:55] ppocr INFO: epoch: [302/500], global_step: 3200, lr: 0.000100, edit distance: 0.001616, exp_rate: 0.929572, exp_rate<=1 : 0.972488, exp_rate<=2 : 0.992652, exp_rate<=3 : 0.997560, loss: 0.002250, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18954 s, avg_samples: 56.0, ips: 295.45770 samples/s, eta: 1 day, 5:23:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:03:14] ppocr INFO: epoch: [302/500], global_step: 3300, lr: 0.000100, edit distance: 0.001628, exp_rate: 0.928946, exp_rate<=1 : 0.972436, exp_rate<=2 : 0.992613, exp_rate<=3 : 0.997500, loss: 0.001873, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19756 s, avg_samples: 56.0, ips: 283.45463 samples/s, eta: 1 day, 5:23:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:03:35] ppocr INFO: epoch: [302/500], global_step: 3400, lr: 0.000100, edit distance: 0.001646, exp_rate: 0.926715, exp_rate<=1 : 0.971648, exp_rate<=2 : 0.992363, exp_rate<=3 : 0.997285, loss: 0.001286, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21104 s, avg_samples: 56.0, ips: 265.34759 samples/s, eta: 1 day, 5:27:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:03:55] ppocr INFO: epoch: [302/500], global_step: 3500, lr: 0.000100, edit distance: 0.001626, exp_rate: 0.927106, exp_rate<=1 : 0.971541, exp_rate<=2 : 0.992252, exp_rate<=3 : 0.997261, loss: 0.001802, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19772 s, avg_samples: 56.0, ips: 283.22779 samples/s, eta: 1 day, 5:28:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:04:14] ppocr INFO: epoch: [302/500], global_step: 3600, lr: 0.000100, edit distance: 0.001583, exp_rate: 0.928813, exp_rate<=1 : 0.972364, exp_rate<=2 : 0.992633, exp_rate<=3 : 0.997516, loss: 0.001882, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18633 s, avg_samples: 56.0, ips: 300.54528 samples/s, eta: 1 day, 5:25:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:04:34] ppocr INFO: epoch: [302/500], global_step: 3700, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.928948, exp_rate<=1 : 0.972374, exp_rate<=2 : 0.992617, exp_rate<=3 : 0.997386, loss: 0.002057, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20227 s, avg_samples: 56.0, ips: 276.85696 samples/s, eta: 1 day, 5:27:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:04:54] ppocr INFO: epoch: [302/500], global_step: 3800, lr: 0.000100, edit distance: 0.001542, exp_rate: 0.929317, exp_rate<=1 : 0.972506, exp_rate<=2 : 0.992615, exp_rate<=3 : 0.997379, loss: 0.001756, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19869 s, avg_samples: 56.0, ips: 281.85298 samples/s, eta: 1 day, 5:27:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:05:14] ppocr INFO: epoch: [302/500], global_step: 3900, lr: 0.000100, edit distance: 0.001546, exp_rate: 0.929676, exp_rate<=1 : 0.972513, exp_rate<=2 : 0.992761, exp_rate<=3 : 0.997408, loss: 0.001701, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19980 s, avg_samples: 56.0, ips: 280.28270 samples/s, eta: 1 day, 5:28:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:05:32] ppocr INFO: epoch: [302/500], global_step: 4000, lr: 0.000100, edit distance: 0.001555, exp_rate: 0.930023, exp_rate<=1 : 0.972579, exp_rate<=2 : 0.992747, exp_rate<=3 : 0.997448, loss: 0.001910, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18392 s, avg_samples: 56.0, ips: 304.47382 samples/s, eta: 1 day, 5:25:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:05:50] ppocr INFO: epoch: [302/500], global_step: 4100, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.930332, exp_rate<=1 : 0.972737, exp_rate<=2 : 0.992870, exp_rate<=3 : 0.997446, loss: 0.001351, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17754 s, avg_samples: 56.0, ips: 315.41839 samples/s, eta: 1 day, 5:21:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:06:09] ppocr INFO: epoch: [302/500], global_step: 4200, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.930147, exp_rate<=1 : 0.972719, exp_rate<=2 : 0.992830, exp_rate<=3 : 0.997474, loss: 0.002117, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19082 s, avg_samples: 56.0, ips: 293.46423 samples/s, eta: 1 day, 5:20:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:06:29] ppocr INFO: epoch: [302/500], global_step: 4300, lr: 0.000100, edit distance: 0.001570, exp_rate: 0.929983, exp_rate<=1 : 0.972453, exp_rate<=2 : 0.992899, exp_rate<=3 : 0.997484, loss: 0.002044, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19202 s, avg_samples: 56.0, ips: 291.63978 samples/s, eta: 1 day, 5:19:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:06:48] ppocr INFO: epoch: [302/500], global_step: 4400, lr: 0.000100, edit distance: 0.001559, exp_rate: 0.929923, exp_rate<=1 : 0.972463, exp_rate<=2 : 0.992903, exp_rate<=3 : 0.997537, loss: 0.001605, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19121 s, avg_samples: 56.0, ips: 292.86653 samples/s, eta: 1 day, 5:18:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:07:06] ppocr INFO: epoch: [302/500], global_step: 4500, lr: 0.000100, edit distance: 0.001574, exp_rate: 0.930081, exp_rate<=1 : 0.972641, exp_rate<=2 : 0.992911, exp_rate<=3 : 0.997514, loss: 0.001919, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18317 s, avg_samples: 56.0, ips: 305.73286 samples/s, eta: 1 day, 5:16:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:07:25] ppocr INFO: epoch: [302/500], global_step: 4600, lr: 0.000100, edit distance: 0.001597, exp_rate: 0.929290, exp_rate<=1 : 0.972269, exp_rate<=2 : 0.992836, exp_rate<=3 : 0.997484, loss: 0.001864, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18969 s, avg_samples: 56.0, ips: 295.22487 samples/s, eta: 1 day, 5:15:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:07:45] ppocr INFO: epoch: [302/500], global_step: 4700, lr: 0.000100, edit distance: 0.001597, exp_rate: 0.928887, exp_rate<=1 : 0.972086, exp_rate<=2 : 0.992776, exp_rate<=3 : 0.997463, loss: 0.001805, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19684 s, avg_samples: 56.0, ips: 284.49184 samples/s, eta: 1 day, 5:15:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:08:02] ppocr INFO: epoch: [302/500], global_step: 4800, lr: 0.000100, edit distance: 0.001601, exp_rate: 0.929216, exp_rate<=1 : 0.972228, exp_rate<=2 : 0.992880, exp_rate<=3 : 0.997498, loss: 0.002020, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17192 s, avg_samples: 56.0, ips: 325.72618 samples/s, eta: 1 day, 5:11:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:08:21] ppocr INFO: epoch: [302/500], global_step: 4900, lr: 0.000100, edit distance: 0.001620, exp_rate: 0.928880, exp_rate<=1 : 0.971990, exp_rate<=2 : 0.992818, exp_rate<=3 : 0.997424, loss: 0.002230, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18679 s, avg_samples: 56.0, ips: 299.80229 samples/s, eta: 1 day, 5:09:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:08:38] ppocr INFO: epoch: [302/500], global_step: 5000, lr: 0.000100, edit distance: 0.001628, exp_rate: 0.928873, exp_rate<=1 : 0.971994, exp_rate<=2 : 0.992942, exp_rate<=3 : 0.997469, loss: 0.001538, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17562 s, avg_samples: 56.0, ips: 318.87191 samples/s, eta: 1 day, 5:05:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:08:58] ppocr INFO: epoch: [302/500], global_step: 5100, lr: 0.000100, edit distance: 0.001631, exp_rate: 0.928423, exp_rate<=1 : 0.971913, exp_rate<=2 : 0.992990, exp_rate<=3 : 0.997523, loss: 0.002569, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19906 s, avg_samples: 56.0, ips: 281.32791 samples/s, eta: 1 day, 5:06:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:09:17] ppocr INFO: epoch: [302/500], global_step: 5200, lr: 0.000100, edit distance: 0.001634, exp_rate: 0.928389, exp_rate<=1 : 0.971911, exp_rate<=2 : 0.993044, exp_rate<=3 : 0.997550, loss: 0.001662, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18481 s, avg_samples: 56.0, ips: 303.01943 samples/s, eta: 1 day, 5:04:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:09:35] ppocr INFO: epoch: [302/500], global_step: 5300, lr: 0.000100, edit distance: 0.001650, exp_rate: 0.928214, exp_rate<=1 : 0.971867, exp_rate<=2 : 0.993048, exp_rate<=3 : 0.997542, loss: 0.002152, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18222 s, avg_samples: 56.0, ips: 307.31884 samples/s, eta: 1 day, 5:02:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:09:54] ppocr INFO: epoch: [302/500], global_step: 5400, lr: 0.000100, edit distance: 0.001658, exp_rate: 0.927878, exp_rate<=1 : 0.971622, exp_rate<=2 : 0.992960, exp_rate<=3 : 0.997528, loss: 0.001509, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19050 s, avg_samples: 56.0, ips: 293.96568 samples/s, eta: 1 day, 5:02:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:10:09] ppocr INFO: epoch: [302/500], global_step: 5480, lr: 0.000100, edit distance: 0.001658, exp_rate: 0.927746, exp_rate<=1 : 0.971546, exp_rate<=2 : 0.992968, exp_rate<=3 : 0.997516, loss: 0.001912, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15173 s, avg_samples: 44.8, ips: 295.26195 samples/s, eta: 1 day, 5:01:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:10:11] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 03:10:15] ppocr INFO: epoch: [303/500], global_step: 5500, lr: 0.000100, edit distance: 0.001464, exp_rate: 0.944792, exp_rate<=1 : 0.969712, exp_rate<=2 : 0.991415, exp_rate<=3 : 0.997351, loss: 0.001787, avg_reader_cost: 0.02551 s, avg_batch_cost: 0.06180 s, avg_samples: 11.2, ips: 181.24373 samples/s, eta: 1 day, 5:05:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:10:33] ppocr INFO: epoch: [303/500], global_step: 5600, lr: 0.000100, edit distance: 0.001558, exp_rate: 0.942404, exp_rate<=1 : 0.974696, exp_rate<=2 : 0.993800, exp_rate<=3 : 0.997961, loss: 0.001799, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17725 s, avg_samples: 56.0, ips: 315.93364 samples/s, eta: 1 day, 5:02:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:10:52] ppocr INFO: epoch: [303/500], global_step: 5700, lr: 0.000100, edit distance: 0.001627, exp_rate: 0.934579, exp_rate<=1 : 0.973715, exp_rate<=2 : 0.993380, exp_rate<=3 : 0.997718, loss: 0.001960, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18836 s, avg_samples: 56.0, ips: 297.30468 samples/s, eta: 1 day, 5:01:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:11:11] ppocr INFO: epoch: [303/500], global_step: 5800, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.936178, exp_rate<=1 : 0.974666, exp_rate<=2 : 0.994249, exp_rate<=3 : 0.998045, loss: 0.001671, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18680 s, avg_samples: 56.0, ips: 299.78059 samples/s, eta: 1 day, 5:00:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:11:29] ppocr INFO: epoch: [303/500], global_step: 5900, lr: 0.000100, edit distance: 0.001484, exp_rate: 0.937261, exp_rate<=1 : 0.975515, exp_rate<=2 : 0.994315, exp_rate<=3 : 0.998119, loss: 0.001444, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18763 s, avg_samples: 56.0, ips: 298.45547 samples/s, eta: 1 day, 4:59:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:11:49] ppocr INFO: epoch: [303/500], global_step: 6000, lr: 0.000100, edit distance: 0.001517, exp_rate: 0.936249, exp_rate<=1 : 0.975385, exp_rate<=2 : 0.994170, exp_rate<=3 : 0.998147, loss: 0.002032, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19701 s, avg_samples: 56.0, ips: 284.24716 samples/s, eta: 1 day, 4:59:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:12:07] ppocr INFO: epoch: [303/500], global_step: 6100, lr: 0.000100, edit distance: 0.001499, exp_rate: 0.936805, exp_rate<=1 : 0.975514, exp_rate<=2 : 0.994111, exp_rate<=3 : 0.998086, loss: 0.001498, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17949 s, avg_samples: 56.0, ips: 311.99728 samples/s, eta: 1 day, 4:57:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:12:26] ppocr INFO: epoch: [303/500], global_step: 6200, lr: 0.000100, edit distance: 0.001506, exp_rate: 0.936212, exp_rate<=1 : 0.975484, exp_rate<=2 : 0.994272, exp_rate<=3 : 0.998079, loss: 0.002149, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18619 s, avg_samples: 56.0, ips: 300.76335 samples/s, eta: 1 day, 4:55:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:12:45] ppocr INFO: epoch: [303/500], global_step: 6300, lr: 0.000100, edit distance: 0.001535, exp_rate: 0.934164, exp_rate<=1 : 0.974866, exp_rate<=2 : 0.994169, exp_rate<=3 : 0.997975, loss: 0.001529, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19207 s, avg_samples: 56.0, ips: 291.56683 samples/s, eta: 1 day, 4:55:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:13:05] ppocr INFO: epoch: [303/500], global_step: 6400, lr: 0.000100, edit distance: 0.001533, exp_rate: 0.933117, exp_rate<=1 : 0.973720, exp_rate<=2 : 0.993530, exp_rate<=3 : 0.997709, loss: 0.002027, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20277 s, avg_samples: 56.0, ips: 276.18046 samples/s, eta: 1 day, 4:56:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:13:24] ppocr INFO: epoch: [303/500], global_step: 6500, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.932447, exp_rate<=1 : 0.973715, exp_rate<=2 : 0.993429, exp_rate<=3 : 0.997798, loss: 0.001785, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18681 s, avg_samples: 56.0, ips: 299.77124 samples/s, eta: 1 day, 4:55:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:13:43] ppocr INFO: epoch: [303/500], global_step: 6600, lr: 0.000100, edit distance: 0.001543, exp_rate: 0.931878, exp_rate<=1 : 0.973384, exp_rate<=2 : 0.993276, exp_rate<=3 : 0.997709, loss: 0.002482, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19353 s, avg_samples: 56.0, ips: 289.35643 samples/s, eta: 1 day, 4:55:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:14:03] ppocr INFO: epoch: [303/500], global_step: 6700, lr: 0.000100, edit distance: 0.001566, exp_rate: 0.930334, exp_rate<=1 : 0.972371, exp_rate<=2 : 0.993160, exp_rate<=3 : 0.997678, loss: 0.002236, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19988 s, avg_samples: 56.0, ips: 280.17193 samples/s, eta: 1 day, 4:56:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:14:22] ppocr INFO: epoch: [303/500], global_step: 6800, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.930935, exp_rate<=1 : 0.972707, exp_rate<=2 : 0.993477, exp_rate<=3 : 0.997787, loss: 0.001214, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18505 s, avg_samples: 56.0, ips: 302.61549 samples/s, eta: 1 day, 4:54:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:14:41] ppocr INFO: epoch: [303/500], global_step: 6900, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.931109, exp_rate<=1 : 0.972765, exp_rate<=2 : 0.993374, exp_rate<=3 : 0.997759, loss: 0.002060, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18900 s, avg_samples: 56.0, ips: 296.28895 samples/s, eta: 1 day, 4:54:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:15:00] ppocr INFO: epoch: [303/500], global_step: 7000, lr: 0.000100, edit distance: 0.001560, exp_rate: 0.930463, exp_rate<=1 : 0.972448, exp_rate<=2 : 0.993227, exp_rate<=3 : 0.997696, loss: 0.001917, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19015 s, avg_samples: 56.0, ips: 294.50396 samples/s, eta: 1 day, 4:53:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:15:20] ppocr INFO: epoch: [303/500], global_step: 7100, lr: 0.000100, edit distance: 0.001559, exp_rate: 0.929915, exp_rate<=1 : 0.972236, exp_rate<=2 : 0.993219, exp_rate<=3 : 0.997720, loss: 0.001259, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20305 s, avg_samples: 56.0, ips: 275.78894 samples/s, eta: 1 day, 4:54:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:15:39] ppocr INFO: epoch: [303/500], global_step: 7200, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.930118, exp_rate<=1 : 0.972281, exp_rate<=2 : 0.993266, exp_rate<=3 : 0.997768, loss: 0.001885, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18926 s, avg_samples: 56.0, ips: 295.88739 samples/s, eta: 1 day, 4:53:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:15:59] ppocr INFO: epoch: [303/500], global_step: 7300, lr: 0.000100, edit distance: 0.001567, exp_rate: 0.929063, exp_rate<=1 : 0.971867, exp_rate<=2 : 0.993067, exp_rate<=3 : 0.997595, loss: 0.002402, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19388 s, avg_samples: 56.0, ips: 288.83405 samples/s, eta: 1 day, 4:53:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:16:18] ppocr INFO: epoch: [303/500], global_step: 7400, lr: 0.000100, edit distance: 0.001577, exp_rate: 0.928450, exp_rate<=1 : 0.971394, exp_rate<=2 : 0.992999, exp_rate<=3 : 0.997548, loss: 0.001715, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19908 s, avg_samples: 56.0, ips: 281.29836 samples/s, eta: 1 day, 4:54:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:16:38] ppocr INFO: epoch: [303/500], global_step: 7500, lr: 0.000100, edit distance: 0.001582, exp_rate: 0.927958, exp_rate<=1 : 0.971227, exp_rate<=2 : 0.992948, exp_rate<=3 : 0.997566, loss: 0.001998, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19883 s, avg_samples: 56.0, ips: 281.64477 samples/s, eta: 1 day, 4:54:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:16:57] ppocr INFO: epoch: [303/500], global_step: 7600, lr: 0.000100, edit distance: 0.001576, exp_rate: 0.928166, exp_rate<=1 : 0.971376, exp_rate<=2 : 0.993014, exp_rate<=3 : 0.997593, loss: 0.001876, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18641 s, avg_samples: 56.0, ips: 300.41628 samples/s, eta: 1 day, 4:53:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:17:16] ppocr INFO: epoch: [303/500], global_step: 7700, lr: 0.000100, edit distance: 0.001576, exp_rate: 0.928075, exp_rate<=1 : 0.971503, exp_rate<=2 : 0.993012, exp_rate<=3 : 0.997535, loss: 0.001872, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19220 s, avg_samples: 56.0, ips: 291.36269 samples/s, eta: 1 day, 4:53:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:17:35] ppocr INFO: epoch: [303/500], global_step: 7800, lr: 0.000100, edit distance: 0.001581, exp_rate: 0.927577, exp_rate<=1 : 0.971143, exp_rate<=2 : 0.992897, exp_rate<=3 : 0.997526, loss: 0.001655, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19075 s, avg_samples: 56.0, ips: 293.58107 samples/s, eta: 1 day, 4:52:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:17:55] ppocr INFO: epoch: [303/500], global_step: 7900, lr: 0.000100, edit distance: 0.001583, exp_rate: 0.927444, exp_rate<=1 : 0.971088, exp_rate<=2 : 0.992828, exp_rate<=3 : 0.997532, loss: 0.002410, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19601 s, avg_samples: 56.0, ips: 285.70532 samples/s, eta: 1 day, 4:52:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:18:14] ppocr INFO: epoch: [303/500], global_step: 8000, lr: 0.000100, edit distance: 0.001588, exp_rate: 0.927279, exp_rate<=1 : 0.970966, exp_rate<=2 : 0.992781, exp_rate<=3 : 0.997505, loss: 0.002186, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19108 s, avg_samples: 56.0, ips: 293.07154 samples/s, eta: 1 day, 4:52:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:18:34] ppocr INFO: epoch: [303/500], global_step: 8100, lr: 0.000100, edit distance: 0.001600, exp_rate: 0.926825, exp_rate<=1 : 0.970879, exp_rate<=2 : 0.992759, exp_rate<=3 : 0.997465, loss: 0.002208, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19616 s, avg_samples: 56.0, ips: 285.47682 samples/s, eta: 1 day, 4:52:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:18:53] ppocr INFO: epoch: [303/500], global_step: 8200, lr: 0.000100, edit distance: 0.001620, exp_rate: 0.926482, exp_rate<=1 : 0.970641, exp_rate<=2 : 0.992735, exp_rate<=3 : 0.997475, loss: 0.002172, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18761 s, avg_samples: 56.0, ips: 298.49851 samples/s, eta: 1 day, 4:51:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:18:56] ppocr INFO: epoch: [303/500], global_step: 8220, lr: 0.000100, edit distance: 0.001625, exp_rate: 0.926519, exp_rate<=1 : 0.970619, exp_rate<=2 : 0.992740, exp_rate<=3 : 0.997482, loss: 0.001859, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03346 s, avg_samples: 11.2, ips: 334.71871 samples/s, eta: 1 day, 4:50:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:18:58] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 03:19:15] ppocr INFO: epoch: [304/500], global_step: 8300, lr: 0.000100, edit distance: 0.002136, exp_rate: 0.919204, exp_rate<=1 : 0.967067, exp_rate<=2 : 0.991679, exp_rate<=3 : 0.997243, loss: 0.002049, avg_reader_cost: 0.02705 s, avg_batch_cost: 0.19152 s, avg_samples: 44.8, ips: 233.91869 samples/s, eta: 1 day, 4:54:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:19:33] ppocr INFO: epoch: [304/500], global_step: 8400, lr: 0.000100, edit distance: 0.001728, exp_rate: 0.928364, exp_rate<=1 : 0.972671, exp_rate<=2 : 0.993152, exp_rate<=3 : 0.997808, loss: 0.001223, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18355 s, avg_samples: 56.0, ips: 305.08863 samples/s, eta: 1 day, 4:53:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:19:52] ppocr INFO: epoch: [304/500], global_step: 8500, lr: 0.000100, edit distance: 0.001570, exp_rate: 0.933489, exp_rate<=1 : 0.975475, exp_rate<=2 : 0.994257, exp_rate<=3 : 0.998152, loss: 0.000992, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18641 s, avg_samples: 56.0, ips: 300.40541 samples/s, eta: 1 day, 4:52:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:20:11] ppocr INFO: epoch: [304/500], global_step: 8600, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.934579, exp_rate<=1 : 0.975758, exp_rate<=2 : 0.994023, exp_rate<=3 : 0.998032, loss: 0.001636, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18975 s, avg_samples: 56.0, ips: 295.13241 samples/s, eta: 1 day, 4:51:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:20:30] ppocr INFO: epoch: [304/500], global_step: 8700, lr: 0.000100, edit distance: 0.001502, exp_rate: 0.935578, exp_rate<=1 : 0.976166, exp_rate<=2 : 0.994099, exp_rate<=3 : 0.997893, loss: 0.002210, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18657 s, avg_samples: 56.0, ips: 300.15069 samples/s, eta: 1 day, 4:50:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:20:49] ppocr INFO: epoch: [304/500], global_step: 8800, lr: 0.000100, edit distance: 0.001545, exp_rate: 0.934409, exp_rate<=1 : 0.975399, exp_rate<=2 : 0.993210, exp_rate<=3 : 0.997545, loss: 0.001430, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19038 s, avg_samples: 56.0, ips: 294.15195 samples/s, eta: 1 day, 4:50:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:21:09] ppocr INFO: epoch: [304/500], global_step: 8900, lr: 0.000100, edit distance: 0.001526, exp_rate: 0.933832, exp_rate<=1 : 0.975058, exp_rate<=2 : 0.993164, exp_rate<=3 : 0.997481, loss: 0.001598, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19658 s, avg_samples: 56.0, ips: 284.87243 samples/s, eta: 1 day, 4:50:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:21:27] ppocr INFO: epoch: [304/500], global_step: 9000, lr: 0.000100, edit distance: 0.001522, exp_rate: 0.933654, exp_rate<=1 : 0.974946, exp_rate<=2 : 0.993238, exp_rate<=3 : 0.997587, loss: 0.002015, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18888 s, avg_samples: 56.0, ips: 296.48975 samples/s, eta: 1 day, 4:49:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:21:46] ppocr INFO: epoch: [304/500], global_step: 9100, lr: 0.000100, edit distance: 0.001536, exp_rate: 0.933254, exp_rate<=1 : 0.974237, exp_rate<=2 : 0.993230, exp_rate<=3 : 0.997474, loss: 0.001657, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18957 s, avg_samples: 56.0, ips: 295.40912 samples/s, eta: 1 day, 4:49:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:22:05] ppocr INFO: epoch: [304/500], global_step: 9200, lr: 0.000100, edit distance: 0.001525, exp_rate: 0.934033, exp_rate<=1 : 0.974718, exp_rate<=2 : 0.993504, exp_rate<=3 : 0.997565, loss: 0.002022, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18656 s, avg_samples: 56.0, ips: 300.16712 samples/s, eta: 1 day, 4:48:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:22:25] ppocr INFO: epoch: [304/500], global_step: 9300, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.933174, exp_rate<=1 : 0.974573, exp_rate<=2 : 0.993421, exp_rate<=3 : 0.997464, loss: 0.001085, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19531 s, avg_samples: 56.0, ips: 286.72097 samples/s, eta: 1 day, 4:48:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:22:44] ppocr INFO: epoch: [304/500], global_step: 9400, lr: 0.000100, edit distance: 0.001541, exp_rate: 0.933553, exp_rate<=1 : 0.974683, exp_rate<=2 : 0.993588, exp_rate<=3 : 0.997586, loss: 0.001781, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19363 s, avg_samples: 56.0, ips: 289.21536 samples/s, eta: 1 day, 4:48:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:23:04] ppocr INFO: epoch: [304/500], global_step: 9500, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.933454, exp_rate<=1 : 0.974414, exp_rate<=2 : 0.993547, exp_rate<=3 : 0.997569, loss: 0.002323, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19679 s, avg_samples: 56.0, ips: 284.56782 samples/s, eta: 1 day, 4:48:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:23:24] ppocr INFO: epoch: [304/500], global_step: 9600, lr: 0.000100, edit distance: 0.001544, exp_rate: 0.932265, exp_rate<=1 : 0.973951, exp_rate<=2 : 0.993472, exp_rate<=3 : 0.997580, loss: 0.001724, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20695 s, avg_samples: 56.0, ips: 270.59349 samples/s, eta: 1 day, 4:49:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:23:44] ppocr INFO: epoch: [304/500], global_step: 9700, lr: 0.000100, edit distance: 0.001550, exp_rate: 0.931430, exp_rate<=1 : 0.973393, exp_rate<=2 : 0.993218, exp_rate<=3 : 0.997375, loss: 0.001551, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19588 s, avg_samples: 56.0, ips: 285.89649 samples/s, eta: 1 day, 4:49:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:24:04] ppocr INFO: epoch: [304/500], global_step: 9800, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.930571, exp_rate<=1 : 0.973070, exp_rate<=2 : 0.993127, exp_rate<=3 : 0.997293, loss: 0.001672, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19470 s, avg_samples: 56.0, ips: 287.61499 samples/s, eta: 1 day, 4:48:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:24:24] ppocr INFO: epoch: [304/500], global_step: 9900, lr: 0.000100, edit distance: 0.001568, exp_rate: 0.929410, exp_rate<=1 : 0.972278, exp_rate<=2 : 0.992979, exp_rate<=3 : 0.997256, loss: 0.001897, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20279 s, avg_samples: 56.0, ips: 276.14635 samples/s, eta: 1 day, 4:49:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:24:42] ppocr INFO: epoch: [304/500], global_step: 10000, lr: 0.000100, edit distance: 0.001568, exp_rate: 0.929407, exp_rate<=1 : 0.972238, exp_rate<=2 : 0.993044, exp_rate<=3 : 0.997327, loss: 0.002132, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18368 s, avg_samples: 56.0, ips: 304.87647 samples/s, eta: 1 day, 4:48:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:25:01] ppocr INFO: epoch: [304/500], global_step: 10100, lr: 0.000100, edit distance: 0.001593, exp_rate: 0.928847, exp_rate<=1 : 0.972081, exp_rate<=2 : 0.992986, exp_rate<=3 : 0.997368, loss: 0.001655, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18361 s, avg_samples: 56.0, ips: 304.99009 samples/s, eta: 1 day, 4:47:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:25:20] ppocr INFO: epoch: [304/500], global_step: 10200, lr: 0.000100, edit distance: 0.001600, exp_rate: 0.928277, exp_rate<=1 : 0.971753, exp_rate<=2 : 0.992838, exp_rate<=3 : 0.997295, loss: 0.002011, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19563 s, avg_samples: 56.0, ips: 286.25463 samples/s, eta: 1 day, 4:47:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:25:39] ppocr INFO: epoch: [304/500], global_step: 10300, lr: 0.000100, edit distance: 0.001611, exp_rate: 0.928046, exp_rate<=1 : 0.971568, exp_rate<=2 : 0.992824, exp_rate<=3 : 0.997273, loss: 0.001453, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.24996 samples/s, eta: 1 day, 4:46:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:25:56] ppocr INFO: epoch: [304/500], global_step: 10400, lr: 0.000100, edit distance: 0.001619, exp_rate: 0.928193, exp_rate<=1 : 0.971666, exp_rate<=2 : 0.992907, exp_rate<=3 : 0.997327, loss: 0.001832, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.17333 s, avg_samples: 56.0, ips: 323.08432 samples/s, eta: 1 day, 4:44:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:26:16] ppocr INFO: epoch: [304/500], global_step: 10500, lr: 0.000100, edit distance: 0.001614, exp_rate: 0.928150, exp_rate<=1 : 0.971688, exp_rate<=2 : 0.992970, exp_rate<=3 : 0.997349, loss: 0.001949, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19460 s, avg_samples: 56.0, ips: 287.76684 samples/s, eta: 1 day, 4:44:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:26:35] ppocr INFO: epoch: [304/500], global_step: 10600, lr: 0.000100, edit distance: 0.001612, exp_rate: 0.928102, exp_rate<=1 : 0.971716, exp_rate<=2 : 0.992987, exp_rate<=3 : 0.997356, loss: 0.002039, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19091 s, avg_samples: 56.0, ips: 293.33069 samples/s, eta: 1 day, 4:43:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:26:55] ppocr INFO: epoch: [304/500], global_step: 10700, lr: 0.000100, edit distance: 0.001604, exp_rate: 0.928359, exp_rate<=1 : 0.971776, exp_rate<=2 : 0.993027, exp_rate<=3 : 0.997390, loss: 0.002155, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19824 s, avg_samples: 56.0, ips: 282.48741 samples/s, eta: 1 day, 4:44:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:27:15] ppocr INFO: epoch: [304/500], global_step: 10800, lr: 0.000100, edit distance: 0.001608, exp_rate: 0.928164, exp_rate<=1 : 0.971678, exp_rate<=2 : 0.992966, exp_rate<=3 : 0.997370, loss: 0.002107, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20032 s, avg_samples: 56.0, ips: 279.55782 samples/s, eta: 1 day, 4:44:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:27:33] ppocr INFO: epoch: [304/500], global_step: 10900, lr: 0.000100, edit distance: 0.001619, exp_rate: 0.927899, exp_rate<=1 : 0.971554, exp_rate<=2 : 0.992922, exp_rate<=3 : 0.997350, loss: 0.002289, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18202 s, avg_samples: 56.0, ips: 307.66647 samples/s, eta: 1 day, 4:43:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:27:44] ppocr INFO: epoch: [304/500], global_step: 10960, lr: 0.000100, edit distance: 0.001612, exp_rate: 0.928068, exp_rate<=1 : 0.971648, exp_rate<=2 : 0.992965, exp_rate<=3 : 0.997364, loss: 0.001811, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10708 s, avg_samples: 33.6, ips: 313.79397 samples/s, eta: 1 day, 4:42:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:27:46] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 03:27:54] ppocr INFO: epoch: [305/500], global_step: 11000, lr: 0.000100, edit distance: 0.001098, exp_rate: 0.945777, exp_rate<=1 : 0.981261, exp_rate<=2 : 0.996007, exp_rate<=3 : 0.999414, loss: 0.001322, avg_reader_cost: 0.02622 s, avg_batch_cost: 0.10037 s, avg_samples: 22.4, ips: 223.16787 samples/s, eta: 1 day, 4:44:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:28:13] ppocr INFO: epoch: [305/500], global_step: 11100, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.942019, exp_rate<=1 : 0.976751, exp_rate<=2 : 0.994299, exp_rate<=3 : 0.998221, loss: 0.001194, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18971 s, avg_samples: 56.0, ips: 295.18965 samples/s, eta: 1 day, 4:43:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:28:33] ppocr INFO: epoch: [305/500], global_step: 11200, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.938893, exp_rate<=1 : 0.975818, exp_rate<=2 : 0.994415, exp_rate<=3 : 0.997908, loss: 0.002044, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20178 s, avg_samples: 56.0, ips: 277.53414 samples/s, eta: 1 day, 4:43:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:28:52] ppocr INFO: epoch: [305/500], global_step: 11300, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.938051, exp_rate<=1 : 0.974918, exp_rate<=2 : 0.993468, exp_rate<=3 : 0.997407, loss: 0.001592, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19457 s, avg_samples: 56.0, ips: 287.80763 samples/s, eta: 1 day, 4:43:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:29:11] ppocr INFO: epoch: [305/500], global_step: 11400, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.939221, exp_rate<=1 : 0.975629, exp_rate<=2 : 0.993985, exp_rate<=3 : 0.997706, loss: 0.001064, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18530 s, avg_samples: 56.0, ips: 302.20868 samples/s, eta: 1 day, 4:42:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:29:30] ppocr INFO: epoch: [305/500], global_step: 11500, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.937877, exp_rate<=1 : 0.975365, exp_rate<=2 : 0.993741, exp_rate<=3 : 0.997757, loss: 0.002117, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18812 s, avg_samples: 56.0, ips: 297.67861 samples/s, eta: 1 day, 4:42:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:29:51] ppocr INFO: epoch: [305/500], global_step: 11600, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.934702, exp_rate<=1 : 0.973327, exp_rate<=2 : 0.993280, exp_rate<=3 : 0.997580, loss: 0.001487, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21365 s, avg_samples: 56.0, ips: 262.10582 samples/s, eta: 1 day, 4:43:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:30:10] ppocr INFO: epoch: [305/500], global_step: 11700, lr: 0.000100, edit distance: 0.001390, exp_rate: 0.933008, exp_rate<=1 : 0.972675, exp_rate<=2 : 0.993020, exp_rate<=3 : 0.997584, loss: 0.002170, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.18557 s, avg_samples: 56.0, ips: 301.76831 samples/s, eta: 1 day, 4:42:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:30:28] ppocr INFO: epoch: [305/500], global_step: 11800, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.932573, exp_rate<=1 : 0.972618, exp_rate<=2 : 0.993370, exp_rate<=3 : 0.997753, loss: 0.001506, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18014 s, avg_samples: 56.0, ips: 310.87622 samples/s, eta: 1 day, 4:41:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:30:47] ppocr INFO: epoch: [305/500], global_step: 11900, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.932883, exp_rate<=1 : 0.973109, exp_rate<=2 : 0.993694, exp_rate<=3 : 0.997893, loss: 0.001613, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18687 s, avg_samples: 56.0, ips: 299.66965 samples/s, eta: 1 day, 4:40:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:31:04] ppocr INFO: epoch: [305/500], global_step: 12000, lr: 0.000100, edit distance: 0.001451, exp_rate: 0.933144, exp_rate<=1 : 0.973518, exp_rate<=2 : 0.993678, exp_rate<=3 : 0.997851, loss: 0.002402, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17480 s, avg_samples: 56.0, ips: 320.36524 samples/s, eta: 1 day, 4:38:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:31:24] ppocr INFO: epoch: [305/500], global_step: 12100, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.931820, exp_rate<=1 : 0.972782, exp_rate<=2 : 0.993446, exp_rate<=3 : 0.997678, loss: 0.001866, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19904 s, avg_samples: 56.0, ips: 281.35641 samples/s, eta: 1 day, 4:39:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:31:43] ppocr INFO: epoch: [305/500], global_step: 12200, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.931578, exp_rate<=1 : 0.972840, exp_rate<=2 : 0.993433, exp_rate<=3 : 0.997678, loss: 0.002325, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19453 s, avg_samples: 56.0, ips: 287.87148 samples/s, eta: 1 day, 4:38:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:32:03] ppocr INFO: epoch: [305/500], global_step: 12300, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.931495, exp_rate<=1 : 0.972767, exp_rate<=2 : 0.993456, exp_rate<=3 : 0.997743, loss: 0.001810, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19732 s, avg_samples: 56.0, ips: 283.80823 samples/s, eta: 1 day, 4:38:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:32:21] ppocr INFO: epoch: [305/500], global_step: 12400, lr: 0.000100, edit distance: 0.001477, exp_rate: 0.931572, exp_rate<=1 : 0.972642, exp_rate<=2 : 0.993340, exp_rate<=3 : 0.997743, loss: 0.001158, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17912 s, avg_samples: 56.0, ips: 312.63934 samples/s, eta: 1 day, 4:37:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:32:41] ppocr INFO: epoch: [305/500], global_step: 12500, lr: 0.000100, edit distance: 0.001490, exp_rate: 0.931292, exp_rate<=1 : 0.972669, exp_rate<=2 : 0.993375, exp_rate<=3 : 0.997649, loss: 0.002777, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19789 s, avg_samples: 56.0, ips: 282.98735 samples/s, eta: 1 day, 4:37:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:33:00] ppocr INFO: epoch: [305/500], global_step: 12600, lr: 0.000100, edit distance: 0.001501, exp_rate: 0.930952, exp_rate<=1 : 0.972536, exp_rate<=2 : 0.993305, exp_rate<=3 : 0.997604, loss: 0.001565, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19334 s, avg_samples: 56.0, ips: 289.64958 samples/s, eta: 1 day, 4:37:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:33:21] ppocr INFO: epoch: [305/500], global_step: 12700, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.929993, exp_rate<=1 : 0.971915, exp_rate<=2 : 0.992928, exp_rate<=3 : 0.997491, loss: 0.001363, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20286 s, avg_samples: 56.0, ips: 276.04992 samples/s, eta: 1 day, 4:37:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:33:39] ppocr INFO: epoch: [305/500], global_step: 12800, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.929912, exp_rate<=1 : 0.971848, exp_rate<=2 : 0.993011, exp_rate<=3 : 0.997523, loss: 0.001888, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18264 s, avg_samples: 56.0, ips: 306.60939 samples/s, eta: 1 day, 4:36:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:33:58] ppocr INFO: epoch: [305/500], global_step: 12900, lr: 0.000100, edit distance: 0.001510, exp_rate: 0.929797, exp_rate<=1 : 0.971758, exp_rate<=2 : 0.992950, exp_rate<=3 : 0.997512, loss: 0.001512, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19368 s, avg_samples: 56.0, ips: 289.13267 samples/s, eta: 1 day, 4:36:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:34:17] ppocr INFO: epoch: [305/500], global_step: 13000, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.929687, exp_rate<=1 : 0.971591, exp_rate<=2 : 0.992923, exp_rate<=3 : 0.997503, loss: 0.001833, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18887 s, avg_samples: 56.0, ips: 296.49770 samples/s, eta: 1 day, 4:36:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:34:36] ppocr INFO: epoch: [305/500], global_step: 13100, lr: 0.000100, edit distance: 0.001529, exp_rate: 0.929439, exp_rate<=1 : 0.971659, exp_rate<=2 : 0.992977, exp_rate<=3 : 0.997515, loss: 0.002080, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18901 s, avg_samples: 56.0, ips: 296.28144 samples/s, eta: 1 day, 4:35:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:34:55] ppocr INFO: epoch: [305/500], global_step: 13200, lr: 0.000100, edit distance: 0.001553, exp_rate: 0.928748, exp_rate<=1 : 0.971519, exp_rate<=2 : 0.992958, exp_rate<=3 : 0.997508, loss: 0.002373, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18950 s, avg_samples: 56.0, ips: 295.52077 samples/s, eta: 1 day, 4:34:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:35:14] ppocr INFO: epoch: [305/500], global_step: 13300, lr: 0.000100, edit distance: 0.001562, exp_rate: 0.928552, exp_rate<=1 : 0.971452, exp_rate<=2 : 0.993002, exp_rate<=3 : 0.997517, loss: 0.001592, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18743 s, avg_samples: 56.0, ips: 298.77766 samples/s, eta: 1 day, 4:34:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:35:34] ppocr INFO: epoch: [305/500], global_step: 13400, lr: 0.000100, edit distance: 0.001565, exp_rate: 0.928417, exp_rate<=1 : 0.971364, exp_rate<=2 : 0.992984, exp_rate<=3 : 0.997493, loss: 0.001925, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20190 s, avg_samples: 56.0, ips: 277.37162 samples/s, eta: 1 day, 4:34:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:35:55] ppocr INFO: epoch: [305/500], global_step: 13500, lr: 0.000100, edit distance: 0.001565, exp_rate: 0.928048, exp_rate<=1 : 0.971133, exp_rate<=2 : 0.992901, exp_rate<=3 : 0.997400, loss: 0.001949, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20738 s, avg_samples: 56.0, ips: 270.04048 samples/s, eta: 1 day, 4:35:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:36:14] ppocr INFO: epoch: [305/500], global_step: 13600, lr: 0.000100, edit distance: 0.001580, exp_rate: 0.927687, exp_rate<=1 : 0.971038, exp_rate<=2 : 0.992890, exp_rate<=3 : 0.997399, loss: 0.002245, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.35964 samples/s, eta: 1 day, 4:35:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:36:32] ppocr INFO: epoch: [305/500], global_step: 13700, lr: 0.000100, edit distance: 0.001591, exp_rate: 0.927430, exp_rate<=1 : 0.970922, exp_rate<=2 : 0.992815, exp_rate<=3 : 0.997364, loss: 0.001940, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17986 s, avg_samples: 56.0, ips: 311.35478 samples/s, eta: 1 day, 4:33:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:36:35] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 03:36:35] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_305 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 03:36:54] ppocr INFO: epoch: [306/500], global_step: 13800, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.938286, exp_rate<=1 : 0.971726, exp_rate<=2 : 0.992742, exp_rate<=3 : 0.997502, loss: 0.001258, avg_reader_cost: 0.03699 s, avg_batch_cost: 0.21936 s, avg_samples: 56.0, ips: 255.28776 samples/s, eta: 1 day, 4:35:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:37:12] ppocr INFO: epoch: [306/500], global_step: 13900, lr: 0.000100, edit distance: 0.001398, exp_rate: 0.939972, exp_rate<=1 : 0.975068, exp_rate<=2 : 0.994337, exp_rate<=3 : 0.997797, loss: 0.001565, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18014 s, avg_samples: 56.0, ips: 310.86902 samples/s, eta: 1 day, 4:34:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:37:33] ppocr INFO: epoch: [306/500], global_step: 14000, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.935229, exp_rate<=1 : 0.973094, exp_rate<=2 : 0.992756, exp_rate<=3 : 0.997136, loss: 0.001626, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20435 s, avg_samples: 56.0, ips: 274.03633 samples/s, eta: 1 day, 4:34:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:37:52] ppocr INFO: epoch: [306/500], global_step: 14100, lr: 0.000100, edit distance: 0.001454, exp_rate: 0.935592, exp_rate<=1 : 0.973465, exp_rate<=2 : 0.993548, exp_rate<=3 : 0.997439, loss: 0.001533, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19142 s, avg_samples: 56.0, ips: 292.54925 samples/s, eta: 1 day, 4:34:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:38:11] ppocr INFO: epoch: [306/500], global_step: 14200, lr: 0.000100, edit distance: 0.001462, exp_rate: 0.935876, exp_rate<=1 : 0.973344, exp_rate<=2 : 0.993629, exp_rate<=3 : 0.997504, loss: 0.001806, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19493 s, avg_samples: 56.0, ips: 287.27842 samples/s, eta: 1 day, 4:34:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:38:30] ppocr INFO: epoch: [306/500], global_step: 14300, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.936623, exp_rate<=1 : 0.974097, exp_rate<=2 : 0.993921, exp_rate<=3 : 0.997698, loss: 0.001813, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18099 s, avg_samples: 56.0, ips: 309.41423 samples/s, eta: 1 day, 4:33:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:38:48] ppocr INFO: epoch: [306/500], global_step: 14400, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.936178, exp_rate<=1 : 0.974309, exp_rate<=2 : 0.993936, exp_rate<=3 : 0.997673, loss: 0.001634, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18774 s, avg_samples: 56.0, ips: 298.29057 samples/s, eta: 1 day, 4:32:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:39:07] ppocr INFO: epoch: [306/500], global_step: 14500, lr: 0.000100, edit distance: 0.001473, exp_rate: 0.935252, exp_rate<=1 : 0.974151, exp_rate<=2 : 0.993827, exp_rate<=3 : 0.997677, loss: 0.001716, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18611 s, avg_samples: 56.0, ips: 300.89256 samples/s, eta: 1 day, 4:31:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:39:27] ppocr INFO: epoch: [306/500], global_step: 14600, lr: 0.000100, edit distance: 0.001529, exp_rate: 0.932660, exp_rate<=1 : 0.973053, exp_rate<=2 : 0.993627, exp_rate<=3 : 0.997734, loss: 0.002023, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20291 s, avg_samples: 56.0, ips: 275.97796 samples/s, eta: 1 day, 4:32:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:39:46] ppocr INFO: epoch: [306/500], global_step: 14700, lr: 0.000100, edit distance: 0.001528, exp_rate: 0.933357, exp_rate<=1 : 0.973728, exp_rate<=2 : 0.993858, exp_rate<=3 : 0.997802, loss: 0.002047, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18349 s, avg_samples: 56.0, ips: 305.18756 samples/s, eta: 1 day, 4:31:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:40:06] ppocr INFO: epoch: [306/500], global_step: 14800, lr: 0.000100, edit distance: 0.001546, exp_rate: 0.932207, exp_rate<=1 : 0.973051, exp_rate<=2 : 0.993674, exp_rate<=3 : 0.997765, loss: 0.001566, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20189 s, avg_samples: 56.0, ips: 277.37712 samples/s, eta: 1 day, 4:31:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:40:24] ppocr INFO: epoch: [306/500], global_step: 14900, lr: 0.000100, edit distance: 0.001537, exp_rate: 0.932771, exp_rate<=1 : 0.973420, exp_rate<=2 : 0.993872, exp_rate<=3 : 0.997841, loss: 0.001628, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17938 s, avg_samples: 56.0, ips: 312.17872 samples/s, eta: 1 day, 4:30:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:40:44] ppocr INFO: epoch: [306/500], global_step: 15000, lr: 0.000100, edit distance: 0.001580, exp_rate: 0.931609, exp_rate<=1 : 0.972715, exp_rate<=2 : 0.993530, exp_rate<=3 : 0.997708, loss: 0.003059, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19956 s, avg_samples: 56.0, ips: 280.62172 samples/s, eta: 1 day, 4:30:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:41:04] ppocr INFO: epoch: [306/500], global_step: 15100, lr: 0.000100, edit distance: 0.001599, exp_rate: 0.930312, exp_rate<=1 : 0.972076, exp_rate<=2 : 0.993250, exp_rate<=3 : 0.997613, loss: 0.001983, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20607 s, avg_samples: 56.0, ips: 271.75681 samples/s, eta: 1 day, 4:30:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:41:24] ppocr INFO: epoch: [306/500], global_step: 15200, lr: 0.000100, edit distance: 0.001621, exp_rate: 0.929489, exp_rate<=1 : 0.971796, exp_rate<=2 : 0.993226, exp_rate<=3 : 0.997568, loss: 0.002096, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19522 s, avg_samples: 56.0, ips: 286.86017 samples/s, eta: 1 day, 4:30:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:41:43] ppocr INFO: epoch: [306/500], global_step: 15300, lr: 0.000100, edit distance: 0.001631, exp_rate: 0.928981, exp_rate<=1 : 0.971653, exp_rate<=2 : 0.993045, exp_rate<=3 : 0.997498, loss: 0.002059, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18705 s, avg_samples: 56.0, ips: 299.39304 samples/s, eta: 1 day, 4:30:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:42:04] ppocr INFO: epoch: [306/500], global_step: 15400, lr: 0.000100, edit distance: 0.001658, exp_rate: 0.927552, exp_rate<=1 : 0.971008, exp_rate<=2 : 0.992754, exp_rate<=3 : 0.997342, loss: 0.001771, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21047 s, avg_samples: 56.0, ips: 266.07258 samples/s, eta: 1 day, 4:30:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:42:24] ppocr INFO: epoch: [306/500], global_step: 15500, lr: 0.000100, edit distance: 0.001651, exp_rate: 0.927288, exp_rate<=1 : 0.970936, exp_rate<=2 : 0.992717, exp_rate<=3 : 0.997280, loss: 0.001597, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.20155 s, avg_samples: 56.0, ips: 277.84607 samples/s, eta: 1 day, 4:30:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:42:43] ppocr INFO: epoch: [306/500], global_step: 15600, lr: 0.000100, edit distance: 0.001638, exp_rate: 0.927775, exp_rate<=1 : 0.971165, exp_rate<=2 : 0.992794, exp_rate<=3 : 0.997308, loss: 0.001970, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19467 s, avg_samples: 56.0, ips: 287.66473 samples/s, eta: 1 day, 4:30:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:43:02] ppocr INFO: epoch: [306/500], global_step: 15700, lr: 0.000100, edit distance: 0.001639, exp_rate: 0.927985, exp_rate<=1 : 0.971279, exp_rate<=2 : 0.992820, exp_rate<=3 : 0.997339, loss: 0.001800, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18638 s, avg_samples: 56.0, ips: 300.46414 samples/s, eta: 1 day, 4:30:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:43:21] ppocr INFO: epoch: [306/500], global_step: 15800, lr: 0.000100, edit distance: 0.001630, exp_rate: 0.928075, exp_rate<=1 : 0.971233, exp_rate<=2 : 0.992726, exp_rate<=3 : 0.997293, loss: 0.001627, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18933 s, avg_samples: 56.0, ips: 295.78764 samples/s, eta: 1 day, 4:29:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:43:41] ppocr INFO: epoch: [306/500], global_step: 15900, lr: 0.000100, edit distance: 0.001617, exp_rate: 0.927844, exp_rate<=1 : 0.971306, exp_rate<=2 : 0.992783, exp_rate<=3 : 0.997327, loss: 0.002252, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20335 s, avg_samples: 56.0, ips: 275.39324 samples/s, eta: 1 day, 4:29:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:44:01] ppocr INFO: epoch: [306/500], global_step: 16000, lr: 0.000100, edit distance: 0.001609, exp_rate: 0.927995, exp_rate<=1 : 0.971421, exp_rate<=2 : 0.992833, exp_rate<=3 : 0.997349, loss: 0.002274, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19157 s, avg_samples: 56.0, ips: 292.32079 samples/s, eta: 1 day, 4:29:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:44:19] ppocr INFO: epoch: [306/500], global_step: 16100, lr: 0.000100, edit distance: 0.001603, exp_rate: 0.928172, exp_rate<=1 : 0.971518, exp_rate<=2 : 0.992893, exp_rate<=3 : 0.997371, loss: 0.001977, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18768 s, avg_samples: 56.0, ips: 298.38483 samples/s, eta: 1 day, 4:28:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:44:38] ppocr INFO: epoch: [306/500], global_step: 16200, lr: 0.000100, edit distance: 0.001606, exp_rate: 0.928057, exp_rate<=1 : 0.971548, exp_rate<=2 : 0.992970, exp_rate<=3 : 0.997402, loss: 0.001713, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18126 s, avg_samples: 56.0, ips: 308.94980 samples/s, eta: 1 day, 4:27:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:44:55] ppocr INFO: epoch: [306/500], global_step: 16300, lr: 0.000100, edit distance: 0.001615, exp_rate: 0.928061, exp_rate<=1 : 0.971569, exp_rate<=2 : 0.993065, exp_rate<=3 : 0.997415, loss: 0.002495, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17539 s, avg_samples: 56.0, ips: 319.29146 samples/s, eta: 1 day, 4:26:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:45:14] ppocr INFO: epoch: [306/500], global_step: 16400, lr: 0.000100, edit distance: 0.001640, exp_rate: 0.927554, exp_rate<=1 : 0.971358, exp_rate<=2 : 0.992994, exp_rate<=3 : 0.997408, loss: 0.002357, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18873 s, avg_samples: 56.0, ips: 296.72033 samples/s, eta: 1 day, 4:26:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:45:22] ppocr INFO: epoch: [306/500], global_step: 16440, lr: 0.000100, edit distance: 0.001649, exp_rate: 0.927480, exp_rate<=1 : 0.971372, exp_rate<=2 : 0.993014, exp_rate<=3 : 0.997435, loss: 0.002534, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07594 s, avg_samples: 22.4, ips: 294.98490 samples/s, eta: 1 day, 4:25:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:45:24] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 03:45:37] ppocr INFO: epoch: [307/500], global_step: 16500, lr: 0.000100, edit distance: 0.001945, exp_rate: 0.922512, exp_rate<=1 : 0.969170, exp_rate<=2 : 0.991687, exp_rate<=3 : 0.995050, loss: 0.002673, avg_reader_cost: 0.02600 s, avg_batch_cost: 0.15463 s, avg_samples: 33.6, ips: 217.29233 samples/s, eta: 1 day, 4:27:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:45:56] ppocr INFO: epoch: [307/500], global_step: 16600, lr: 0.000100, edit distance: 0.001756, exp_rate: 0.924441, exp_rate<=1 : 0.971666, exp_rate<=2 : 0.993949, exp_rate<=3 : 0.997508, loss: 0.001577, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18448 s, avg_samples: 56.0, ips: 303.55377 samples/s, eta: 1 day, 4:27:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:46:14] ppocr INFO: epoch: [307/500], global_step: 16700, lr: 0.000100, edit distance: 0.001653, exp_rate: 0.927405, exp_rate<=1 : 0.973976, exp_rate<=2 : 0.994313, exp_rate<=3 : 0.998116, loss: 0.001407, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18901 s, avg_samples: 56.0, ips: 296.28524 samples/s, eta: 1 day, 4:26:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:46:34] ppocr INFO: epoch: [307/500], global_step: 16800, lr: 0.000100, edit distance: 0.001627, exp_rate: 0.925343, exp_rate<=1 : 0.972023, exp_rate<=2 : 0.993606, exp_rate<=3 : 0.997804, loss: 0.002062, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19965 s, avg_samples: 56.0, ips: 280.49056 samples/s, eta: 1 day, 4:26:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:46:55] ppocr INFO: epoch: [307/500], global_step: 16900, lr: 0.000100, edit distance: 0.001617, exp_rate: 0.923458, exp_rate<=1 : 0.970905, exp_rate<=2 : 0.992392, exp_rate<=3 : 0.996735, loss: 0.002661, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20399 s, avg_samples: 56.0, ips: 274.52437 samples/s, eta: 1 day, 4:26:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:47:15] ppocr INFO: epoch: [307/500], global_step: 17000, lr: 0.000100, edit distance: 0.001637, exp_rate: 0.923213, exp_rate<=1 : 0.969653, exp_rate<=2 : 0.992251, exp_rate<=3 : 0.996794, loss: 0.001190, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19907 s, avg_samples: 56.0, ips: 281.30997 samples/s, eta: 1 day, 4:26:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:47:33] ppocr INFO: epoch: [307/500], global_step: 17100, lr: 0.000100, edit distance: 0.001655, exp_rate: 0.923973, exp_rate<=1 : 0.969697, exp_rate<=2 : 0.992685, exp_rate<=3 : 0.997063, loss: 0.001278, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18304 s, avg_samples: 56.0, ips: 305.94808 samples/s, eta: 1 day, 4:25:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:47:52] ppocr INFO: epoch: [307/500], global_step: 17200, lr: 0.000100, edit distance: 0.001672, exp_rate: 0.925243, exp_rate<=1 : 0.970179, exp_rate<=2 : 0.992879, exp_rate<=3 : 0.997111, loss: 0.001602, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.25136 samples/s, eta: 1 day, 4:25:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:48:11] ppocr INFO: epoch: [307/500], global_step: 17300, lr: 0.000100, edit distance: 0.001646, exp_rate: 0.925990, exp_rate<=1 : 0.970420, exp_rate<=2 : 0.992823, exp_rate<=3 : 0.997176, loss: 0.001941, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18780 s, avg_samples: 56.0, ips: 298.18464 samples/s, eta: 1 day, 4:24:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:48:30] ppocr INFO: epoch: [307/500], global_step: 17400, lr: 0.000100, edit distance: 0.001624, exp_rate: 0.927013, exp_rate<=1 : 0.970843, exp_rate<=2 : 0.992966, exp_rate<=3 : 0.997270, loss: 0.002363, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18896 s, avg_samples: 56.0, ips: 296.36342 samples/s, eta: 1 day, 4:24:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:48:47] ppocr INFO: epoch: [307/500], global_step: 17500, lr: 0.000100, edit distance: 0.001620, exp_rate: 0.927323, exp_rate<=1 : 0.970930, exp_rate<=2 : 0.992971, exp_rate<=3 : 0.997272, loss: 0.001263, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17743 s, avg_samples: 56.0, ips: 315.61249 samples/s, eta: 1 day, 4:23:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:49:07] ppocr INFO: epoch: [307/500], global_step: 17600, lr: 0.000100, edit distance: 0.001629, exp_rate: 0.927485, exp_rate<=1 : 0.970994, exp_rate<=2 : 0.992781, exp_rate<=3 : 0.997175, loss: 0.002124, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19542 s, avg_samples: 56.0, ips: 286.55754 samples/s, eta: 1 day, 4:23:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:49:25] ppocr INFO: epoch: [307/500], global_step: 17700, lr: 0.000100, edit distance: 0.001605, exp_rate: 0.928429, exp_rate<=1 : 0.971440, exp_rate<=2 : 0.992916, exp_rate<=3 : 0.997215, loss: 0.001640, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18057 s, avg_samples: 56.0, ips: 310.13609 samples/s, eta: 1 day, 4:22:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:49:46] ppocr INFO: epoch: [307/500], global_step: 17800, lr: 0.000100, edit distance: 0.001598, exp_rate: 0.927964, exp_rate<=1 : 0.970968, exp_rate<=2 : 0.992601, exp_rate<=3 : 0.997071, loss: 0.001747, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20519 s, avg_samples: 56.0, ips: 272.91324 samples/s, eta: 1 day, 4:22:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:50:05] ppocr INFO: epoch: [307/500], global_step: 17900, lr: 0.000100, edit distance: 0.001581, exp_rate: 0.928128, exp_rate<=1 : 0.971029, exp_rate<=2 : 0.992625, exp_rate<=3 : 0.997058, loss: 0.001430, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19883 s, avg_samples: 56.0, ips: 281.64483 samples/s, eta: 1 day, 4:22:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:50:26] ppocr INFO: epoch: [307/500], global_step: 18000, lr: 0.000100, edit distance: 0.001571, exp_rate: 0.927983, exp_rate<=1 : 0.971032, exp_rate<=2 : 0.992697, exp_rate<=3 : 0.997075, loss: 0.002425, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20494 s, avg_samples: 56.0, ips: 273.24421 samples/s, eta: 1 day, 4:22:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:50:46] ppocr INFO: epoch: [307/500], global_step: 18100, lr: 0.000100, edit distance: 0.001572, exp_rate: 0.927976, exp_rate<=1 : 0.970915, exp_rate<=2 : 0.992506, exp_rate<=3 : 0.997017, loss: 0.002192, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20075 s, avg_samples: 56.0, ips: 278.95645 samples/s, eta: 1 day, 4:22:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:51:05] ppocr INFO: epoch: [307/500], global_step: 18200, lr: 0.000100, edit distance: 0.001573, exp_rate: 0.927770, exp_rate<=1 : 0.970858, exp_rate<=2 : 0.992584, exp_rate<=3 : 0.997021, loss: 0.001273, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18927 s, avg_samples: 56.0, ips: 295.87227 samples/s, eta: 1 day, 4:22:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:51:25] ppocr INFO: epoch: [307/500], global_step: 18300, lr: 0.000100, edit distance: 0.001573, exp_rate: 0.927605, exp_rate<=1 : 0.970946, exp_rate<=2 : 0.992752, exp_rate<=3 : 0.997078, loss: 0.001944, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19509 s, avg_samples: 56.0, ips: 287.05422 samples/s, eta: 1 day, 4:22:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:51:45] ppocr INFO: epoch: [307/500], global_step: 18400, lr: 0.000100, edit distance: 0.001568, exp_rate: 0.927507, exp_rate<=1 : 0.970952, exp_rate<=2 : 0.992727, exp_rate<=3 : 0.997089, loss: 0.001949, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20508 s, avg_samples: 56.0, ips: 273.06269 samples/s, eta: 1 day, 4:22:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:52:04] ppocr INFO: epoch: [307/500], global_step: 18500, lr: 0.000100, edit distance: 0.001567, exp_rate: 0.927543, exp_rate<=1 : 0.970886, exp_rate<=2 : 0.992667, exp_rate<=3 : 0.997068, loss: 0.001664, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19133 s, avg_samples: 56.0, ips: 292.69037 samples/s, eta: 1 day, 4:21:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:52:23] ppocr INFO: epoch: [307/500], global_step: 18600, lr: 0.000100, edit distance: 0.001573, exp_rate: 0.927349, exp_rate<=1 : 0.970760, exp_rate<=2 : 0.992656, exp_rate<=3 : 0.997084, loss: 0.001886, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18809 s, avg_samples: 56.0, ips: 297.73576 samples/s, eta: 1 day, 4:21:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:52:41] ppocr INFO: epoch: [307/500], global_step: 18700, lr: 0.000100, edit distance: 0.001582, exp_rate: 0.927674, exp_rate<=1 : 0.970937, exp_rate<=2 : 0.992709, exp_rate<=3 : 0.997126, loss: 0.001882, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17804 s, avg_samples: 56.0, ips: 314.53205 samples/s, eta: 1 day, 4:20:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:53:00] ppocr INFO: epoch: [307/500], global_step: 18800, lr: 0.000100, edit distance: 0.001579, exp_rate: 0.927934, exp_rate<=1 : 0.971007, exp_rate<=2 : 0.992783, exp_rate<=3 : 0.997173, loss: 0.001827, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18597 s, avg_samples: 56.0, ips: 301.11591 samples/s, eta: 1 day, 4:19:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:53:17] ppocr INFO: epoch: [307/500], global_step: 18900, lr: 0.000100, edit distance: 0.001585, exp_rate: 0.928341, exp_rate<=1 : 0.971189, exp_rate<=2 : 0.992859, exp_rate<=3 : 0.997235, loss: 0.001758, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17700 s, avg_samples: 56.0, ips: 316.38545 samples/s, eta: 1 day, 4:18:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:53:36] ppocr INFO: epoch: [307/500], global_step: 19000, lr: 0.000100, edit distance: 0.001598, exp_rate: 0.928231, exp_rate<=1 : 0.971210, exp_rate<=2 : 0.992905, exp_rate<=3 : 0.997274, loss: 0.002846, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18551 s, avg_samples: 56.0, ips: 301.87799 samples/s, eta: 1 day, 4:18:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:53:54] ppocr INFO: epoch: [307/500], global_step: 19100, lr: 0.000100, edit distance: 0.001596, exp_rate: 0.928255, exp_rate<=1 : 0.971298, exp_rate<=2 : 0.992961, exp_rate<=3 : 0.997333, loss: 0.001060, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18300 s, avg_samples: 56.0, ips: 306.01639 samples/s, eta: 1 day, 4:17:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:54:10] ppocr INFO: epoch: [307/500], global_step: 19180, lr: 0.000100, edit distance: 0.001603, exp_rate: 0.927856, exp_rate<=1 : 0.971161, exp_rate<=2 : 0.992922, exp_rate<=3 : 0.997355, loss: 0.001794, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.15594 s, avg_samples: 44.8, ips: 287.29906 samples/s, eta: 1 day, 4:17:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:54:12] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 03:54:17] ppocr INFO: epoch: [308/500], global_step: 19200, lr: 0.000100, edit distance: 0.001547, exp_rate: 0.921397, exp_rate<=1 : 0.964286, exp_rate<=2 : 0.988690, exp_rate<=3 : 0.998569, loss: 0.002684, avg_reader_cost: 0.02512 s, avg_batch_cost: 0.06881 s, avg_samples: 11.2, ips: 162.76339 samples/s, eta: 1 day, 4:18:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:54:35] ppocr INFO: epoch: [308/500], global_step: 19300, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.933968, exp_rate<=1 : 0.974547, exp_rate<=2 : 0.994257, exp_rate<=3 : 0.997576, loss: 0.001428, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18277 s, avg_samples: 56.0, ips: 306.38886 samples/s, eta: 1 day, 4:17:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:54:54] ppocr INFO: epoch: [308/500], global_step: 19400, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.936612, exp_rate<=1 : 0.976388, exp_rate<=2 : 0.994545, exp_rate<=3 : 0.997636, loss: 0.002218, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19302 s, avg_samples: 56.0, ips: 290.12316 samples/s, eta: 1 day, 4:17:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:55:12] ppocr INFO: epoch: [308/500], global_step: 19500, lr: 0.000100, edit distance: 0.001393, exp_rate: 0.935227, exp_rate<=1 : 0.975428, exp_rate<=2 : 0.994620, exp_rate<=3 : 0.997506, loss: 0.001554, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.17832 s, avg_samples: 56.0, ips: 314.05056 samples/s, eta: 1 day, 4:16:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:55:32] ppocr INFO: epoch: [308/500], global_step: 19600, lr: 0.000100, edit distance: 0.001463, exp_rate: 0.935649, exp_rate<=1 : 0.975492, exp_rate<=2 : 0.994739, exp_rate<=3 : 0.997760, loss: 0.001923, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19440 s, avg_samples: 56.0, ips: 288.07138 samples/s, eta: 1 day, 4:16:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:55:51] ppocr INFO: epoch: [308/500], global_step: 19700, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.934964, exp_rate<=1 : 0.974607, exp_rate<=2 : 0.994347, exp_rate<=3 : 0.997492, loss: 0.001463, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19728 s, avg_samples: 56.0, ips: 283.86146 samples/s, eta: 1 day, 4:16:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:56:11] ppocr INFO: epoch: [308/500], global_step: 19800, lr: 0.000100, edit distance: 0.001499, exp_rate: 0.934227, exp_rate<=1 : 0.974273, exp_rate<=2 : 0.994454, exp_rate<=3 : 0.997701, loss: 0.001502, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19894 s, avg_samples: 56.0, ips: 281.49631 samples/s, eta: 1 day, 4:15:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:56:31] ppocr INFO: epoch: [308/500], global_step: 19900, lr: 0.000100, edit distance: 0.001488, exp_rate: 0.932963, exp_rate<=1 : 0.973705, exp_rate<=2 : 0.994328, exp_rate<=3 : 0.997707, loss: 0.001353, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19936 s, avg_samples: 56.0, ips: 280.89916 samples/s, eta: 1 day, 4:15:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:56:51] ppocr INFO: epoch: [308/500], global_step: 20000, lr: 0.000100, edit distance: 0.001500, exp_rate: 0.933215, exp_rate<=1 : 0.973669, exp_rate<=2 : 0.994029, exp_rate<=3 : 0.997680, loss: 0.002005, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19825 s, avg_samples: 56.0, ips: 282.46715 samples/s, eta: 1 day, 4:15:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:57:10] ppocr INFO: epoch: [308/500], global_step: 20100, lr: 0.000100, edit distance: 0.001481, exp_rate: 0.933021, exp_rate<=1 : 0.973386, exp_rate<=2 : 0.993959, exp_rate<=3 : 0.997705, loss: 0.001263, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18665 s, avg_samples: 56.0, ips: 300.02498 samples/s, eta: 1 day, 4:15:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:57:29] ppocr INFO: epoch: [308/500], global_step: 20200, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.932879, exp_rate<=1 : 0.973533, exp_rate<=2 : 0.993841, exp_rate<=3 : 0.997633, loss: 0.001488, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19276 s, avg_samples: 56.0, ips: 290.52346 samples/s, eta: 1 day, 4:14:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:57:48] ppocr INFO: epoch: [308/500], global_step: 20300, lr: 0.000100, edit distance: 0.001511, exp_rate: 0.932819, exp_rate<=1 : 0.973483, exp_rate<=2 : 0.993742, exp_rate<=3 : 0.997706, loss: 0.001675, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19152 s, avg_samples: 56.0, ips: 292.39999 samples/s, eta: 1 day, 4:14:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:58:07] ppocr INFO: epoch: [308/500], global_step: 20400, lr: 0.000100, edit distance: 0.001493, exp_rate: 0.933340, exp_rate<=1 : 0.973702, exp_rate<=2 : 0.993730, exp_rate<=3 : 0.997744, loss: 0.001846, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18605 s, avg_samples: 56.0, ips: 300.99893 samples/s, eta: 1 day, 4:13:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:58:27] ppocr INFO: epoch: [308/500], global_step: 20500, lr: 0.000100, edit distance: 0.001493, exp_rate: 0.932478, exp_rate<=1 : 0.973409, exp_rate<=2 : 0.993681, exp_rate<=3 : 0.997702, loss: 0.002031, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20100 s, avg_samples: 56.0, ips: 278.60269 samples/s, eta: 1 day, 4:14:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:58:47] ppocr INFO: epoch: [308/500], global_step: 20600, lr: 0.000100, edit distance: 0.001485, exp_rate: 0.931985, exp_rate<=1 : 0.973236, exp_rate<=2 : 0.993680, exp_rate<=3 : 0.997630, loss: 0.001845, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19862 s, avg_samples: 56.0, ips: 281.94516 samples/s, eta: 1 day, 4:13:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:59:07] ppocr INFO: epoch: [308/500], global_step: 20700, lr: 0.000100, edit distance: 0.001532, exp_rate: 0.930539, exp_rate<=1 : 0.972505, exp_rate<=2 : 0.993277, exp_rate<=3 : 0.997503, loss: 0.002062, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19896 s, avg_samples: 56.0, ips: 281.45678 samples/s, eta: 1 day, 4:13:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:59:25] ppocr INFO: epoch: [308/500], global_step: 20800, lr: 0.000100, edit distance: 0.001566, exp_rate: 0.929652, exp_rate<=1 : 0.972131, exp_rate<=2 : 0.993045, exp_rate<=3 : 0.997445, loss: 0.002214, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18567 s, avg_samples: 56.0, ips: 301.61286 samples/s, eta: 1 day, 4:13:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 03:59:44] ppocr INFO: epoch: [308/500], global_step: 20900, lr: 0.000100, edit distance: 0.001572, exp_rate: 0.929586, exp_rate<=1 : 0.972194, exp_rate<=2 : 0.993068, exp_rate<=3 : 0.997451, loss: 0.002288, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18458 s, avg_samples: 56.0, ips: 303.39564 samples/s, eta: 1 day, 4:12:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:00:04] ppocr INFO: epoch: [308/500], global_step: 21000, lr: 0.000100, edit distance: 0.001582, exp_rate: 0.928611, exp_rate<=1 : 0.971694, exp_rate<=2 : 0.992858, exp_rate<=3 : 0.997329, loss: 0.001572, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20240 s, avg_samples: 56.0, ips: 276.67379 samples/s, eta: 1 day, 4:12:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:00:23] ppocr INFO: epoch: [308/500], global_step: 21100, lr: 0.000100, edit distance: 0.001592, exp_rate: 0.928468, exp_rate<=1 : 0.971670, exp_rate<=2 : 0.992882, exp_rate<=3 : 0.997312, loss: 0.001800, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19046 s, avg_samples: 56.0, ips: 294.01798 samples/s, eta: 1 day, 4:12:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:00:45] ppocr INFO: epoch: [308/500], global_step: 21200, lr: 0.000100, edit distance: 0.001598, exp_rate: 0.927501, exp_rate<=1 : 0.971220, exp_rate<=2 : 0.992652, exp_rate<=3 : 0.997241, loss: 0.001911, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.21483 s, avg_samples: 56.0, ips: 260.67245 samples/s, eta: 1 day, 4:12:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:01:04] ppocr INFO: epoch: [308/500], global_step: 21300, lr: 0.000100, edit distance: 0.001618, exp_rate: 0.926980, exp_rate<=1 : 0.970947, exp_rate<=2 : 0.992577, exp_rate<=3 : 0.997216, loss: 0.001666, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19065 s, avg_samples: 56.0, ips: 293.72717 samples/s, eta: 1 day, 4:12:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:01:22] ppocr INFO: epoch: [308/500], global_step: 21400, lr: 0.000100, edit distance: 0.001620, exp_rate: 0.926972, exp_rate<=1 : 0.970999, exp_rate<=2 : 0.992705, exp_rate<=3 : 0.997291, loss: 0.002156, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18606 s, avg_samples: 56.0, ips: 300.98370 samples/s, eta: 1 day, 4:11:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:01:42] ppocr INFO: epoch: [308/500], global_step: 21500, lr: 0.000100, edit distance: 0.001627, exp_rate: 0.926497, exp_rate<=1 : 0.970838, exp_rate<=2 : 0.992695, exp_rate<=3 : 0.997264, loss: 0.001568, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19668 s, avg_samples: 56.0, ips: 284.72378 samples/s, eta: 1 day, 4:11:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:01:59] ppocr INFO: epoch: [308/500], global_step: 21600, lr: 0.000100, edit distance: 0.001619, exp_rate: 0.927075, exp_rate<=1 : 0.971183, exp_rate<=2 : 0.992829, exp_rate<=3 : 0.997326, loss: 0.001355, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.16862 s, avg_samples: 56.0, ips: 332.11695 samples/s, eta: 1 day, 4:10:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:02:16] ppocr INFO: epoch: [308/500], global_step: 21700, lr: 0.000100, edit distance: 0.001618, exp_rate: 0.927362, exp_rate<=1 : 0.971283, exp_rate<=2 : 0.992918, exp_rate<=3 : 0.997383, loss: 0.001815, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17359 s, avg_samples: 56.0, ips: 322.60763 samples/s, eta: 1 day, 4:09:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:02:36] ppocr INFO: epoch: [308/500], global_step: 21800, lr: 0.000100, edit distance: 0.001621, exp_rate: 0.927116, exp_rate<=1 : 0.971208, exp_rate<=2 : 0.992923, exp_rate<=3 : 0.997401, loss: 0.001532, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19351 s, avg_samples: 56.0, ips: 289.39346 samples/s, eta: 1 day, 4:09:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:02:54] ppocr INFO: epoch: [308/500], global_step: 21900, lr: 0.000100, edit distance: 0.001611, exp_rate: 0.927366, exp_rate<=1 : 0.971355, exp_rate<=2 : 0.992977, exp_rate<=3 : 0.997403, loss: 0.001188, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18236 s, avg_samples: 56.0, ips: 307.08202 samples/s, eta: 1 day, 4:08:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:02:58] ppocr INFO: epoch: [308/500], global_step: 21920, lr: 0.000100, edit distance: 0.001611, exp_rate: 0.927424, exp_rate<=1 : 0.971385, exp_rate<=2 : 0.992989, exp_rate<=3 : 0.997408, loss: 0.001855, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.04400 s, avg_samples: 11.2, ips: 254.52952 samples/s, eta: 1 day, 4:08:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:03:00] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 04:03:17] ppocr INFO: epoch: [309/500], global_step: 22000, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.935190, exp_rate<=1 : 0.969770, exp_rate<=2 : 0.991940, exp_rate<=3 : 0.997214, loss: 0.001399, avg_reader_cost: 0.02617 s, avg_batch_cost: 0.18417 s, avg_samples: 44.8, ips: 243.25251 samples/s, eta: 1 day, 4:09:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:03:36] ppocr INFO: epoch: [309/500], global_step: 22100, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.939765, exp_rate<=1 : 0.975010, exp_rate<=2 : 0.994169, exp_rate<=3 : 0.997813, loss: 0.002179, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19646 s, avg_samples: 56.0, ips: 285.04344 samples/s, eta: 1 day, 4:09:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:03:56] ppocr INFO: epoch: [309/500], global_step: 22200, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.935767, exp_rate<=1 : 0.973066, exp_rate<=2 : 0.992936, exp_rate<=3 : 0.997217, loss: 0.001569, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19068 s, avg_samples: 56.0, ips: 293.68356 samples/s, eta: 1 day, 4:08:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:04:15] ppocr INFO: epoch: [309/500], global_step: 22300, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.936612, exp_rate<=1 : 0.974359, exp_rate<=2 : 0.993681, exp_rate<=3 : 0.997484, loss: 0.001344, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19163 s, avg_samples: 56.0, ips: 292.22967 samples/s, eta: 1 day, 4:08:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:04:32] ppocr INFO: epoch: [309/500], global_step: 22400, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.938070, exp_rate<=1 : 0.975153, exp_rate<=2 : 0.993872, exp_rate<=3 : 0.997625, loss: 0.001001, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17386 s, avg_samples: 56.0, ips: 322.10603 samples/s, eta: 1 day, 4:07:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:04:51] ppocr INFO: epoch: [309/500], global_step: 22500, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.937908, exp_rate<=1 : 0.974965, exp_rate<=2 : 0.993975, exp_rate<=3 : 0.997780, loss: 0.001644, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18408 s, avg_samples: 56.0, ips: 304.20890 samples/s, eta: 1 day, 4:06:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:05:10] ppocr INFO: epoch: [309/500], global_step: 22600, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.936467, exp_rate<=1 : 0.974101, exp_rate<=2 : 0.993475, exp_rate<=3 : 0.997390, loss: 0.001940, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19434 s, avg_samples: 56.0, ips: 288.15902 samples/s, eta: 1 day, 4:06:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:05:28] ppocr INFO: epoch: [309/500], global_step: 22700, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.937223, exp_rate<=1 : 0.974583, exp_rate<=2 : 0.993764, exp_rate<=3 : 0.997538, loss: 0.001501, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18076 s, avg_samples: 56.0, ips: 309.80444 samples/s, eta: 1 day, 4:05:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:05:48] ppocr INFO: epoch: [309/500], global_step: 22800, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.936044, exp_rate<=1 : 0.974443, exp_rate<=2 : 0.993738, exp_rate<=3 : 0.997622, loss: 0.001491, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19804 s, avg_samples: 56.0, ips: 282.77220 samples/s, eta: 1 day, 4:05:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:06:07] ppocr INFO: epoch: [309/500], global_step: 22900, lr: 0.000100, edit distance: 0.001413, exp_rate: 0.935311, exp_rate<=1 : 0.974003, exp_rate<=2 : 0.993727, exp_rate<=3 : 0.997595, loss: 0.001893, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18906 s, avg_samples: 56.0, ips: 296.20258 samples/s, eta: 1 day, 4:05:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:06:26] ppocr INFO: epoch: [309/500], global_step: 23000, lr: 0.000100, edit distance: 0.001431, exp_rate: 0.934894, exp_rate<=1 : 0.973657, exp_rate<=2 : 0.993786, exp_rate<=3 : 0.997651, loss: 0.001765, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19363 s, avg_samples: 56.0, ips: 289.21598 samples/s, eta: 1 day, 4:04:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:06:45] ppocr INFO: epoch: [309/500], global_step: 23100, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.934293, exp_rate<=1 : 0.973663, exp_rate<=2 : 0.993816, exp_rate<=3 : 0.997720, loss: 0.001572, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18698 s, avg_samples: 56.0, ips: 299.49049 samples/s, eta: 1 day, 4:04:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:07:04] ppocr INFO: epoch: [309/500], global_step: 23200, lr: 0.000100, edit distance: 0.001479, exp_rate: 0.933613, exp_rate<=1 : 0.973617, exp_rate<=2 : 0.993669, exp_rate<=3 : 0.997725, loss: 0.001770, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19238 s, avg_samples: 56.0, ips: 291.08438 samples/s, eta: 1 day, 4:04:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:07:24] ppocr INFO: epoch: [309/500], global_step: 23300, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.932588, exp_rate<=1 : 0.973136, exp_rate<=2 : 0.993493, exp_rate<=3 : 0.997669, loss: 0.001897, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19424 s, avg_samples: 56.0, ips: 288.29977 samples/s, eta: 1 day, 4:03:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:07:42] ppocr INFO: epoch: [309/500], global_step: 23400, lr: 0.000100, edit distance: 0.001503, exp_rate: 0.932490, exp_rate<=1 : 0.973092, exp_rate<=2 : 0.993524, exp_rate<=3 : 0.997678, loss: 0.001311, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18001 s, avg_samples: 56.0, ips: 311.09504 samples/s, eta: 1 day, 4:02:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:08:00] ppocr INFO: epoch: [309/500], global_step: 23500, lr: 0.000100, edit distance: 0.001504, exp_rate: 0.932614, exp_rate<=1 : 0.973098, exp_rate<=2 : 0.993573, exp_rate<=3 : 0.997693, loss: 0.001490, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18458 s, avg_samples: 56.0, ips: 303.39437 samples/s, eta: 1 day, 4:02:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:08:20] ppocr INFO: epoch: [309/500], global_step: 23600, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.932432, exp_rate<=1 : 0.972936, exp_rate<=2 : 0.993447, exp_rate<=3 : 0.997616, loss: 0.001447, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19399 s, avg_samples: 56.0, ips: 288.66789 samples/s, eta: 1 day, 4:02:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:08:41] ppocr INFO: epoch: [309/500], global_step: 23700, lr: 0.000100, edit distance: 0.001529, exp_rate: 0.931840, exp_rate<=1 : 0.972493, exp_rate<=2 : 0.993350, exp_rate<=3 : 0.997517, loss: 0.001609, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21159 s, avg_samples: 56.0, ips: 264.66170 samples/s, eta: 1 day, 4:02:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:09:00] ppocr INFO: epoch: [309/500], global_step: 23800, lr: 0.000100, edit distance: 0.001536, exp_rate: 0.931515, exp_rate<=1 : 0.972305, exp_rate<=2 : 0.993282, exp_rate<=3 : 0.997480, loss: 0.001948, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19181 s, avg_samples: 56.0, ips: 291.96198 samples/s, eta: 1 day, 4:02:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:09:19] ppocr INFO: epoch: [309/500], global_step: 23900, lr: 0.000100, edit distance: 0.001536, exp_rate: 0.931081, exp_rate<=1 : 0.972059, exp_rate<=2 : 0.993298, exp_rate<=3 : 0.997529, loss: 0.001444, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19336 s, avg_samples: 56.0, ips: 289.61525 samples/s, eta: 1 day, 4:01:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:09:39] ppocr INFO: epoch: [309/500], global_step: 24000, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.931121, exp_rate<=1 : 0.972103, exp_rate<=2 : 0.993267, exp_rate<=3 : 0.997506, loss: 0.002240, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19445 s, avg_samples: 56.0, ips: 287.98458 samples/s, eta: 1 day, 4:01:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:09:59] ppocr INFO: epoch: [309/500], global_step: 24100, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.930461, exp_rate<=1 : 0.971840, exp_rate<=2 : 0.993122, exp_rate<=3 : 0.997428, loss: 0.001799, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20167 s, avg_samples: 56.0, ips: 277.67544 samples/s, eta: 1 day, 4:01:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:10:18] ppocr INFO: epoch: [309/500], global_step: 24200, lr: 0.000100, edit distance: 0.001569, exp_rate: 0.929847, exp_rate<=1 : 0.971626, exp_rate<=2 : 0.993083, exp_rate<=3 : 0.997412, loss: 0.001810, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19405 s, avg_samples: 56.0, ips: 288.58000 samples/s, eta: 1 day, 4:01:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:10:37] ppocr INFO: epoch: [309/500], global_step: 24300, lr: 0.000100, edit distance: 0.001605, exp_rate: 0.929268, exp_rate<=1 : 0.971431, exp_rate<=2 : 0.993034, exp_rate<=3 : 0.997441, loss: 0.002677, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18066 s, avg_samples: 56.0, ips: 309.97450 samples/s, eta: 1 day, 4:00:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:10:56] ppocr INFO: epoch: [309/500], global_step: 24400, lr: 0.000100, edit distance: 0.001631, exp_rate: 0.928828, exp_rate<=1 : 0.971283, exp_rate<=2 : 0.992926, exp_rate<=3 : 0.997386, loss: 0.002111, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19498 s, avg_samples: 56.0, ips: 287.21137 samples/s, eta: 1 day, 4:00:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:11:16] ppocr INFO: epoch: [309/500], global_step: 24500, lr: 0.000100, edit distance: 0.001636, exp_rate: 0.928530, exp_rate<=1 : 0.971214, exp_rate<=2 : 0.992918, exp_rate<=3 : 0.997416, loss: 0.001257, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19952 s, avg_samples: 56.0, ips: 280.67318 samples/s, eta: 1 day, 4:00:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:11:35] ppocr INFO: epoch: [309/500], global_step: 24600, lr: 0.000100, edit distance: 0.001630, exp_rate: 0.928732, exp_rate<=1 : 0.971309, exp_rate<=2 : 0.993005, exp_rate<=3 : 0.997459, loss: 0.001654, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.74647 samples/s, eta: 1 day, 3:59:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:11:47] ppocr INFO: epoch: [309/500], global_step: 24660, lr: 0.000100, edit distance: 0.001629, exp_rate: 0.928705, exp_rate<=1 : 0.971300, exp_rate<=2 : 0.992983, exp_rate<=3 : 0.997463, loss: 0.001457, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.11487 s, avg_samples: 33.6, ips: 292.50651 samples/s, eta: 1 day, 3:59:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:11:49] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 04:11:57] ppocr INFO: epoch: [310/500], global_step: 24700, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.942702, exp_rate<=1 : 0.980194, exp_rate<=2 : 0.996567, exp_rate<=3 : 0.998120, loss: 0.002007, avg_reader_cost: 0.02621 s, avg_batch_cost: 0.09799 s, avg_samples: 22.4, ips: 228.59040 samples/s, eta: 1 day, 4:00:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:12:14] ppocr INFO: epoch: [310/500], global_step: 24800, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.941667, exp_rate<=1 : 0.979962, exp_rate<=2 : 0.994849, exp_rate<=3 : 0.997895, loss: 0.001533, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17711 s, avg_samples: 56.0, ips: 316.19235 samples/s, eta: 1 day, 3:59:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:12:35] ppocr INFO: epoch: [310/500], global_step: 24900, lr: 0.000100, edit distance: 0.001475, exp_rate: 0.935523, exp_rate<=1 : 0.975818, exp_rate<=2 : 0.993366, exp_rate<=3 : 0.997471, loss: 0.002170, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20223 s, avg_samples: 56.0, ips: 276.91181 samples/s, eta: 1 day, 3:59:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:12:55] ppocr INFO: epoch: [310/500], global_step: 25000, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.935065, exp_rate<=1 : 0.974668, exp_rate<=2 : 0.993501, exp_rate<=3 : 0.997693, loss: 0.001522, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20040 s, avg_samples: 56.0, ips: 279.43885 samples/s, eta: 1 day, 3:59:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:13:14] ppocr INFO: epoch: [310/500], global_step: 25100, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.934361, exp_rate<=1 : 0.974324, exp_rate<=2 : 0.993643, exp_rate<=3 : 0.997675, loss: 0.002405, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19424 s, avg_samples: 56.0, ips: 288.29777 samples/s, eta: 1 day, 3:59:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:13:33] ppocr INFO: epoch: [310/500], global_step: 25200, lr: 0.000100, edit distance: 0.001489, exp_rate: 0.932586, exp_rate<=1 : 0.973610, exp_rate<=2 : 0.993133, exp_rate<=3 : 0.997442, loss: 0.001955, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19243 s, avg_samples: 56.0, ips: 291.01746 samples/s, eta: 1 day, 3:58:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:13:53] ppocr INFO: epoch: [310/500], global_step: 25300, lr: 0.000100, edit distance: 0.001521, exp_rate: 0.931590, exp_rate<=1 : 0.973512, exp_rate<=2 : 0.992786, exp_rate<=3 : 0.997208, loss: 0.002230, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20081 s, avg_samples: 56.0, ips: 278.86627 samples/s, eta: 1 day, 3:58:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:14:12] ppocr INFO: epoch: [310/500], global_step: 25400, lr: 0.000100, edit distance: 0.001587, exp_rate: 0.931341, exp_rate<=1 : 0.973220, exp_rate<=2 : 0.992938, exp_rate<=3 : 0.997262, loss: 0.001604, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18291 s, avg_samples: 56.0, ips: 306.16887 samples/s, eta: 1 day, 3:58:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:14:31] ppocr INFO: epoch: [310/500], global_step: 25500, lr: 0.000100, edit distance: 0.001574, exp_rate: 0.931762, exp_rate<=1 : 0.973439, exp_rate<=2 : 0.993173, exp_rate<=3 : 0.997420, loss: 0.001415, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18814 s, avg_samples: 56.0, ips: 297.65470 samples/s, eta: 1 day, 3:57:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:14:50] ppocr INFO: epoch: [310/500], global_step: 25600, lr: 0.000100, edit distance: 0.001569, exp_rate: 0.931275, exp_rate<=1 : 0.973291, exp_rate<=2 : 0.993238, exp_rate<=3 : 0.997508, loss: 0.001581, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19784 s, avg_samples: 56.0, ips: 283.06095 samples/s, eta: 1 day, 3:57:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:15:09] ppocr INFO: epoch: [310/500], global_step: 25700, lr: 0.000100, edit distance: 0.001575, exp_rate: 0.930994, exp_rate<=1 : 0.973072, exp_rate<=2 : 0.993029, exp_rate<=3 : 0.997471, loss: 0.002190, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18999 s, avg_samples: 56.0, ips: 294.75218 samples/s, eta: 1 day, 3:57:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:15:29] ppocr INFO: epoch: [310/500], global_step: 25800, lr: 0.000100, edit distance: 0.001594, exp_rate: 0.930357, exp_rate<=1 : 0.972546, exp_rate<=2 : 0.992946, exp_rate<=3 : 0.997396, loss: 0.002031, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19672 s, avg_samples: 56.0, ips: 284.67250 samples/s, eta: 1 day, 3:56:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:15:47] ppocr INFO: epoch: [310/500], global_step: 25900, lr: 0.000100, edit distance: 0.001604, exp_rate: 0.930085, exp_rate<=1 : 0.972423, exp_rate<=2 : 0.993007, exp_rate<=3 : 0.997450, loss: 0.002103, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18315 s, avg_samples: 56.0, ips: 305.76104 samples/s, eta: 1 day, 3:56:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:16:08] ppocr INFO: epoch: [310/500], global_step: 26000, lr: 0.000100, edit distance: 0.001606, exp_rate: 0.929461, exp_rate<=1 : 0.972079, exp_rate<=2 : 0.992972, exp_rate<=3 : 0.997421, loss: 0.001687, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20365 s, avg_samples: 56.0, ips: 274.98586 samples/s, eta: 1 day, 3:56:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:16:28] ppocr INFO: epoch: [310/500], global_step: 26100, lr: 0.000100, edit distance: 0.001593, exp_rate: 0.929571, exp_rate<=1 : 0.972144, exp_rate<=2 : 0.992913, exp_rate<=3 : 0.997396, loss: 0.001334, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19833 s, avg_samples: 56.0, ips: 282.35895 samples/s, eta: 1 day, 3:56:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:16:45] ppocr INFO: epoch: [310/500], global_step: 26200, lr: 0.000100, edit distance: 0.001584, exp_rate: 0.930181, exp_rate<=1 : 0.972492, exp_rate<=2 : 0.993073, exp_rate<=3 : 0.997465, loss: 0.002176, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16899 s, avg_samples: 56.0, ips: 331.38978 samples/s, eta: 1 day, 3:55:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:17:03] ppocr INFO: epoch: [310/500], global_step: 26300, lr: 0.000100, edit distance: 0.001581, exp_rate: 0.930277, exp_rate<=1 : 0.972627, exp_rate<=2 : 0.993129, exp_rate<=3 : 0.997448, loss: 0.001265, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18340 s, avg_samples: 56.0, ips: 305.34179 samples/s, eta: 1 day, 3:54:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:17:22] ppocr INFO: epoch: [310/500], global_step: 26400, lr: 0.000100, edit distance: 0.001601, exp_rate: 0.929752, exp_rate<=1 : 0.972308, exp_rate<=2 : 0.993030, exp_rate<=3 : 0.997462, loss: 0.001857, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19440 s, avg_samples: 56.0, ips: 288.06365 samples/s, eta: 1 day, 3:54:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:17:41] ppocr INFO: epoch: [310/500], global_step: 26500, lr: 0.000100, edit distance: 0.001633, exp_rate: 0.929480, exp_rate<=1 : 0.972217, exp_rate<=2 : 0.993035, exp_rate<=3 : 0.997488, loss: 0.001720, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18456 s, avg_samples: 56.0, ips: 303.41951 samples/s, eta: 1 day, 3:53:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:18:00] ppocr INFO: epoch: [310/500], global_step: 26600, lr: 0.000100, edit distance: 0.001629, exp_rate: 0.929342, exp_rate<=1 : 0.972326, exp_rate<=2 : 0.993092, exp_rate<=3 : 0.997541, loss: 0.002013, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19055 s, avg_samples: 56.0, ips: 293.88213 samples/s, eta: 1 day, 3:53:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:18:20] ppocr INFO: epoch: [310/500], global_step: 26700, lr: 0.000100, edit distance: 0.001623, exp_rate: 0.928826, exp_rate<=1 : 0.972092, exp_rate<=2 : 0.993065, exp_rate<=3 : 0.997572, loss: 0.002705, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19756 s, avg_samples: 56.0, ips: 283.45222 samples/s, eta: 1 day, 3:53:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:18:39] ppocr INFO: epoch: [310/500], global_step: 26800, lr: 0.000100, edit distance: 0.001636, exp_rate: 0.928198, exp_rate<=1 : 0.971874, exp_rate<=2 : 0.993024, exp_rate<=3 : 0.997551, loss: 0.002338, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18799 s, avg_samples: 56.0, ips: 297.88842 samples/s, eta: 1 day, 3:52:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:18:58] ppocr INFO: epoch: [310/500], global_step: 26900, lr: 0.000100, edit distance: 0.001656, exp_rate: 0.927254, exp_rate<=1 : 0.971266, exp_rate<=2 : 0.992710, exp_rate<=3 : 0.997354, loss: 0.001598, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19791 s, avg_samples: 56.0, ips: 282.95835 samples/s, eta: 1 day, 3:52:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:19:18] ppocr INFO: epoch: [310/500], global_step: 27000, lr: 0.000100, edit distance: 0.001652, exp_rate: 0.927302, exp_rate<=1 : 0.971288, exp_rate<=2 : 0.992728, exp_rate<=3 : 0.997350, loss: 0.001528, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19229 s, avg_samples: 56.0, ips: 291.22011 samples/s, eta: 1 day, 3:52:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:19:36] ppocr INFO: epoch: [310/500], global_step: 27100, lr: 0.000100, edit distance: 0.001658, exp_rate: 0.927273, exp_rate<=1 : 0.971270, exp_rate<=2 : 0.992773, exp_rate<=3 : 0.997406, loss: 0.001662, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18309 s, avg_samples: 56.0, ips: 305.86693 samples/s, eta: 1 day, 3:51:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:19:55] ppocr INFO: epoch: [310/500], global_step: 27200, lr: 0.000100, edit distance: 0.001670, exp_rate: 0.927179, exp_rate<=1 : 0.971239, exp_rate<=2 : 0.992772, exp_rate<=3 : 0.997426, loss: 0.002242, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18874 s, avg_samples: 56.0, ips: 296.69852 samples/s, eta: 1 day, 3:51:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:20:14] ppocr INFO: epoch: [310/500], global_step: 27300, lr: 0.000100, edit distance: 0.001676, exp_rate: 0.926906, exp_rate<=1 : 0.971230, exp_rate<=2 : 0.992787, exp_rate<=3 : 0.997454, loss: 0.002067, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19197 s, avg_samples: 56.0, ips: 291.70588 samples/s, eta: 1 day, 3:50:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:20:33] ppocr INFO: epoch: [310/500], global_step: 27400, lr: 0.000100, edit distance: 0.001670, exp_rate: 0.927242, exp_rate<=1 : 0.971388, exp_rate<=2 : 0.992834, exp_rate<=3 : 0.997495, loss: 0.001906, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18623 s, avg_samples: 56.0, ips: 300.70791 samples/s, eta: 1 day, 3:50:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:20:36] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 04:20:36] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_310 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 04:20:57] ppocr INFO: epoch: [311/500], global_step: 27500, lr: 0.000100, edit distance: 0.001584, exp_rate: 0.924032, exp_rate<=1 : 0.970304, exp_rate<=2 : 0.992304, exp_rate<=3 : 0.996646, loss: 0.001263, avg_reader_cost: 0.04064 s, avg_batch_cost: 0.24570 s, avg_samples: 56.0, ips: 227.92144 samples/s, eta: 1 day, 3:51:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:21:16] ppocr INFO: epoch: [311/500], global_step: 27600, lr: 0.000100, edit distance: 0.001512, exp_rate: 0.933586, exp_rate<=1 : 0.973723, exp_rate<=2 : 0.993345, exp_rate<=3 : 0.997185, loss: 0.001713, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18493 s, avg_samples: 56.0, ips: 302.82062 samples/s, eta: 1 day, 3:51:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:21:35] ppocr INFO: epoch: [311/500], global_step: 27700, lr: 0.000100, edit distance: 0.001466, exp_rate: 0.935684, exp_rate<=1 : 0.974165, exp_rate<=2 : 0.993739, exp_rate<=3 : 0.997627, loss: 0.001703, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19414 s, avg_samples: 56.0, ips: 288.44646 samples/s, eta: 1 day, 3:50:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:21:55] ppocr INFO: epoch: [311/500], global_step: 27800, lr: 0.000100, edit distance: 0.001413, exp_rate: 0.936428, exp_rate<=1 : 0.975614, exp_rate<=2 : 0.994425, exp_rate<=3 : 0.997847, loss: 0.001554, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20103 s, avg_samples: 56.0, ips: 278.56212 samples/s, eta: 1 day, 3:50:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:22:14] ppocr INFO: epoch: [311/500], global_step: 27900, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.936701, exp_rate<=1 : 0.975855, exp_rate<=2 : 0.994503, exp_rate<=3 : 0.998034, loss: 0.001772, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19042 s, avg_samples: 56.0, ips: 294.08961 samples/s, eta: 1 day, 3:50:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:22:34] ppocr INFO: epoch: [311/500], global_step: 28000, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.937130, exp_rate<=1 : 0.976019, exp_rate<=2 : 0.994406, exp_rate<=3 : 0.997974, loss: 0.001188, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19245 s, avg_samples: 56.0, ips: 290.98725 samples/s, eta: 1 day, 3:49:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:22:53] ppocr INFO: epoch: [311/500], global_step: 28100, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.936075, exp_rate<=1 : 0.975186, exp_rate<=2 : 0.993726, exp_rate<=3 : 0.997567, loss: 0.001512, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19482 s, avg_samples: 56.0, ips: 287.44423 samples/s, eta: 1 day, 3:49:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:23:11] ppocr INFO: epoch: [311/500], global_step: 28200, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.936999, exp_rate<=1 : 0.975402, exp_rate<=2 : 0.993852, exp_rate<=3 : 0.997704, loss: 0.001311, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17796 s, avg_samples: 56.0, ips: 314.68426 samples/s, eta: 1 day, 3:48:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:23:31] ppocr INFO: epoch: [311/500], global_step: 28300, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.935213, exp_rate<=1 : 0.974900, exp_rate<=2 : 0.993603, exp_rate<=3 : 0.997589, loss: 0.001744, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19541 s, avg_samples: 56.0, ips: 286.57488 samples/s, eta: 1 day, 3:48:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:23:49] ppocr INFO: epoch: [311/500], global_step: 28400, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.934899, exp_rate<=1 : 0.974890, exp_rate<=2 : 0.993775, exp_rate<=3 : 0.997635, loss: 0.001947, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18328 s, avg_samples: 56.0, ips: 305.54743 samples/s, eta: 1 day, 3:48:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:24:07] ppocr INFO: epoch: [311/500], global_step: 28500, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.935125, exp_rate<=1 : 0.974970, exp_rate<=2 : 0.993906, exp_rate<=3 : 0.997729, loss: 0.001407, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18083 s, avg_samples: 56.0, ips: 309.68580 samples/s, eta: 1 day, 3:47:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:24:26] ppocr INFO: epoch: [311/500], global_step: 28600, lr: 0.000100, edit distance: 0.001497, exp_rate: 0.934955, exp_rate<=1 : 0.974904, exp_rate<=2 : 0.993888, exp_rate<=3 : 0.997623, loss: 0.001833, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18482 s, avg_samples: 56.0, ips: 302.98990 samples/s, eta: 1 day, 3:46:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:24:45] ppocr INFO: epoch: [311/500], global_step: 28700, lr: 0.000100, edit distance: 0.001521, exp_rate: 0.934473, exp_rate<=1 : 0.974650, exp_rate<=2 : 0.993767, exp_rate<=3 : 0.997532, loss: 0.001779, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19219 s, avg_samples: 56.0, ips: 291.38565 samples/s, eta: 1 day, 3:46:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:25:05] ppocr INFO: epoch: [311/500], global_step: 28800, lr: 0.000100, edit distance: 0.001546, exp_rate: 0.933295, exp_rate<=1 : 0.974313, exp_rate<=2 : 0.993751, exp_rate<=3 : 0.997518, loss: 0.002219, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20109 s, avg_samples: 56.0, ips: 278.47811 samples/s, eta: 1 day, 3:46:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:25:23] ppocr INFO: epoch: [311/500], global_step: 28900, lr: 0.000100, edit distance: 0.001544, exp_rate: 0.933429, exp_rate<=1 : 0.974391, exp_rate<=2 : 0.993786, exp_rate<=3 : 0.997568, loss: 0.001679, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18184 s, avg_samples: 56.0, ips: 307.95991 samples/s, eta: 1 day, 3:45:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:25:43] ppocr INFO: epoch: [311/500], global_step: 29000, lr: 0.000100, edit distance: 0.001551, exp_rate: 0.933040, exp_rate<=1 : 0.974050, exp_rate<=2 : 0.993748, exp_rate<=3 : 0.997577, loss: 0.002271, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19791 s, avg_samples: 56.0, ips: 282.95815 samples/s, eta: 1 day, 3:45:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:26:01] ppocr INFO: epoch: [311/500], global_step: 29100, lr: 0.000100, edit distance: 0.001553, exp_rate: 0.932976, exp_rate<=1 : 0.973930, exp_rate<=2 : 0.993742, exp_rate<=3 : 0.997546, loss: 0.001881, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17937 s, avg_samples: 56.0, ips: 312.20606 samples/s, eta: 1 day, 3:44:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:26:19] ppocr INFO: epoch: [311/500], global_step: 29200, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.932910, exp_rate<=1 : 0.973798, exp_rate<=2 : 0.993655, exp_rate<=3 : 0.997518, loss: 0.001947, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18525 s, avg_samples: 56.0, ips: 302.29748 samples/s, eta: 1 day, 3:44:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:26:39] ppocr INFO: epoch: [311/500], global_step: 29300, lr: 0.000100, edit distance: 0.001593, exp_rate: 0.932087, exp_rate<=1 : 0.973633, exp_rate<=2 : 0.993526, exp_rate<=3 : 0.997477, loss: 0.003273, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19474 s, avg_samples: 56.0, ips: 287.56599 samples/s, eta: 1 day, 3:44:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:27:01] ppocr INFO: epoch: [311/500], global_step: 29400, lr: 0.000100, edit distance: 0.001650, exp_rate: 0.929218, exp_rate<=1 : 0.972294, exp_rate<=2 : 0.993142, exp_rate<=3 : 0.997269, loss: 0.002058, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.22085 s, avg_samples: 56.0, ips: 253.56922 samples/s, eta: 1 day, 3:44:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:27:20] ppocr INFO: epoch: [311/500], global_step: 29500, lr: 0.000100, edit distance: 0.001645, exp_rate: 0.929204, exp_rate<=1 : 0.972288, exp_rate<=2 : 0.993115, exp_rate<=3 : 0.997254, loss: 0.002205, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18478 s, avg_samples: 56.0, ips: 303.05737 samples/s, eta: 1 day, 3:44:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:27:38] ppocr INFO: epoch: [311/500], global_step: 29600, lr: 0.000100, edit distance: 0.001644, exp_rate: 0.928938, exp_rate<=1 : 0.972206, exp_rate<=2 : 0.993140, exp_rate<=3 : 0.997277, loss: 0.001817, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18854 s, avg_samples: 56.0, ips: 297.02270 samples/s, eta: 1 day, 3:43:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:27:58] ppocr INFO: epoch: [311/500], global_step: 29700, lr: 0.000100, edit distance: 0.001634, exp_rate: 0.929042, exp_rate<=1 : 0.972389, exp_rate<=2 : 0.993168, exp_rate<=3 : 0.997294, loss: 0.001719, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19269 s, avg_samples: 56.0, ips: 290.62426 samples/s, eta: 1 day, 3:43:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:28:17] ppocr INFO: epoch: [311/500], global_step: 29800, lr: 0.000100, edit distance: 0.001638, exp_rate: 0.928725, exp_rate<=1 : 0.972247, exp_rate<=2 : 0.993190, exp_rate<=3 : 0.997304, loss: 0.001962, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19409 s, avg_samples: 56.0, ips: 288.53149 samples/s, eta: 1 day, 3:43:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:28:37] ppocr INFO: epoch: [311/500], global_step: 29900, lr: 0.000100, edit distance: 0.001634, exp_rate: 0.928521, exp_rate<=1 : 0.972234, exp_rate<=2 : 0.993164, exp_rate<=3 : 0.997307, loss: 0.001573, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19511 s, avg_samples: 56.0, ips: 287.01803 samples/s, eta: 1 day, 3:42:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:28:57] ppocr INFO: epoch: [311/500], global_step: 30000, lr: 0.000100, edit distance: 0.001634, exp_rate: 0.928278, exp_rate<=1 : 0.971992, exp_rate<=2 : 0.993073, exp_rate<=3 : 0.997308, loss: 0.002448, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20125 s, avg_samples: 56.0, ips: 278.26324 samples/s, eta: 1 day, 3:42:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:29:16] ppocr INFO: epoch: [311/500], global_step: 30100, lr: 0.000100, edit distance: 0.001641, exp_rate: 0.927706, exp_rate<=1 : 0.971579, exp_rate<=2 : 0.992955, exp_rate<=3 : 0.997244, loss: 0.002214, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19445 s, avg_samples: 56.0, ips: 287.99352 samples/s, eta: 1 day, 3:42:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:29:24] ppocr INFO: epoch: [311/500], global_step: 30140, lr: 0.000100, edit distance: 0.001642, exp_rate: 0.927527, exp_rate<=1 : 0.971472, exp_rate<=2 : 0.992906, exp_rate<=3 : 0.997251, loss: 0.002074, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.07743 s, avg_samples: 22.4, ips: 289.30529 samples/s, eta: 1 day, 3:42:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:29:26] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 04:29:39] ppocr INFO: epoch: [312/500], global_step: 30200, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.928750, exp_rate<=1 : 0.973635, exp_rate<=2 : 0.992718, exp_rate<=3 : 0.997171, loss: 0.001268, avg_reader_cost: 0.02598 s, avg_batch_cost: 0.14665 s, avg_samples: 33.6, ips: 229.11195 samples/s, eta: 1 day, 3:43:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:29:57] ppocr INFO: epoch: [312/500], global_step: 30300, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.938705, exp_rate<=1 : 0.977041, exp_rate<=2 : 0.994615, exp_rate<=3 : 0.997600, loss: 0.001933, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18103 s, avg_samples: 56.0, ips: 309.33710 samples/s, eta: 1 day, 3:42:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:30:14] ppocr INFO: epoch: [312/500], global_step: 30400, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.940774, exp_rate<=1 : 0.977925, exp_rate<=2 : 0.994939, exp_rate<=3 : 0.998096, loss: 0.001106, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17191 s, avg_samples: 56.0, ips: 325.74253 samples/s, eta: 1 day, 3:41:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:30:32] ppocr INFO: epoch: [312/500], global_step: 30500, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.937741, exp_rate<=1 : 0.977019, exp_rate<=2 : 0.994301, exp_rate<=3 : 0.997799, loss: 0.001176, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18292 s, avg_samples: 56.0, ips: 306.14575 samples/s, eta: 1 day, 3:40:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:30:52] ppocr INFO: epoch: [312/500], global_step: 30600, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.937108, exp_rate<=1 : 0.976659, exp_rate<=2 : 0.993625, exp_rate<=3 : 0.997549, loss: 0.001574, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19682 s, avg_samples: 56.0, ips: 284.52159 samples/s, eta: 1 day, 3:40:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:31:11] ppocr INFO: epoch: [312/500], global_step: 30700, lr: 0.000100, edit distance: 0.001416, exp_rate: 0.935954, exp_rate<=1 : 0.975471, exp_rate<=2 : 0.993352, exp_rate<=3 : 0.997479, loss: 0.001941, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18750 s, avg_samples: 56.0, ips: 298.66815 samples/s, eta: 1 day, 3:40:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:31:31] ppocr INFO: epoch: [312/500], global_step: 30800, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.935122, exp_rate<=1 : 0.975792, exp_rate<=2 : 0.993731, exp_rate<=3 : 0.997633, loss: 0.001655, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19994 s, avg_samples: 56.0, ips: 280.08749 samples/s, eta: 1 day, 3:40:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:31:52] ppocr INFO: epoch: [312/500], global_step: 30900, lr: 0.000100, edit distance: 0.001460, exp_rate: 0.932789, exp_rate<=1 : 0.974204, exp_rate<=2 : 0.993290, exp_rate<=3 : 0.997502, loss: 0.001668, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21128 s, avg_samples: 56.0, ips: 265.04546 samples/s, eta: 1 day, 3:40:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:32:11] ppocr INFO: epoch: [312/500], global_step: 31000, lr: 0.000100, edit distance: 0.001463, exp_rate: 0.933372, exp_rate<=1 : 0.974331, exp_rate<=2 : 0.993430, exp_rate<=3 : 0.997502, loss: 0.002444, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18870 s, avg_samples: 56.0, ips: 296.76452 samples/s, eta: 1 day, 3:39:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:32:29] ppocr INFO: epoch: [312/500], global_step: 31100, lr: 0.000100, edit distance: 0.001484, exp_rate: 0.933522, exp_rate<=1 : 0.974491, exp_rate<=2 : 0.993577, exp_rate<=3 : 0.997581, loss: 0.001415, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18549 s, avg_samples: 56.0, ips: 301.90229 samples/s, eta: 1 day, 3:39:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:32:49] ppocr INFO: epoch: [312/500], global_step: 31200, lr: 0.000100, edit distance: 0.001471, exp_rate: 0.933443, exp_rate<=1 : 0.974440, exp_rate<=2 : 0.993724, exp_rate<=3 : 0.997681, loss: 0.001602, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19111 s, avg_samples: 56.0, ips: 293.03092 samples/s, eta: 1 day, 3:39:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:33:07] ppocr INFO: epoch: [312/500], global_step: 31300, lr: 0.000100, edit distance: 0.001462, exp_rate: 0.934076, exp_rate<=1 : 0.974581, exp_rate<=2 : 0.993861, exp_rate<=3 : 0.997749, loss: 0.002025, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18572 s, avg_samples: 56.0, ips: 301.52892 samples/s, eta: 1 day, 3:38:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:33:27] ppocr INFO: epoch: [312/500], global_step: 31400, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.933005, exp_rate<=1 : 0.974077, exp_rate<=2 : 0.993646, exp_rate<=3 : 0.997721, loss: 0.001865, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19652 s, avg_samples: 56.0, ips: 284.95492 samples/s, eta: 1 day, 3:38:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:33:46] ppocr INFO: epoch: [312/500], global_step: 31500, lr: 0.000100, edit distance: 0.001473, exp_rate: 0.933248, exp_rate<=1 : 0.974037, exp_rate<=2 : 0.993595, exp_rate<=3 : 0.997755, loss: 0.001854, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18824 s, avg_samples: 56.0, ips: 297.48812 samples/s, eta: 1 day, 3:37:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:34:05] ppocr INFO: epoch: [312/500], global_step: 31600, lr: 0.000100, edit distance: 0.001489, exp_rate: 0.932670, exp_rate<=1 : 0.974043, exp_rate<=2 : 0.993602, exp_rate<=3 : 0.997728, loss: 0.002910, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19154 s, avg_samples: 56.0, ips: 292.35986 samples/s, eta: 1 day, 3:37:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:34:25] ppocr INFO: epoch: [312/500], global_step: 31700, lr: 0.000100, edit distance: 0.001498, exp_rate: 0.931714, exp_rate<=1 : 0.973669, exp_rate<=2 : 0.993554, exp_rate<=3 : 0.997659, loss: 0.001552, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20513 s, avg_samples: 56.0, ips: 273.00034 samples/s, eta: 1 day, 3:37:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:34:44] ppocr INFO: epoch: [312/500], global_step: 31800, lr: 0.000100, edit distance: 0.001496, exp_rate: 0.931834, exp_rate<=1 : 0.973579, exp_rate<=2 : 0.993627, exp_rate<=3 : 0.997686, loss: 0.001823, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18239 s, avg_samples: 56.0, ips: 307.02832 samples/s, eta: 1 day, 3:36:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:35:03] ppocr INFO: epoch: [312/500], global_step: 31900, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.931090, exp_rate<=1 : 0.973334, exp_rate<=2 : 0.993578, exp_rate<=3 : 0.997590, loss: 0.001711, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19485 s, avg_samples: 56.0, ips: 287.40582 samples/s, eta: 1 day, 3:36:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:35:21] ppocr INFO: epoch: [312/500], global_step: 32000, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.931526, exp_rate<=1 : 0.973577, exp_rate<=2 : 0.993664, exp_rate<=3 : 0.997629, loss: 0.001651, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17409 s, avg_samples: 56.0, ips: 321.67085 samples/s, eta: 1 day, 3:35:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:35:39] ppocr INFO: epoch: [312/500], global_step: 32100, lr: 0.000100, edit distance: 0.001512, exp_rate: 0.931616, exp_rate<=1 : 0.973512, exp_rate<=2 : 0.993656, exp_rate<=3 : 0.997653, loss: 0.001941, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18272 s, avg_samples: 56.0, ips: 306.47834 samples/s, eta: 1 day, 3:35:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:35:59] ppocr INFO: epoch: [312/500], global_step: 32200, lr: 0.000100, edit distance: 0.001512, exp_rate: 0.931397, exp_rate<=1 : 0.973423, exp_rate<=2 : 0.993657, exp_rate<=3 : 0.997692, loss: 0.002094, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19950 s, avg_samples: 56.0, ips: 280.70868 samples/s, eta: 1 day, 3:35:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:36:18] ppocr INFO: epoch: [312/500], global_step: 32300, lr: 0.000100, edit distance: 0.001512, exp_rate: 0.931391, exp_rate<=1 : 0.973432, exp_rate<=2 : 0.993721, exp_rate<=3 : 0.997703, loss: 0.002246, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19482 s, avg_samples: 56.0, ips: 287.44612 samples/s, eta: 1 day, 3:34:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:36:37] ppocr INFO: epoch: [312/500], global_step: 32400, lr: 0.000100, edit distance: 0.001522, exp_rate: 0.931186, exp_rate<=1 : 0.973325, exp_rate<=2 : 0.993692, exp_rate<=3 : 0.997699, loss: 0.001481, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18862 s, avg_samples: 56.0, ips: 296.88565 samples/s, eta: 1 day, 3:34:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:36:58] ppocr INFO: epoch: [312/500], global_step: 32500, lr: 0.000100, edit distance: 0.001522, exp_rate: 0.931003, exp_rate<=1 : 0.973250, exp_rate<=2 : 0.993714, exp_rate<=3 : 0.997708, loss: 0.001793, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20802 s, avg_samples: 56.0, ips: 269.20977 samples/s, eta: 1 day, 3:34:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:37:18] ppocr INFO: epoch: [312/500], global_step: 32600, lr: 0.000100, edit distance: 0.001527, exp_rate: 0.930671, exp_rate<=1 : 0.973054, exp_rate<=2 : 0.993640, exp_rate<=3 : 0.997629, loss: 0.001429, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20196 s, avg_samples: 56.0, ips: 277.27612 samples/s, eta: 1 day, 3:34:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:37:37] ppocr INFO: epoch: [312/500], global_step: 32700, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.930809, exp_rate<=1 : 0.973130, exp_rate<=2 : 0.993650, exp_rate<=3 : 0.997641, loss: 0.001766, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19120 s, avg_samples: 56.0, ips: 292.89205 samples/s, eta: 1 day, 3:34:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:37:57] ppocr INFO: epoch: [312/500], global_step: 32800, lr: 0.000100, edit distance: 0.001534, exp_rate: 0.929985, exp_rate<=1 : 0.972735, exp_rate<=2 : 0.993514, exp_rate<=3 : 0.997565, loss: 0.001826, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19478 s, avg_samples: 56.0, ips: 287.50427 samples/s, eta: 1 day, 3:33:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:38:12] ppocr INFO: epoch: [312/500], global_step: 32880, lr: 0.000100, edit distance: 0.001541, exp_rate: 0.929892, exp_rate<=1 : 0.972587, exp_rate<=2 : 0.993376, exp_rate<=3 : 0.997480, loss: 0.001306, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.15039 s, avg_samples: 44.8, ips: 297.90139 samples/s, eta: 1 day, 3:33:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:38:14] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 04:38:19] ppocr INFO: epoch: [313/500], global_step: 32900, lr: 0.000100, edit distance: 0.001044, exp_rate: 0.931548, exp_rate<=1 : 0.977232, exp_rate<=2 : 0.996540, exp_rate<=3 : 0.999554, loss: 0.001928, avg_reader_cost: 0.02845 s, avg_batch_cost: 0.07394 s, avg_samples: 11.2, ips: 151.46451 samples/s, eta: 1 day, 3:34:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:38:39] ppocr INFO: epoch: [313/500], global_step: 33000, lr: 0.000100, edit distance: 0.001498, exp_rate: 0.929934, exp_rate<=1 : 0.972372, exp_rate<=2 : 0.992214, exp_rate<=3 : 0.998173, loss: 0.001937, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19257 s, avg_samples: 56.0, ips: 290.80076 samples/s, eta: 1 day, 3:34:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:38:57] ppocr INFO: epoch: [313/500], global_step: 33100, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.933174, exp_rate<=1 : 0.973532, exp_rate<=2 : 0.992806, exp_rate<=3 : 0.997879, loss: 0.001234, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18378 s, avg_samples: 56.0, ips: 304.70637 samples/s, eta: 1 day, 3:33:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:39:16] ppocr INFO: epoch: [313/500], global_step: 33200, lr: 0.000100, edit distance: 0.001377, exp_rate: 0.936632, exp_rate<=1 : 0.974866, exp_rate<=2 : 0.993307, exp_rate<=3 : 0.997965, loss: 0.001221, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18767 s, avg_samples: 56.0, ips: 298.39151 samples/s, eta: 1 day, 3:33:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:39:34] ppocr INFO: epoch: [313/500], global_step: 33300, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.937148, exp_rate<=1 : 0.975144, exp_rate<=2 : 0.993491, exp_rate<=3 : 0.998048, loss: 0.001891, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18257 s, avg_samples: 56.0, ips: 306.73314 samples/s, eta: 1 day, 3:32:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:39:53] ppocr INFO: epoch: [313/500], global_step: 33400, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.936715, exp_rate<=1 : 0.974881, exp_rate<=2 : 0.993583, exp_rate<=3 : 0.998053, loss: 0.001177, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18464 s, avg_samples: 56.0, ips: 303.28877 samples/s, eta: 1 day, 3:31:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:40:12] ppocr INFO: epoch: [313/500], global_step: 33500, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.935339, exp_rate<=1 : 0.973930, exp_rate<=2 : 0.993071, exp_rate<=3 : 0.997782, loss: 0.001628, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19446 s, avg_samples: 56.0, ips: 287.97444 samples/s, eta: 1 day, 3:31:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:40:31] ppocr INFO: epoch: [313/500], global_step: 33600, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.935279, exp_rate<=1 : 0.974016, exp_rate<=2 : 0.993267, exp_rate<=3 : 0.997746, loss: 0.001599, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19139 s, avg_samples: 56.0, ips: 292.59603 samples/s, eta: 1 day, 3:31:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:40:51] ppocr INFO: epoch: [313/500], global_step: 33700, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.934872, exp_rate<=1 : 0.973958, exp_rate<=2 : 0.993368, exp_rate<=3 : 0.997636, loss: 0.001327, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19199 s, avg_samples: 56.0, ips: 291.67748 samples/s, eta: 1 day, 3:30:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:41:10] ppocr INFO: epoch: [313/500], global_step: 33800, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.934130, exp_rate<=1 : 0.973905, exp_rate<=2 : 0.993552, exp_rate<=3 : 0.997705, loss: 0.001758, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19210 s, avg_samples: 56.0, ips: 291.51195 samples/s, eta: 1 day, 3:30:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:41:31] ppocr INFO: epoch: [313/500], global_step: 33900, lr: 0.000100, edit distance: 0.001400, exp_rate: 0.931941, exp_rate<=1 : 0.972958, exp_rate<=2 : 0.993007, exp_rate<=3 : 0.997586, loss: 0.001841, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21510 s, avg_samples: 56.0, ips: 260.34535 samples/s, eta: 1 day, 3:30:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:41:52] ppocr INFO: epoch: [313/500], global_step: 34000, lr: 0.000100, edit distance: 0.001428, exp_rate: 0.930963, exp_rate<=1 : 0.972726, exp_rate<=2 : 0.992843, exp_rate<=3 : 0.997517, loss: 0.001668, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20857 s, avg_samples: 56.0, ips: 268.49003 samples/s, eta: 1 day, 3:30:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:42:12] ppocr INFO: epoch: [313/500], global_step: 34100, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.931000, exp_rate<=1 : 0.972650, exp_rate<=2 : 0.992806, exp_rate<=3 : 0.997498, loss: 0.002477, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19933 s, avg_samples: 56.0, ips: 280.94648 samples/s, eta: 1 day, 3:30:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:42:30] ppocr INFO: epoch: [313/500], global_step: 34200, lr: 0.000100, edit distance: 0.001466, exp_rate: 0.930945, exp_rate<=1 : 0.972649, exp_rate<=2 : 0.992816, exp_rate<=3 : 0.997471, loss: 0.001904, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17903 s, avg_samples: 56.0, ips: 312.79247 samples/s, eta: 1 day, 3:30:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:42:49] ppocr INFO: epoch: [313/500], global_step: 34300, lr: 0.000100, edit distance: 0.001496, exp_rate: 0.930005, exp_rate<=1 : 0.972327, exp_rate<=2 : 0.992660, exp_rate<=3 : 0.997372, loss: 0.001885, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19246 s, avg_samples: 56.0, ips: 290.97173 samples/s, eta: 1 day, 3:29:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:43:09] ppocr INFO: epoch: [313/500], global_step: 34400, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.928165, exp_rate<=1 : 0.971669, exp_rate<=2 : 0.992482, exp_rate<=3 : 0.997282, loss: 0.001407, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19590 s, avg_samples: 56.0, ips: 285.85932 samples/s, eta: 1 day, 3:29:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:43:27] ppocr INFO: epoch: [313/500], global_step: 34500, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.927929, exp_rate<=1 : 0.971676, exp_rate<=2 : 0.992527, exp_rate<=3 : 0.997326, loss: 0.001898, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17972 s, avg_samples: 56.0, ips: 311.60095 samples/s, eta: 1 day, 3:28:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:43:46] ppocr INFO: epoch: [313/500], global_step: 34600, lr: 0.000100, edit distance: 0.001578, exp_rate: 0.927403, exp_rate<=1 : 0.971543, exp_rate<=2 : 0.992490, exp_rate<=3 : 0.997316, loss: 0.002135, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19198 s, avg_samples: 56.0, ips: 291.69001 samples/s, eta: 1 day, 3:28:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:44:04] ppocr INFO: epoch: [313/500], global_step: 34700, lr: 0.000100, edit distance: 0.001600, exp_rate: 0.926953, exp_rate<=1 : 0.971408, exp_rate<=2 : 0.992467, exp_rate<=3 : 0.997342, loss: 0.002339, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17610 s, avg_samples: 56.0, ips: 317.99508 samples/s, eta: 1 day, 3:27:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:44:23] ppocr INFO: epoch: [313/500], global_step: 34800, lr: 0.000100, edit distance: 0.001601, exp_rate: 0.926777, exp_rate<=1 : 0.971289, exp_rate<=2 : 0.992358, exp_rate<=3 : 0.997244, loss: 0.001837, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19048 s, avg_samples: 56.0, ips: 293.98782 samples/s, eta: 1 day, 3:27:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:44:42] ppocr INFO: epoch: [313/500], global_step: 34900, lr: 0.000100, edit distance: 0.001613, exp_rate: 0.926434, exp_rate<=1 : 0.971080, exp_rate<=2 : 0.992327, exp_rate<=3 : 0.997264, loss: 0.000961, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19582 s, avg_samples: 56.0, ips: 285.97642 samples/s, eta: 1 day, 3:27:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:45:01] ppocr INFO: epoch: [313/500], global_step: 35000, lr: 0.000100, edit distance: 0.001617, exp_rate: 0.926566, exp_rate<=1 : 0.971070, exp_rate<=2 : 0.992365, exp_rate<=3 : 0.997319, loss: 0.001487, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18213 s, avg_samples: 56.0, ips: 307.46488 samples/s, eta: 1 day, 3:26:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:45:20] ppocr INFO: epoch: [313/500], global_step: 35100, lr: 0.000100, edit distance: 0.001611, exp_rate: 0.926676, exp_rate<=1 : 0.971005, exp_rate<=2 : 0.992359, exp_rate<=3 : 0.997359, loss: 0.001982, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18853 s, avg_samples: 56.0, ips: 297.03528 samples/s, eta: 1 day, 3:26:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:45:39] ppocr INFO: epoch: [313/500], global_step: 35200, lr: 0.000100, edit distance: 0.001612, exp_rate: 0.926588, exp_rate<=1 : 0.971043, exp_rate<=2 : 0.992357, exp_rate<=3 : 0.997379, loss: 0.002392, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19276 s, avg_samples: 56.0, ips: 290.51173 samples/s, eta: 1 day, 3:25:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:45:59] ppocr INFO: epoch: [313/500], global_step: 35300, lr: 0.000100, edit distance: 0.001612, exp_rate: 0.926625, exp_rate<=1 : 0.971047, exp_rate<=2 : 0.992347, exp_rate<=3 : 0.997414, loss: 0.002198, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19873 s, avg_samples: 56.0, ips: 281.78705 samples/s, eta: 1 day, 3:25:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:46:19] ppocr INFO: epoch: [313/500], global_step: 35400, lr: 0.000100, edit distance: 0.001610, exp_rate: 0.926611, exp_rate<=1 : 0.971035, exp_rate<=2 : 0.992388, exp_rate<=3 : 0.997425, loss: 0.001642, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20147 s, avg_samples: 56.0, ips: 277.95708 samples/s, eta: 1 day, 3:25:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:46:38] ppocr INFO: epoch: [313/500], global_step: 35500, lr: 0.000100, edit distance: 0.001604, exp_rate: 0.926748, exp_rate<=1 : 0.971025, exp_rate<=2 : 0.992402, exp_rate<=3 : 0.997417, loss: 0.001460, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19292 s, avg_samples: 56.0, ips: 290.27066 samples/s, eta: 1 day, 3:25:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:46:57] ppocr INFO: epoch: [313/500], global_step: 35600, lr: 0.000100, edit distance: 0.001606, exp_rate: 0.926945, exp_rate<=1 : 0.971090, exp_rate<=2 : 0.992443, exp_rate<=3 : 0.997449, loss: 0.001628, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18375 s, avg_samples: 56.0, ips: 304.76697 samples/s, eta: 1 day, 3:24:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:47:00] ppocr INFO: epoch: [313/500], global_step: 35620, lr: 0.000100, edit distance: 0.001607, exp_rate: 0.926964, exp_rate<=1 : 0.971058, exp_rate<=2 : 0.992442, exp_rate<=3 : 0.997448, loss: 0.001713, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03684 s, avg_samples: 11.2, ips: 304.00681 samples/s, eta: 1 day, 3:24:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:47:02] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 04:47:18] ppocr INFO: epoch: [314/500], global_step: 35700, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.945203, exp_rate<=1 : 0.979812, exp_rate<=2 : 0.994951, exp_rate<=3 : 0.999727, loss: 0.001948, avg_reader_cost: 0.02589 s, avg_batch_cost: 0.18037 s, avg_samples: 44.8, ips: 248.38020 samples/s, eta: 1 day, 3:25:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:47:38] ppocr INFO: epoch: [314/500], global_step: 35800, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.943178, exp_rate<=1 : 0.978494, exp_rate<=2 : 0.995267, exp_rate<=3 : 0.999382, loss: 0.001436, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19132 s, avg_samples: 56.0, ips: 292.70665 samples/s, eta: 1 day, 3:24:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:47:57] ppocr INFO: epoch: [314/500], global_step: 35900, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.941948, exp_rate<=1 : 0.978262, exp_rate<=2 : 0.994369, exp_rate<=3 : 0.998634, loss: 0.001566, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19558 s, avg_samples: 56.0, ips: 286.32952 samples/s, eta: 1 day, 3:24:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:48:16] ppocr INFO: epoch: [314/500], global_step: 36000, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.940682, exp_rate<=1 : 0.977499, exp_rate<=2 : 0.994129, exp_rate<=3 : 0.998374, loss: 0.001463, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18913 s, avg_samples: 56.0, ips: 296.10006 samples/s, eta: 1 day, 3:24:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:48:36] ppocr INFO: epoch: [314/500], global_step: 36100, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.939792, exp_rate<=1 : 0.976481, exp_rate<=2 : 0.993973, exp_rate<=3 : 0.998220, loss: 0.002210, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19466 s, avg_samples: 56.0, ips: 287.67671 samples/s, eta: 1 day, 3:23:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:48:56] ppocr INFO: epoch: [314/500], global_step: 36200, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.935528, exp_rate<=1 : 0.975072, exp_rate<=2 : 0.993284, exp_rate<=3 : 0.998031, loss: 0.001766, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20144 s, avg_samples: 56.0, ips: 278.00425 samples/s, eta: 1 day, 3:23:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:49:15] ppocr INFO: epoch: [314/500], global_step: 36300, lr: 0.000100, edit distance: 0.001479, exp_rate: 0.933307, exp_rate<=1 : 0.974136, exp_rate<=2 : 0.992809, exp_rate<=3 : 0.997991, loss: 0.001687, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19305 s, avg_samples: 56.0, ips: 290.08761 samples/s, eta: 1 day, 3:23:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:49:33] ppocr INFO: epoch: [314/500], global_step: 36400, lr: 0.000100, edit distance: 0.001480, exp_rate: 0.933529, exp_rate<=1 : 0.973978, exp_rate<=2 : 0.992727, exp_rate<=3 : 0.998033, loss: 0.001616, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18362 s, avg_samples: 56.0, ips: 304.98193 samples/s, eta: 1 day, 3:22:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:49:53] ppocr INFO: epoch: [314/500], global_step: 36500, lr: 0.000100, edit distance: 0.001460, exp_rate: 0.933679, exp_rate<=1 : 0.973916, exp_rate<=2 : 0.992892, exp_rate<=3 : 0.998032, loss: 0.001668, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19254 s, avg_samples: 56.0, ips: 290.84721 samples/s, eta: 1 day, 3:22:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:50:11] ppocr INFO: epoch: [314/500], global_step: 36600, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.933609, exp_rate<=1 : 0.974022, exp_rate<=2 : 0.992994, exp_rate<=3 : 0.997961, loss: 0.001520, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18646 s, avg_samples: 56.0, ips: 300.33291 samples/s, eta: 1 day, 3:22:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:50:31] ppocr INFO: epoch: [314/500], global_step: 36700, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.934024, exp_rate<=1 : 0.974047, exp_rate<=2 : 0.993087, exp_rate<=3 : 0.997942, loss: 0.001837, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19441 s, avg_samples: 56.0, ips: 288.04629 samples/s, eta: 1 day, 3:21:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:50:49] ppocr INFO: epoch: [314/500], global_step: 36800, lr: 0.000100, edit distance: 0.001450, exp_rate: 0.934417, exp_rate<=1 : 0.974290, exp_rate<=2 : 0.993191, exp_rate<=3 : 0.997910, loss: 0.001065, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18536 s, avg_samples: 56.0, ips: 302.11846 samples/s, eta: 1 day, 3:21:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:51:09] ppocr INFO: epoch: [314/500], global_step: 36900, lr: 0.000100, edit distance: 0.001457, exp_rate: 0.934128, exp_rate<=1 : 0.974213, exp_rate<=2 : 0.993110, exp_rate<=3 : 0.997855, loss: 0.001809, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19552 s, avg_samples: 56.0, ips: 286.41033 samples/s, eta: 1 day, 3:20:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:51:29] ppocr INFO: epoch: [314/500], global_step: 37000, lr: 0.000100, edit distance: 0.001458, exp_rate: 0.933674, exp_rate<=1 : 0.973977, exp_rate<=2 : 0.993072, exp_rate<=3 : 0.997858, loss: 0.001416, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19891 s, avg_samples: 56.0, ips: 281.53099 samples/s, eta: 1 day, 3:20:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:51:48] ppocr INFO: epoch: [314/500], global_step: 37100, lr: 0.000100, edit distance: 0.001477, exp_rate: 0.932980, exp_rate<=1 : 0.973688, exp_rate<=2 : 0.992974, exp_rate<=3 : 0.997801, loss: 0.002423, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18763 s, avg_samples: 56.0, ips: 298.46454 samples/s, eta: 1 day, 3:20:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:52:07] ppocr INFO: epoch: [314/500], global_step: 37200, lr: 0.000100, edit distance: 0.001501, exp_rate: 0.932282, exp_rate<=1 : 0.973405, exp_rate<=2 : 0.992956, exp_rate<=3 : 0.997723, loss: 0.002330, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19507 s, avg_samples: 56.0, ips: 287.07424 samples/s, eta: 1 day, 3:20:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:52:26] ppocr INFO: epoch: [314/500], global_step: 37300, lr: 0.000100, edit distance: 0.001499, exp_rate: 0.931844, exp_rate<=1 : 0.973204, exp_rate<=2 : 0.992936, exp_rate<=3 : 0.997627, loss: 0.001253, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18805 s, avg_samples: 56.0, ips: 297.78945 samples/s, eta: 1 day, 3:19:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:52:46] ppocr INFO: epoch: [314/500], global_step: 37400, lr: 0.000100, edit distance: 0.001519, exp_rate: 0.931297, exp_rate<=1 : 0.972990, exp_rate<=2 : 0.992823, exp_rate<=3 : 0.997642, loss: 0.002275, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20058 s, avg_samples: 56.0, ips: 279.19715 samples/s, eta: 1 day, 3:19:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:53:05] ppocr INFO: epoch: [314/500], global_step: 37500, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.930949, exp_rate<=1 : 0.972845, exp_rate<=2 : 0.992814, exp_rate<=3 : 0.997596, loss: 0.001872, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18787 s, avg_samples: 56.0, ips: 298.08377 samples/s, eta: 1 day, 3:19:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:53:22] ppocr INFO: epoch: [314/500], global_step: 37600, lr: 0.000100, edit distance: 0.001534, exp_rate: 0.930933, exp_rate<=1 : 0.972835, exp_rate<=2 : 0.992900, exp_rate<=3 : 0.997594, loss: 0.002395, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16895 s, avg_samples: 56.0, ips: 331.45780 samples/s, eta: 1 day, 3:18:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:53:40] ppocr INFO: epoch: [314/500], global_step: 37700, lr: 0.000100, edit distance: 0.001548, exp_rate: 0.930698, exp_rate<=1 : 0.972893, exp_rate<=2 : 0.992980, exp_rate<=3 : 0.997663, loss: 0.001901, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18437 s, avg_samples: 56.0, ips: 303.74244 samples/s, eta: 1 day, 3:17:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:54:00] ppocr INFO: epoch: [314/500], global_step: 37800, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.930056, exp_rate<=1 : 0.972556, exp_rate<=2 : 0.992965, exp_rate<=3 : 0.997679, loss: 0.002228, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19991 s, avg_samples: 56.0, ips: 280.13279 samples/s, eta: 1 day, 3:17:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:54:19] ppocr INFO: epoch: [314/500], global_step: 37900, lr: 0.000100, edit distance: 0.001558, exp_rate: 0.929882, exp_rate<=1 : 0.972380, exp_rate<=2 : 0.992946, exp_rate<=3 : 0.997657, loss: 0.001904, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18757 s, avg_samples: 56.0, ips: 298.55083 samples/s, eta: 1 day, 3:17:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:54:40] ppocr INFO: epoch: [314/500], global_step: 38000, lr: 0.000100, edit distance: 0.001561, exp_rate: 0.929359, exp_rate<=1 : 0.972036, exp_rate<=2 : 0.992863, exp_rate<=3 : 0.997617, loss: 0.001705, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20545 s, avg_samples: 56.0, ips: 272.57527 samples/s, eta: 1 day, 3:17:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:54:59] ppocr INFO: epoch: [314/500], global_step: 38100, lr: 0.000100, edit distance: 0.001551, exp_rate: 0.929706, exp_rate<=1 : 0.972207, exp_rate<=2 : 0.992946, exp_rate<=3 : 0.997663, loss: 0.002598, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19647 s, avg_samples: 56.0, ips: 285.03773 samples/s, eta: 1 day, 3:16:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:55:17] ppocr INFO: epoch: [314/500], global_step: 38200, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.929712, exp_rate<=1 : 0.972175, exp_rate<=2 : 0.992984, exp_rate<=3 : 0.997680, loss: 0.001533, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17990 s, avg_samples: 56.0, ips: 311.29030 samples/s, eta: 1 day, 3:16:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:55:37] ppocr INFO: epoch: [314/500], global_step: 38300, lr: 0.000100, edit distance: 0.001560, exp_rate: 0.929559, exp_rate<=1 : 0.972169, exp_rate<=2 : 0.993002, exp_rate<=3 : 0.997680, loss: 0.002639, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19342 s, avg_samples: 56.0, ips: 289.52731 samples/s, eta: 1 day, 3:15:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:55:48] ppocr INFO: epoch: [314/500], global_step: 38360, lr: 0.000100, edit distance: 0.001574, exp_rate: 0.928990, exp_rate<=1 : 0.971927, exp_rate<=2 : 0.992924, exp_rate<=3 : 0.997661, loss: 0.002785, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.11810 s, avg_samples: 33.6, ips: 284.50225 samples/s, eta: 1 day, 3:15:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:55:51] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 04:55:59] ppocr INFO: epoch: [315/500], global_step: 38400, lr: 0.000100, edit distance: 0.001650, exp_rate: 0.934879, exp_rate<=1 : 0.970779, exp_rate<=2 : 0.992435, exp_rate<=3 : 0.998608, loss: 0.001661, avg_reader_cost: 0.02600 s, avg_batch_cost: 0.10028 s, avg_samples: 22.4, ips: 223.37906 samples/s, eta: 1 day, 3:16:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:56:18] ppocr INFO: epoch: [315/500], global_step: 38500, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.937500, exp_rate<=1 : 0.976792, exp_rate<=2 : 0.995067, exp_rate<=3 : 0.998768, loss: 0.001400, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19254 s, avg_samples: 56.0, ips: 290.85600 samples/s, eta: 1 day, 3:15:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:56:36] ppocr INFO: epoch: [315/500], global_step: 38600, lr: 0.000100, edit distance: 0.001553, exp_rate: 0.937056, exp_rate<=1 : 0.977178, exp_rate<=2 : 0.994799, exp_rate<=3 : 0.998439, loss: 0.001524, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18491 s, avg_samples: 56.0, ips: 302.85186 samples/s, eta: 1 day, 3:15:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:56:57] ppocr INFO: epoch: [315/500], global_step: 38700, lr: 0.000100, edit distance: 0.001558, exp_rate: 0.934337, exp_rate<=1 : 0.975613, exp_rate<=2 : 0.993538, exp_rate<=3 : 0.998049, loss: 0.001763, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20341 s, avg_samples: 56.0, ips: 275.30599 samples/s, eta: 1 day, 3:15:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:57:17] ppocr INFO: epoch: [315/500], global_step: 38800, lr: 0.000100, edit distance: 0.001504, exp_rate: 0.934282, exp_rate<=1 : 0.975273, exp_rate<=2 : 0.993743, exp_rate<=3 : 0.997878, loss: 0.002253, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19923 s, avg_samples: 56.0, ips: 281.08135 samples/s, eta: 1 day, 3:15:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:57:35] ppocr INFO: epoch: [315/500], global_step: 38900, lr: 0.000100, edit distance: 0.001492, exp_rate: 0.933930, exp_rate<=1 : 0.974641, exp_rate<=2 : 0.993824, exp_rate<=3 : 0.997840, loss: 0.001751, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18569 s, avg_samples: 56.0, ips: 301.58142 samples/s, eta: 1 day, 3:14:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:57:54] ppocr INFO: epoch: [315/500], global_step: 39000, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.934392, exp_rate<=1 : 0.974891, exp_rate<=2 : 0.993736, exp_rate<=3 : 0.997711, loss: 0.001094, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19061 s, avg_samples: 56.0, ips: 293.79348 samples/s, eta: 1 day, 3:14:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:58:15] ppocr INFO: epoch: [315/500], global_step: 39100, lr: 0.000100, edit distance: 0.001461, exp_rate: 0.933796, exp_rate<=1 : 0.974631, exp_rate<=2 : 0.993813, exp_rate<=3 : 0.997654, loss: 0.001808, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20452 s, avg_samples: 56.0, ips: 273.81705 samples/s, eta: 1 day, 3:14:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:58:34] ppocr INFO: epoch: [315/500], global_step: 39200, lr: 0.000100, edit distance: 0.001449, exp_rate: 0.934732, exp_rate<=1 : 0.974843, exp_rate<=2 : 0.993677, exp_rate<=3 : 0.997565, loss: 0.001130, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19154 s, avg_samples: 56.0, ips: 292.36439 samples/s, eta: 1 day, 3:13:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:58:53] ppocr INFO: epoch: [315/500], global_step: 39300, lr: 0.000100, edit distance: 0.001471, exp_rate: 0.934745, exp_rate<=1 : 0.974878, exp_rate<=2 : 0.993727, exp_rate<=3 : 0.997590, loss: 0.002199, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18927 s, avg_samples: 56.0, ips: 295.87584 samples/s, eta: 1 day, 3:13:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:59:12] ppocr INFO: epoch: [315/500], global_step: 39400, lr: 0.000100, edit distance: 0.001484, exp_rate: 0.934211, exp_rate<=1 : 0.974560, exp_rate<=2 : 0.993788, exp_rate<=3 : 0.997551, loss: 0.001761, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18901 s, avg_samples: 56.0, ips: 296.28029 samples/s, eta: 1 day, 3:13:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:59:29] ppocr INFO: epoch: [315/500], global_step: 39500, lr: 0.000100, edit distance: 0.001463, exp_rate: 0.934805, exp_rate<=1 : 0.974742, exp_rate<=2 : 0.993869, exp_rate<=3 : 0.997579, loss: 0.002073, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17672 s, avg_samples: 56.0, ips: 316.89337 samples/s, eta: 1 day, 3:12:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 04:59:49] ppocr INFO: epoch: [315/500], global_step: 39600, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.934362, exp_rate<=1 : 0.974750, exp_rate<=2 : 0.993914, exp_rate<=3 : 0.997614, loss: 0.002633, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19380 s, avg_samples: 56.0, ips: 288.96011 samples/s, eta: 1 day, 3:12:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:00:10] ppocr INFO: epoch: [315/500], global_step: 39700, lr: 0.000100, edit distance: 0.001490, exp_rate: 0.933071, exp_rate<=1 : 0.974218, exp_rate<=2 : 0.993650, exp_rate<=3 : 0.997484, loss: 0.001718, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20927 s, avg_samples: 56.0, ips: 267.60316 samples/s, eta: 1 day, 3:12:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:00:29] ppocr INFO: epoch: [315/500], global_step: 39800, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.932090, exp_rate<=1 : 0.973881, exp_rate<=2 : 0.993592, exp_rate<=3 : 0.997477, loss: 0.002008, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19399 s, avg_samples: 56.0, ips: 288.68210 samples/s, eta: 1 day, 3:11:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:00:48] ppocr INFO: epoch: [315/500], global_step: 39900, lr: 0.000100, edit distance: 0.001533, exp_rate: 0.931956, exp_rate<=1 : 0.973707, exp_rate<=2 : 0.993604, exp_rate<=3 : 0.997513, loss: 0.001694, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18914 s, avg_samples: 56.0, ips: 296.08161 samples/s, eta: 1 day, 3:11:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:01:06] ppocr INFO: epoch: [315/500], global_step: 40000, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.932170, exp_rate<=1 : 0.973792, exp_rate<=2 : 0.993654, exp_rate<=3 : 0.997587, loss: 0.002494, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18113 s, avg_samples: 56.0, ips: 309.16529 samples/s, eta: 1 day, 3:10:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:01:26] ppocr INFO: epoch: [315/500], global_step: 40100, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.931269, exp_rate<=1 : 0.973574, exp_rate<=2 : 0.993602, exp_rate<=3 : 0.997624, loss: 0.001868, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19825 s, avg_samples: 56.0, ips: 282.47116 samples/s, eta: 1 day, 3:10:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:01:45] ppocr INFO: epoch: [315/500], global_step: 40200, lr: 0.000100, edit distance: 0.001561, exp_rate: 0.930906, exp_rate<=1 : 0.973280, exp_rate<=2 : 0.993623, exp_rate<=3 : 0.997645, loss: 0.001830, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19103 s, avg_samples: 56.0, ips: 293.15125 samples/s, eta: 1 day, 3:10:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:02:07] ppocr INFO: epoch: [315/500], global_step: 40300, lr: 0.000100, edit distance: 0.001577, exp_rate: 0.929912, exp_rate<=1 : 0.972910, exp_rate<=2 : 0.993501, exp_rate<=3 : 0.997560, loss: 0.002594, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21550 s, avg_samples: 56.0, ips: 259.86296 samples/s, eta: 1 day, 3:10:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:02:25] ppocr INFO: epoch: [315/500], global_step: 40400, lr: 0.000100, edit distance: 0.001578, exp_rate: 0.929999, exp_rate<=1 : 0.972931, exp_rate<=2 : 0.993520, exp_rate<=3 : 0.997553, loss: 0.001608, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18144 s, avg_samples: 56.0, ips: 308.64616 samples/s, eta: 1 day, 3:09:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:02:45] ppocr INFO: epoch: [315/500], global_step: 40500, lr: 0.000100, edit distance: 0.001591, exp_rate: 0.928659, exp_rate<=1 : 0.972422, exp_rate<=2 : 0.993359, exp_rate<=3 : 0.997480, loss: 0.002066, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19964 s, avg_samples: 56.0, ips: 280.50098 samples/s, eta: 1 day, 3:09:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:03:04] ppocr INFO: epoch: [315/500], global_step: 40600, lr: 0.000100, edit distance: 0.001607, exp_rate: 0.928299, exp_rate<=1 : 0.972182, exp_rate<=2 : 0.993218, exp_rate<=3 : 0.997448, loss: 0.002805, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19164 s, avg_samples: 56.0, ips: 292.20847 samples/s, eta: 1 day, 3:09:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:03:22] ppocr INFO: epoch: [315/500], global_step: 40700, lr: 0.000100, edit distance: 0.001613, exp_rate: 0.928422, exp_rate<=1 : 0.972187, exp_rate<=2 : 0.993237, exp_rate<=3 : 0.997433, loss: 0.002036, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17904 s, avg_samples: 56.0, ips: 312.77673 samples/s, eta: 1 day, 3:08:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:03:40] ppocr INFO: epoch: [315/500], global_step: 40800, lr: 0.000100, edit distance: 0.001617, exp_rate: 0.928428, exp_rate<=1 : 0.972134, exp_rate<=2 : 0.993274, exp_rate<=3 : 0.997427, loss: 0.001812, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18143 s, avg_samples: 56.0, ips: 308.65524 samples/s, eta: 1 day, 3:08:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:03:59] ppocr INFO: epoch: [315/500], global_step: 40900, lr: 0.000100, edit distance: 0.001630, exp_rate: 0.928265, exp_rate<=1 : 0.972094, exp_rate<=2 : 0.993261, exp_rate<=3 : 0.997390, loss: 0.001938, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18654 s, avg_samples: 56.0, ips: 300.19844 samples/s, eta: 1 day, 3:07:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:04:18] ppocr INFO: epoch: [315/500], global_step: 41000, lr: 0.000100, edit distance: 0.001635, exp_rate: 0.927949, exp_rate<=1 : 0.971872, exp_rate<=2 : 0.993171, exp_rate<=3 : 0.997393, loss: 0.001650, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19245 s, avg_samples: 56.0, ips: 290.98083 samples/s, eta: 1 day, 3:07:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:04:37] ppocr INFO: epoch: [315/500], global_step: 41100, lr: 0.000100, edit distance: 0.001634, exp_rate: 0.927947, exp_rate<=1 : 0.971764, exp_rate<=2 : 0.993132, exp_rate<=3 : 0.997360, loss: 0.002229, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18923 s, avg_samples: 56.0, ips: 295.92994 samples/s, eta: 1 day, 3:07:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:04:40] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 05:04:40] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_315 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 05:05:00] ppocr INFO: epoch: [316/500], global_step: 41200, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.945435, exp_rate<=1 : 0.979058, exp_rate<=2 : 0.994401, exp_rate<=3 : 0.997951, loss: 0.001405, avg_reader_cost: 0.03574 s, avg_batch_cost: 0.22797 s, avg_samples: 56.0, ips: 245.64451 samples/s, eta: 1 day, 3:07:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:05:19] ppocr INFO: epoch: [316/500], global_step: 41300, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.944347, exp_rate<=1 : 0.979189, exp_rate<=2 : 0.994657, exp_rate<=3 : 0.998031, loss: 0.001530, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19009 s, avg_samples: 56.0, ips: 294.59262 samples/s, eta: 1 day, 3:07:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:05:38] ppocr INFO: epoch: [316/500], global_step: 41400, lr: 0.000100, edit distance: 0.001419, exp_rate: 0.941527, exp_rate<=1 : 0.976732, exp_rate<=2 : 0.993870, exp_rate<=3 : 0.997936, loss: 0.001557, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19208 s, avg_samples: 56.0, ips: 291.55085 samples/s, eta: 1 day, 3:06:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:05:57] ppocr INFO: epoch: [316/500], global_step: 41500, lr: 0.000100, edit distance: 0.001562, exp_rate: 0.938497, exp_rate<=1 : 0.975449, exp_rate<=2 : 0.993755, exp_rate<=3 : 0.997666, loss: 0.002304, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18519 s, avg_samples: 56.0, ips: 302.39129 samples/s, eta: 1 day, 3:06:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:06:16] ppocr INFO: epoch: [316/500], global_step: 41600, lr: 0.000100, edit distance: 0.001567, exp_rate: 0.937159, exp_rate<=1 : 0.975458, exp_rate<=2 : 0.993582, exp_rate<=3 : 0.997720, loss: 0.001747, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18975 s, avg_samples: 56.0, ips: 295.12390 samples/s, eta: 1 day, 3:05:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:06:35] ppocr INFO: epoch: [316/500], global_step: 41700, lr: 0.000100, edit distance: 0.001550, exp_rate: 0.936797, exp_rate<=1 : 0.975003, exp_rate<=2 : 0.993500, exp_rate<=3 : 0.997762, loss: 0.001897, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18718 s, avg_samples: 56.0, ips: 299.17713 samples/s, eta: 1 day, 3:05:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:06:53] ppocr INFO: epoch: [316/500], global_step: 41800, lr: 0.000100, edit distance: 0.001545, exp_rate: 0.935859, exp_rate<=1 : 0.974678, exp_rate<=2 : 0.993476, exp_rate<=3 : 0.997727, loss: 0.002013, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18491 s, avg_samples: 56.0, ips: 302.84592 samples/s, eta: 1 day, 3:05:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:07:13] ppocr INFO: epoch: [316/500], global_step: 41900, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.934813, exp_rate<=1 : 0.974241, exp_rate<=2 : 0.993077, exp_rate<=3 : 0.997505, loss: 0.001946, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20205 s, avg_samples: 56.0, ips: 277.16067 samples/s, eta: 1 day, 3:04:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:07:33] ppocr INFO: epoch: [316/500], global_step: 42000, lr: 0.000100, edit distance: 0.001533, exp_rate: 0.935716, exp_rate<=1 : 0.974650, exp_rate<=2 : 0.993217, exp_rate<=3 : 0.997589, loss: 0.001685, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19203 s, avg_samples: 56.0, ips: 291.61900 samples/s, eta: 1 day, 3:04:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:07:52] ppocr INFO: epoch: [316/500], global_step: 42100, lr: 0.000100, edit distance: 0.001528, exp_rate: 0.935168, exp_rate<=1 : 0.974594, exp_rate<=2 : 0.993368, exp_rate<=3 : 0.997623, loss: 0.001546, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19167 s, avg_samples: 56.0, ips: 292.16402 samples/s, eta: 1 day, 3:04:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:08:12] ppocr INFO: epoch: [316/500], global_step: 42200, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.933658, exp_rate<=1 : 0.973862, exp_rate<=2 : 0.993155, exp_rate<=3 : 0.997525, loss: 0.001854, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20146 s, avg_samples: 56.0, ips: 277.96727 samples/s, eta: 1 day, 3:04:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:08:31] ppocr INFO: epoch: [316/500], global_step: 42300, lr: 0.000100, edit distance: 0.001538, exp_rate: 0.934039, exp_rate<=1 : 0.974100, exp_rate<=2 : 0.993205, exp_rate<=3 : 0.997569, loss: 0.001797, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18836 s, avg_samples: 56.0, ips: 297.30746 samples/s, eta: 1 day, 3:03:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:08:50] ppocr INFO: epoch: [316/500], global_step: 42400, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.934611, exp_rate<=1 : 0.974287, exp_rate<=2 : 0.993240, exp_rate<=3 : 0.997562, loss: 0.001163, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19536 s, avg_samples: 56.0, ips: 286.65059 samples/s, eta: 1 day, 3:03:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:09:10] ppocr INFO: epoch: [316/500], global_step: 42500, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.934825, exp_rate<=1 : 0.974571, exp_rate<=2 : 0.993341, exp_rate<=3 : 0.997586, loss: 0.002204, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19316 s, avg_samples: 56.0, ips: 289.91286 samples/s, eta: 1 day, 3:03:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:09:28] ppocr INFO: epoch: [316/500], global_step: 42600, lr: 0.000100, edit distance: 0.001522, exp_rate: 0.934391, exp_rate<=1 : 0.974277, exp_rate<=2 : 0.993111, exp_rate<=3 : 0.997446, loss: 0.001580, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17938 s, avg_samples: 56.0, ips: 312.19183 samples/s, eta: 1 day, 3:02:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:09:48] ppocr INFO: epoch: [316/500], global_step: 42700, lr: 0.000100, edit distance: 0.001548, exp_rate: 0.933369, exp_rate<=1 : 0.973722, exp_rate<=2 : 0.992977, exp_rate<=3 : 0.997463, loss: 0.002128, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20066 s, avg_samples: 56.0, ips: 279.07704 samples/s, eta: 1 day, 3:02:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:10:06] ppocr INFO: epoch: [316/500], global_step: 42800, lr: 0.000100, edit distance: 0.001585, exp_rate: 0.932553, exp_rate<=1 : 0.973502, exp_rate<=2 : 0.992971, exp_rate<=3 : 0.997477, loss: 0.003212, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18098 s, avg_samples: 56.0, ips: 309.43272 samples/s, eta: 1 day, 3:01:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:10:24] ppocr INFO: epoch: [316/500], global_step: 42900, lr: 0.000100, edit distance: 0.001619, exp_rate: 0.931504, exp_rate<=1 : 0.973072, exp_rate<=2 : 0.992784, exp_rate<=3 : 0.997450, loss: 0.002331, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18077 s, avg_samples: 56.0, ips: 309.78206 samples/s, eta: 1 day, 3:01:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:10:41] ppocr INFO: epoch: [316/500], global_step: 43000, lr: 0.000100, edit distance: 0.001624, exp_rate: 0.931631, exp_rate<=1 : 0.973080, exp_rate<=2 : 0.992801, exp_rate<=3 : 0.997440, loss: 0.001926, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17455 s, avg_samples: 56.0, ips: 320.82639 samples/s, eta: 1 day, 3:00:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:11:00] ppocr INFO: epoch: [316/500], global_step: 43100, lr: 0.000100, edit distance: 0.001634, exp_rate: 0.931258, exp_rate<=1 : 0.972827, exp_rate<=2 : 0.992710, exp_rate<=3 : 0.997385, loss: 0.001693, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19097 s, avg_samples: 56.0, ips: 293.24462 samples/s, eta: 1 day, 3:00:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:11:19] ppocr INFO: epoch: [316/500], global_step: 43200, lr: 0.000100, edit distance: 0.001631, exp_rate: 0.931262, exp_rate<=1 : 0.972790, exp_rate<=2 : 0.992667, exp_rate<=3 : 0.997399, loss: 0.001385, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18662 s, avg_samples: 56.0, ips: 300.07250 samples/s, eta: 1 day, 2:59:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:11:39] ppocr INFO: epoch: [316/500], global_step: 43300, lr: 0.000100, edit distance: 0.001636, exp_rate: 0.930947, exp_rate<=1 : 0.972601, exp_rate<=2 : 0.992558, exp_rate<=3 : 0.997329, loss: 0.002169, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19799 s, avg_samples: 56.0, ips: 282.83934 samples/s, eta: 1 day, 2:59:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:11:59] ppocr INFO: epoch: [316/500], global_step: 43400, lr: 0.000100, edit distance: 0.001628, exp_rate: 0.930435, exp_rate<=1 : 0.972422, exp_rate<=2 : 0.992582, exp_rate<=3 : 0.997386, loss: 0.001370, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20008 s, avg_samples: 56.0, ips: 279.89310 samples/s, eta: 1 day, 2:59:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:12:20] ppocr INFO: epoch: [316/500], global_step: 43500, lr: 0.000100, edit distance: 0.001628, exp_rate: 0.929908, exp_rate<=1 : 0.972308, exp_rate<=2 : 0.992547, exp_rate<=3 : 0.997397, loss: 0.001986, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21366 s, avg_samples: 56.0, ips: 262.09990 samples/s, eta: 1 day, 2:59:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:12:40] ppocr INFO: epoch: [316/500], global_step: 43600, lr: 0.000100, edit distance: 0.001618, exp_rate: 0.930075, exp_rate<=1 : 0.972477, exp_rate<=2 : 0.992677, exp_rate<=3 : 0.997433, loss: 0.001720, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19385 s, avg_samples: 56.0, ips: 288.88521 samples/s, eta: 1 day, 2:59:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:12:59] ppocr INFO: epoch: [316/500], global_step: 43700, lr: 0.000100, edit distance: 0.001615, exp_rate: 0.930014, exp_rate<=1 : 0.972575, exp_rate<=2 : 0.992817, exp_rate<=3 : 0.997464, loss: 0.002542, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19483 s, avg_samples: 56.0, ips: 287.42816 samples/s, eta: 1 day, 2:58:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:13:18] ppocr INFO: epoch: [316/500], global_step: 43800, lr: 0.000100, edit distance: 0.001632, exp_rate: 0.929782, exp_rate<=1 : 0.972477, exp_rate<=2 : 0.992824, exp_rate<=3 : 0.997455, loss: 0.001794, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18374 s, avg_samples: 56.0, ips: 304.78391 samples/s, eta: 1 day, 2:58:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:13:26] ppocr INFO: epoch: [316/500], global_step: 43840, lr: 0.000100, edit distance: 0.001635, exp_rate: 0.929605, exp_rate<=1 : 0.972428, exp_rate<=2 : 0.992817, exp_rate<=3 : 0.997465, loss: 0.001851, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.07817 s, avg_samples: 22.4, ips: 286.55214 samples/s, eta: 1 day, 2:58:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:13:28] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 05:13:39] ppocr INFO: epoch: [317/500], global_step: 43900, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.949003, exp_rate<=1 : 0.980196, exp_rate<=2 : 0.997171, exp_rate<=3 : 0.999293, loss: 0.001651, avg_reader_cost: 0.02622 s, avg_batch_cost: 0.13364 s, avg_samples: 33.6, ips: 251.41509 samples/s, eta: 1 day, 2:58:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:13:59] ppocr INFO: epoch: [317/500], global_step: 44000, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.941481, exp_rate<=1 : 0.978852, exp_rate<=2 : 0.995652, exp_rate<=3 : 0.998925, loss: 0.001266, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20440 s, avg_samples: 56.0, ips: 273.97277 samples/s, eta: 1 day, 2:58:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:14:19] ppocr INFO: epoch: [317/500], global_step: 44100, lr: 0.000100, edit distance: 0.001169, exp_rate: 0.943227, exp_rate<=1 : 0.978382, exp_rate<=2 : 0.995159, exp_rate<=3 : 0.998574, loss: 0.001611, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19179 s, avg_samples: 56.0, ips: 291.98459 samples/s, eta: 1 day, 2:58:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:14:38] ppocr INFO: epoch: [317/500], global_step: 44200, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.940468, exp_rate<=1 : 0.977226, exp_rate<=2 : 0.994710, exp_rate<=3 : 0.998488, loss: 0.001448, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19615 s, avg_samples: 56.0, ips: 285.49461 samples/s, eta: 1 day, 2:57:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:14:56] ppocr INFO: epoch: [317/500], global_step: 44300, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.941349, exp_rate<=1 : 0.978226, exp_rate<=2 : 0.995239, exp_rate<=3 : 0.998533, loss: 0.001276, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17801 s, avg_samples: 56.0, ips: 314.58227 samples/s, eta: 1 day, 2:57:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:15:16] ppocr INFO: epoch: [317/500], global_step: 44400, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.939192, exp_rate<=1 : 0.977262, exp_rate<=2 : 0.995164, exp_rate<=3 : 0.998481, loss: 0.001457, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19905 s, avg_samples: 56.0, ips: 281.34277 samples/s, eta: 1 day, 2:57:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:15:34] ppocr INFO: epoch: [317/500], global_step: 44500, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.939578, exp_rate<=1 : 0.977254, exp_rate<=2 : 0.994722, exp_rate<=3 : 0.998363, loss: 0.002131, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18423 s, avg_samples: 56.0, ips: 303.96235 samples/s, eta: 1 day, 2:56:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:15:54] ppocr INFO: epoch: [317/500], global_step: 44600, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.938542, exp_rate<=1 : 0.977007, exp_rate<=2 : 0.994500, exp_rate<=3 : 0.998182, loss: 0.001740, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19225 s, avg_samples: 56.0, ips: 291.28503 samples/s, eta: 1 day, 2:56:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:16:13] ppocr INFO: epoch: [317/500], global_step: 44700, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.937542, exp_rate<=1 : 0.976439, exp_rate<=2 : 0.994289, exp_rate<=3 : 0.998070, loss: 0.002278, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19256 s, avg_samples: 56.0, ips: 290.81790 samples/s, eta: 1 day, 2:55:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:16:32] ppocr INFO: epoch: [317/500], global_step: 44800, lr: 0.000100, edit distance: 0.001435, exp_rate: 0.935857, exp_rate<=1 : 0.975795, exp_rate<=2 : 0.994161, exp_rate<=3 : 0.998009, loss: 0.002481, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18558 s, avg_samples: 56.0, ips: 301.75686 samples/s, eta: 1 day, 2:55:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:16:50] ppocr INFO: epoch: [317/500], global_step: 44900, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.935275, exp_rate<=1 : 0.975590, exp_rate<=2 : 0.994326, exp_rate<=3 : 0.998061, loss: 0.002673, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18575 s, avg_samples: 56.0, ips: 301.48136 samples/s, eta: 1 day, 2:54:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:17:09] ppocr INFO: epoch: [317/500], global_step: 45000, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.934978, exp_rate<=1 : 0.975476, exp_rate<=2 : 0.994345, exp_rate<=3 : 0.998066, loss: 0.002512, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18531 s, avg_samples: 56.0, ips: 302.20371 samples/s, eta: 1 day, 2:54:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:17:28] ppocr INFO: epoch: [317/500], global_step: 45100, lr: 0.000100, edit distance: 0.001503, exp_rate: 0.933581, exp_rate<=1 : 0.974809, exp_rate<=2 : 0.994124, exp_rate<=3 : 0.997892, loss: 0.001807, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19474 s, avg_samples: 56.0, ips: 287.56824 samples/s, eta: 1 day, 2:54:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:17:47] ppocr INFO: epoch: [317/500], global_step: 45200, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.932438, exp_rate<=1 : 0.974394, exp_rate<=2 : 0.993951, exp_rate<=3 : 0.997825, loss: 0.001714, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19146 s, avg_samples: 56.0, ips: 292.49144 samples/s, eta: 1 day, 2:53:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:18:07] ppocr INFO: epoch: [317/500], global_step: 45300, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.932523, exp_rate<=1 : 0.974332, exp_rate<=2 : 0.993968, exp_rate<=3 : 0.997839, loss: 0.001685, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.35600 samples/s, eta: 1 day, 2:53:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:18:27] ppocr INFO: epoch: [317/500], global_step: 45400, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.931887, exp_rate<=1 : 0.973841, exp_rate<=2 : 0.993749, exp_rate<=3 : 0.997698, loss: 0.001841, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19972 s, avg_samples: 56.0, ips: 280.39553 samples/s, eta: 1 day, 2:53:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:18:47] ppocr INFO: epoch: [317/500], global_step: 45500, lr: 0.000100, edit distance: 0.001524, exp_rate: 0.931732, exp_rate<=1 : 0.973865, exp_rate<=2 : 0.993791, exp_rate<=3 : 0.997696, loss: 0.001929, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20091 s, avg_samples: 56.0, ips: 278.72551 samples/s, eta: 1 day, 2:53:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:19:05] ppocr INFO: epoch: [317/500], global_step: 45600, lr: 0.000100, edit distance: 0.001526, exp_rate: 0.931826, exp_rate<=1 : 0.973882, exp_rate<=2 : 0.993793, exp_rate<=3 : 0.997664, loss: 0.002004, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18179 s, avg_samples: 56.0, ips: 308.04653 samples/s, eta: 1 day, 2:52:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:19:25] ppocr INFO: epoch: [317/500], global_step: 45700, lr: 0.000100, edit distance: 0.001532, exp_rate: 0.931500, exp_rate<=1 : 0.973669, exp_rate<=2 : 0.993692, exp_rate<=3 : 0.997655, loss: 0.001710, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20268 s, avg_samples: 56.0, ips: 276.29677 samples/s, eta: 1 day, 2:52:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:19:44] ppocr INFO: epoch: [317/500], global_step: 45800, lr: 0.000100, edit distance: 0.001537, exp_rate: 0.931437, exp_rate<=1 : 0.973496, exp_rate<=2 : 0.993641, exp_rate<=3 : 0.997636, loss: 0.001215, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18827 s, avg_samples: 56.0, ips: 297.45195 samples/s, eta: 1 day, 2:52:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:20:03] ppocr INFO: epoch: [317/500], global_step: 45900, lr: 0.000100, edit distance: 0.001533, exp_rate: 0.931935, exp_rate<=1 : 0.973684, exp_rate<=2 : 0.993766, exp_rate<=3 : 0.997675, loss: 0.001524, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18375 s, avg_samples: 56.0, ips: 304.76870 samples/s, eta: 1 day, 2:51:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:20:22] ppocr INFO: epoch: [317/500], global_step: 46000, lr: 0.000100, edit distance: 0.001534, exp_rate: 0.931501, exp_rate<=1 : 0.973554, exp_rate<=2 : 0.993764, exp_rate<=3 : 0.997695, loss: 0.001846, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19503 s, avg_samples: 56.0, ips: 287.13470 samples/s, eta: 1 day, 2:51:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:20:42] ppocr INFO: epoch: [317/500], global_step: 46100, lr: 0.000100, edit distance: 0.001537, exp_rate: 0.931034, exp_rate<=1 : 0.973202, exp_rate<=2 : 0.993656, exp_rate<=3 : 0.997691, loss: 0.001601, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20054 s, avg_samples: 56.0, ips: 279.24428 samples/s, eta: 1 day, 2:51:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:21:02] ppocr INFO: epoch: [317/500], global_step: 46200, lr: 0.000100, edit distance: 0.001535, exp_rate: 0.930581, exp_rate<=1 : 0.972809, exp_rate<=2 : 0.993497, exp_rate<=3 : 0.997622, loss: 0.001506, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20043 s, avg_samples: 56.0, ips: 279.40158 samples/s, eta: 1 day, 2:51:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:21:21] ppocr INFO: epoch: [317/500], global_step: 46300, lr: 0.000100, edit distance: 0.001537, exp_rate: 0.930375, exp_rate<=1 : 0.972739, exp_rate<=2 : 0.993481, exp_rate<=3 : 0.997612, loss: 0.002517, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19056 s, avg_samples: 56.0, ips: 293.87378 samples/s, eta: 1 day, 2:50:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:21:41] ppocr INFO: epoch: [317/500], global_step: 46400, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.929532, exp_rate<=1 : 0.972367, exp_rate<=2 : 0.993270, exp_rate<=3 : 0.997504, loss: 0.002333, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19672 s, avg_samples: 56.0, ips: 284.67312 samples/s, eta: 1 day, 2:50:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:21:59] ppocr INFO: epoch: [317/500], global_step: 46500, lr: 0.000100, edit distance: 0.001559, exp_rate: 0.929759, exp_rate<=1 : 0.972576, exp_rate<=2 : 0.993352, exp_rate<=3 : 0.997551, loss: 0.001438, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17441 s, avg_samples: 56.0, ips: 321.07512 samples/s, eta: 1 day, 2:49:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:22:13] ppocr INFO: epoch: [317/500], global_step: 46580, lr: 0.000100, edit distance: 0.001559, exp_rate: 0.929548, exp_rate<=1 : 0.972525, exp_rate<=2 : 0.993414, exp_rate<=3 : 0.997587, loss: 0.001744, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14261 s, avg_samples: 44.8, ips: 314.13533 samples/s, eta: 1 day, 2:49:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:22:15] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 05:22:19] ppocr INFO: epoch: [318/500], global_step: 46600, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.943395, exp_rate<=1 : 0.980587, exp_rate<=2 : 0.995536, exp_rate<=3 : 1.000000, loss: 0.001730, avg_reader_cost: 0.02539 s, avg_batch_cost: 0.06532 s, avg_samples: 11.2, ips: 171.47284 samples/s, eta: 1 day, 2:49:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:22:40] ppocr INFO: epoch: [318/500], global_step: 46700, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.930982, exp_rate<=1 : 0.973749, exp_rate<=2 : 0.993697, exp_rate<=3 : 0.997738, loss: 0.001120, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20675 s, avg_samples: 56.0, ips: 270.85915 samples/s, eta: 1 day, 2:49:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:22:59] ppocr INFO: epoch: [318/500], global_step: 46800, lr: 0.000100, edit distance: 0.001414, exp_rate: 0.930304, exp_rate<=1 : 0.973318, exp_rate<=2 : 0.993638, exp_rate<=3 : 0.997616, loss: 0.001959, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18983 s, avg_samples: 56.0, ips: 294.99946 samples/s, eta: 1 day, 2:49:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:23:19] ppocr INFO: epoch: [318/500], global_step: 46900, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.931925, exp_rate<=1 : 0.973317, exp_rate<=2 : 0.993666, exp_rate<=3 : 0.997486, loss: 0.001177, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19663 s, avg_samples: 56.0, ips: 284.79196 samples/s, eta: 1 day, 2:49:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:23:39] ppocr INFO: epoch: [318/500], global_step: 47000, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.931843, exp_rate<=1 : 0.972517, exp_rate<=2 : 0.993233, exp_rate<=3 : 0.997471, loss: 0.001216, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20273 s, avg_samples: 56.0, ips: 276.23155 samples/s, eta: 1 day, 2:48:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:23:58] ppocr INFO: epoch: [318/500], global_step: 47100, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.931750, exp_rate<=1 : 0.972333, exp_rate<=2 : 0.992836, exp_rate<=3 : 0.997342, loss: 0.001746, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19419 s, avg_samples: 56.0, ips: 288.38439 samples/s, eta: 1 day, 2:48:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:24:17] ppocr INFO: epoch: [318/500], global_step: 47200, lr: 0.000100, edit distance: 0.001441, exp_rate: 0.931545, exp_rate<=1 : 0.972524, exp_rate<=2 : 0.992898, exp_rate<=3 : 0.997276, loss: 0.001547, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18825 s, avg_samples: 56.0, ips: 297.46981 samples/s, eta: 1 day, 2:48:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:24:35] ppocr INFO: epoch: [318/500], global_step: 47300, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.933275, exp_rate<=1 : 0.973535, exp_rate<=2 : 0.993383, exp_rate<=3 : 0.997556, loss: 0.001467, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17865 s, avg_samples: 56.0, ips: 313.45987 samples/s, eta: 1 day, 2:47:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:24:54] ppocr INFO: epoch: [318/500], global_step: 47400, lr: 0.000100, edit distance: 0.001443, exp_rate: 0.933508, exp_rate<=1 : 0.973604, exp_rate<=2 : 0.993705, exp_rate<=3 : 0.997621, loss: 0.002093, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18430 s, avg_samples: 56.0, ips: 303.84531 samples/s, eta: 1 day, 2:47:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:25:13] ppocr INFO: epoch: [318/500], global_step: 47500, lr: 0.000100, edit distance: 0.001490, exp_rate: 0.931518, exp_rate<=1 : 0.972537, exp_rate<=2 : 0.993284, exp_rate<=3 : 0.997469, loss: 0.001609, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19738 s, avg_samples: 56.0, ips: 283.72125 samples/s, eta: 1 day, 2:46:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:25:32] ppocr INFO: epoch: [318/500], global_step: 47600, lr: 0.000100, edit distance: 0.001539, exp_rate: 0.930878, exp_rate<=1 : 0.972421, exp_rate<=2 : 0.993315, exp_rate<=3 : 0.997374, loss: 0.001886, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18697 s, avg_samples: 56.0, ips: 299.51962 samples/s, eta: 1 day, 2:46:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:25:51] ppocr INFO: epoch: [318/500], global_step: 47700, lr: 0.000100, edit distance: 0.001553, exp_rate: 0.930777, exp_rate<=1 : 0.972620, exp_rate<=2 : 0.993550, exp_rate<=3 : 0.997457, loss: 0.001565, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18834 s, avg_samples: 56.0, ips: 297.32832 samples/s, eta: 1 day, 2:46:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:26:11] ppocr INFO: epoch: [318/500], global_step: 47800, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.930599, exp_rate<=1 : 0.972425, exp_rate<=2 : 0.993456, exp_rate<=3 : 0.997523, loss: 0.002058, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19611 s, avg_samples: 56.0, ips: 285.54851 samples/s, eta: 1 day, 2:45:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:26:29] ppocr INFO: epoch: [318/500], global_step: 47900, lr: 0.000100, edit distance: 0.001545, exp_rate: 0.930950, exp_rate<=1 : 0.972402, exp_rate<=2 : 0.993533, exp_rate<=3 : 0.997597, loss: 0.001569, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18861 s, avg_samples: 56.0, ips: 296.91131 samples/s, eta: 1 day, 2:45:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:26:49] ppocr INFO: epoch: [318/500], global_step: 48000, lr: 0.000100, edit distance: 0.001541, exp_rate: 0.931490, exp_rate<=1 : 0.972687, exp_rate<=2 : 0.993536, exp_rate<=3 : 0.997617, loss: 0.001512, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19050 s, avg_samples: 56.0, ips: 293.95902 samples/s, eta: 1 day, 2:45:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:27:06] ppocr INFO: epoch: [318/500], global_step: 48100, lr: 0.000100, edit distance: 0.001550, exp_rate: 0.931602, exp_rate<=1 : 0.972710, exp_rate<=2 : 0.993651, exp_rate<=3 : 0.997692, loss: 0.002022, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17792 s, avg_samples: 56.0, ips: 314.74289 samples/s, eta: 1 day, 2:44:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:27:27] ppocr INFO: epoch: [318/500], global_step: 48200, lr: 0.000100, edit distance: 0.001547, exp_rate: 0.931153, exp_rate<=1 : 0.972485, exp_rate<=2 : 0.993504, exp_rate<=3 : 0.997628, loss: 0.001861, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20993 s, avg_samples: 56.0, ips: 266.75014 samples/s, eta: 1 day, 2:44:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:27:46] ppocr INFO: epoch: [318/500], global_step: 48300, lr: 0.000100, edit distance: 0.001555, exp_rate: 0.930979, exp_rate<=1 : 0.972330, exp_rate<=2 : 0.993543, exp_rate<=3 : 0.997599, loss: 0.001445, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18882 s, avg_samples: 56.0, ips: 296.57290 samples/s, eta: 1 day, 2:44:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:28:05] ppocr INFO: epoch: [318/500], global_step: 48400, lr: 0.000100, edit distance: 0.001553, exp_rate: 0.931345, exp_rate<=1 : 0.972357, exp_rate<=2 : 0.993550, exp_rate<=3 : 0.997584, loss: 0.001659, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18186 s, avg_samples: 56.0, ips: 307.92196 samples/s, eta: 1 day, 2:43:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:28:24] ppocr INFO: epoch: [318/500], global_step: 48500, lr: 0.000100, edit distance: 0.001555, exp_rate: 0.931176, exp_rate<=1 : 0.972288, exp_rate<=2 : 0.993431, exp_rate<=3 : 0.997584, loss: 0.001462, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19206 s, avg_samples: 56.0, ips: 291.57001 samples/s, eta: 1 day, 2:43:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:28:43] ppocr INFO: epoch: [318/500], global_step: 48600, lr: 0.000100, edit distance: 0.001559, exp_rate: 0.931043, exp_rate<=1 : 0.972350, exp_rate<=2 : 0.993447, exp_rate<=3 : 0.997578, loss: 0.001413, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18820 s, avg_samples: 56.0, ips: 297.56273 samples/s, eta: 1 day, 2:42:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:29:03] ppocr INFO: epoch: [318/500], global_step: 48700, lr: 0.000100, edit distance: 0.001551, exp_rate: 0.931081, exp_rate<=1 : 0.972407, exp_rate<=2 : 0.993450, exp_rate<=3 : 0.997608, loss: 0.001749, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20023 s, avg_samples: 56.0, ips: 279.67529 samples/s, eta: 1 day, 2:42:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:29:20] ppocr INFO: epoch: [318/500], global_step: 48800, lr: 0.000100, edit distance: 0.001553, exp_rate: 0.931506, exp_rate<=1 : 0.972549, exp_rate<=2 : 0.993442, exp_rate<=3 : 0.997611, loss: 0.001549, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17487 s, avg_samples: 56.0, ips: 320.23384 samples/s, eta: 1 day, 2:42:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:29:40] ppocr INFO: epoch: [318/500], global_step: 48900, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.931210, exp_rate<=1 : 0.972342, exp_rate<=2 : 0.993380, exp_rate<=3 : 0.997587, loss: 0.001278, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.20012 s, avg_samples: 56.0, ips: 279.83368 samples/s, eta: 1 day, 2:41:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:30:00] ppocr INFO: epoch: [318/500], global_step: 49000, lr: 0.000100, edit distance: 0.001560, exp_rate: 0.931185, exp_rate<=1 : 0.972377, exp_rate<=2 : 0.993354, exp_rate<=3 : 0.997560, loss: 0.001529, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20289 s, avg_samples: 56.0, ips: 276.00687 samples/s, eta: 1 day, 2:41:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:30:20] ppocr INFO: epoch: [318/500], global_step: 49100, lr: 0.000100, edit distance: 0.001568, exp_rate: 0.931211, exp_rate<=1 : 0.972519, exp_rate<=2 : 0.993373, exp_rate<=3 : 0.997609, loss: 0.002361, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19212 s, avg_samples: 56.0, ips: 291.48288 samples/s, eta: 1 day, 2:41:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:30:39] ppocr INFO: epoch: [318/500], global_step: 49200, lr: 0.000100, edit distance: 0.001580, exp_rate: 0.930803, exp_rate<=1 : 0.972340, exp_rate<=2 : 0.993331, exp_rate<=3 : 0.997586, loss: 0.001778, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19238 s, avg_samples: 56.0, ips: 291.08944 samples/s, eta: 1 day, 2:41:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:30:58] ppocr INFO: epoch: [318/500], global_step: 49300, lr: 0.000100, edit distance: 0.001576, exp_rate: 0.930756, exp_rate<=1 : 0.972342, exp_rate<=2 : 0.993384, exp_rate<=3 : 0.997623, loss: 0.002326, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18743 s, avg_samples: 56.0, ips: 298.77356 samples/s, eta: 1 day, 2:40:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:31:01] ppocr INFO: epoch: [318/500], global_step: 49320, lr: 0.000100, edit distance: 0.001577, exp_rate: 0.930609, exp_rate<=1 : 0.972289, exp_rate<=2 : 0.993369, exp_rate<=3 : 0.997622, loss: 0.002560, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03737 s, avg_samples: 11.2, ips: 299.69647 samples/s, eta: 1 day, 2:40:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:31:03] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 05:31:19] ppocr INFO: epoch: [319/500], global_step: 49400, lr: 0.000100, edit distance: 0.001652, exp_rate: 0.935440, exp_rate<=1 : 0.975385, exp_rate<=2 : 0.993203, exp_rate<=3 : 0.997664, loss: 0.001671, avg_reader_cost: 0.02556 s, avg_batch_cost: 0.17604 s, avg_samples: 44.8, ips: 254.48479 samples/s, eta: 1 day, 2:40:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:31:39] ppocr INFO: epoch: [319/500], global_step: 49500, lr: 0.000100, edit distance: 0.001551, exp_rate: 0.934646, exp_rate<=1 : 0.975654, exp_rate<=2 : 0.994627, exp_rate<=3 : 0.997682, loss: 0.001948, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19458 s, avg_samples: 56.0, ips: 287.79340 samples/s, eta: 1 day, 2:40:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:31:58] ppocr INFO: epoch: [319/500], global_step: 49600, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.933197, exp_rate<=1 : 0.974292, exp_rate<=2 : 0.994389, exp_rate<=3 : 0.997768, loss: 0.001844, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19229 s, avg_samples: 56.0, ips: 291.22672 samples/s, eta: 1 day, 2:40:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:32:17] ppocr INFO: epoch: [319/500], global_step: 49700, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.932767, exp_rate<=1 : 0.973321, exp_rate<=2 : 0.994470, exp_rate<=3 : 0.998013, loss: 0.001439, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19054 s, avg_samples: 56.0, ips: 293.89559 samples/s, eta: 1 day, 2:39:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:32:35] ppocr INFO: epoch: [319/500], global_step: 49800, lr: 0.000100, edit distance: 0.001578, exp_rate: 0.931803, exp_rate<=1 : 0.973310, exp_rate<=2 : 0.994611, exp_rate<=3 : 0.998075, loss: 0.002388, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18401 s, avg_samples: 56.0, ips: 304.33711 samples/s, eta: 1 day, 2:39:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:32:54] ppocr INFO: epoch: [319/500], global_step: 49900, lr: 0.000100, edit distance: 0.001573, exp_rate: 0.932554, exp_rate<=1 : 0.973952, exp_rate<=2 : 0.994668, exp_rate<=3 : 0.998065, loss: 0.001812, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18719 s, avg_samples: 56.0, ips: 299.15595 samples/s, eta: 1 day, 2:38:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:33:14] ppocr INFO: epoch: [319/500], global_step: 50000, lr: 0.000100, edit distance: 0.001627, exp_rate: 0.931287, exp_rate<=1 : 0.973674, exp_rate<=2 : 0.994488, exp_rate<=3 : 0.998029, loss: 0.001941, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19429 s, avg_samples: 56.0, ips: 288.23285 samples/s, eta: 1 day, 2:38:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:33:33] ppocr INFO: epoch: [319/500], global_step: 50100, lr: 0.000100, edit distance: 0.001592, exp_rate: 0.931844, exp_rate<=1 : 0.973904, exp_rate<=2 : 0.994569, exp_rate<=3 : 0.998071, loss: 0.001648, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19429 s, avg_samples: 56.0, ips: 288.23399 samples/s, eta: 1 day, 2:38:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:33:53] ppocr INFO: epoch: [319/500], global_step: 50200, lr: 0.000100, edit distance: 0.001590, exp_rate: 0.931618, exp_rate<=1 : 0.973726, exp_rate<=2 : 0.994492, exp_rate<=3 : 0.998122, loss: 0.002547, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19689 s, avg_samples: 56.0, ips: 284.42468 samples/s, eta: 1 day, 2:38:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:34:12] ppocr INFO: epoch: [319/500], global_step: 50300, lr: 0.000100, edit distance: 0.001580, exp_rate: 0.931280, exp_rate<=1 : 0.973418, exp_rate<=2 : 0.994343, exp_rate<=3 : 0.998031, loss: 0.001503, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19673 s, avg_samples: 56.0, ips: 284.65533 samples/s, eta: 1 day, 2:37:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:34:31] ppocr INFO: epoch: [319/500], global_step: 50400, lr: 0.000100, edit distance: 0.001547, exp_rate: 0.931504, exp_rate<=1 : 0.973273, exp_rate<=2 : 0.994133, exp_rate<=3 : 0.998025, loss: 0.001192, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18732 s, avg_samples: 56.0, ips: 298.95412 samples/s, eta: 1 day, 2:37:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:34:50] ppocr INFO: epoch: [319/500], global_step: 50500, lr: 0.000100, edit distance: 0.001532, exp_rate: 0.932023, exp_rate<=1 : 0.973294, exp_rate<=2 : 0.994093, exp_rate<=3 : 0.997925, loss: 0.001379, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19238 s, avg_samples: 56.0, ips: 291.09397 samples/s, eta: 1 day, 2:37:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:35:10] ppocr INFO: epoch: [319/500], global_step: 50600, lr: 0.000100, edit distance: 0.001534, exp_rate: 0.932521, exp_rate<=1 : 0.973618, exp_rate<=2 : 0.994141, exp_rate<=3 : 0.998004, loss: 0.001583, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19230 s, avg_samples: 56.0, ips: 291.21373 samples/s, eta: 1 day, 2:36:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:35:28] ppocr INFO: epoch: [319/500], global_step: 50700, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.932239, exp_rate<=1 : 0.973471, exp_rate<=2 : 0.994037, exp_rate<=3 : 0.997948, loss: 0.001488, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18094 s, avg_samples: 56.0, ips: 309.48838 samples/s, eta: 1 day, 2:36:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:35:49] ppocr INFO: epoch: [319/500], global_step: 50800, lr: 0.000100, edit distance: 0.001542, exp_rate: 0.931782, exp_rate<=1 : 0.973257, exp_rate<=2 : 0.993882, exp_rate<=3 : 0.997904, loss: 0.001772, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20851 s, avg_samples: 56.0, ips: 268.57119 samples/s, eta: 1 day, 2:36:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:36:07] ppocr INFO: epoch: [319/500], global_step: 50900, lr: 0.000100, edit distance: 0.001567, exp_rate: 0.931843, exp_rate<=1 : 0.973379, exp_rate<=2 : 0.993906, exp_rate<=3 : 0.997932, loss: 0.001829, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18088 s, avg_samples: 56.0, ips: 309.59796 samples/s, eta: 1 day, 2:35:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:36:27] ppocr INFO: epoch: [319/500], global_step: 51000, lr: 0.000100, edit distance: 0.001553, exp_rate: 0.931547, exp_rate<=1 : 0.973206, exp_rate<=2 : 0.993784, exp_rate<=3 : 0.997923, loss: 0.001565, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20593 s, avg_samples: 56.0, ips: 271.94255 samples/s, eta: 1 day, 2:35:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:36:45] ppocr INFO: epoch: [319/500], global_step: 51100, lr: 0.000100, edit distance: 0.001532, exp_rate: 0.932021, exp_rate<=1 : 0.973295, exp_rate<=2 : 0.993858, exp_rate<=3 : 0.997915, loss: 0.001162, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17653 s, avg_samples: 56.0, ips: 317.23451 samples/s, eta: 1 day, 2:34:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:37:03] ppocr INFO: epoch: [319/500], global_step: 51200, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.932471, exp_rate<=1 : 0.973588, exp_rate<=2 : 0.993947, exp_rate<=3 : 0.997954, loss: 0.002023, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17881 s, avg_samples: 56.0, ips: 313.18954 samples/s, eta: 1 day, 2:34:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:37:23] ppocr INFO: epoch: [319/500], global_step: 51300, lr: 0.000100, edit distance: 0.001538, exp_rate: 0.932112, exp_rate<=1 : 0.973608, exp_rate<=2 : 0.993915, exp_rate<=3 : 0.997959, loss: 0.001441, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19688 s, avg_samples: 56.0, ips: 284.43273 samples/s, eta: 1 day, 2:34:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:37:42] ppocr INFO: epoch: [319/500], global_step: 51400, lr: 0.000100, edit distance: 0.001538, exp_rate: 0.931897, exp_rate<=1 : 0.973450, exp_rate<=2 : 0.993842, exp_rate<=3 : 0.997919, loss: 0.001584, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19725 s, avg_samples: 56.0, ips: 283.89718 samples/s, eta: 1 day, 2:33:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:38:01] ppocr INFO: epoch: [319/500], global_step: 51500, lr: 0.000100, edit distance: 0.001552, exp_rate: 0.931514, exp_rate<=1 : 0.973323, exp_rate<=2 : 0.993875, exp_rate<=3 : 0.997890, loss: 0.002175, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18644 s, avg_samples: 56.0, ips: 300.36051 samples/s, eta: 1 day, 2:33:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:38:20] ppocr INFO: epoch: [319/500], global_step: 51600, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.931107, exp_rate<=1 : 0.973092, exp_rate<=2 : 0.993845, exp_rate<=3 : 0.997878, loss: 0.001800, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18440 s, avg_samples: 56.0, ips: 303.69309 samples/s, eta: 1 day, 2:33:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:38:39] ppocr INFO: epoch: [319/500], global_step: 51700, lr: 0.000100, edit distance: 0.001559, exp_rate: 0.930694, exp_rate<=1 : 0.972872, exp_rate<=2 : 0.993848, exp_rate<=3 : 0.997877, loss: 0.002919, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19684 s, avg_samples: 56.0, ips: 284.50092 samples/s, eta: 1 day, 2:32:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:38:59] ppocr INFO: epoch: [319/500], global_step: 51800, lr: 0.000100, edit distance: 0.001566, exp_rate: 0.930231, exp_rate<=1 : 0.972638, exp_rate<=2 : 0.993743, exp_rate<=3 : 0.997852, loss: 0.002235, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19935 s, avg_samples: 56.0, ips: 280.91088 samples/s, eta: 1 day, 2:32:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:39:18] ppocr INFO: epoch: [319/500], global_step: 51900, lr: 0.000100, edit distance: 0.001584, exp_rate: 0.929497, exp_rate<=1 : 0.972365, exp_rate<=2 : 0.993565, exp_rate<=3 : 0.997773, loss: 0.002334, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19233 s, avg_samples: 56.0, ips: 291.16132 samples/s, eta: 1 day, 2:32:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:39:37] ppocr INFO: epoch: [319/500], global_step: 52000, lr: 0.000100, edit distance: 0.001590, exp_rate: 0.929396, exp_rate<=1 : 0.972465, exp_rate<=2 : 0.993594, exp_rate<=3 : 0.997787, loss: 0.002172, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18153 s, avg_samples: 56.0, ips: 308.48306 samples/s, eta: 1 day, 2:31:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:39:48] ppocr INFO: epoch: [319/500], global_step: 52060, lr: 0.000100, edit distance: 0.001595, exp_rate: 0.928892, exp_rate<=1 : 0.972218, exp_rate<=2 : 0.993431, exp_rate<=3 : 0.997703, loss: 0.002462, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.11851 s, avg_samples: 33.6, ips: 283.52850 samples/s, eta: 1 day, 2:31:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:39:50] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 05:39:58] ppocr INFO: epoch: [320/500], global_step: 52100, lr: 0.000100, edit distance: 0.001441, exp_rate: 0.934428, exp_rate<=1 : 0.975800, exp_rate<=2 : 0.997732, exp_rate<=3 : 1.000000, loss: 0.001639, avg_reader_cost: 0.02415 s, avg_batch_cost: 0.09739 s, avg_samples: 22.4, ips: 230.00373 samples/s, eta: 1 day, 2:31:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:40:17] ppocr INFO: epoch: [320/500], global_step: 52200, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.935185, exp_rate<=1 : 0.974613, exp_rate<=2 : 0.995643, exp_rate<=3 : 0.998332, loss: 0.001600, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19073 s, avg_samples: 56.0, ips: 293.60861 samples/s, eta: 1 day, 2:31:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:40:36] ppocr INFO: epoch: [320/500], global_step: 52300, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.936665, exp_rate<=1 : 0.976357, exp_rate<=2 : 0.995487, exp_rate<=3 : 0.998352, loss: 0.001897, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18342 s, avg_samples: 56.0, ips: 305.30912 samples/s, eta: 1 day, 2:31:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:40:55] ppocr INFO: epoch: [320/500], global_step: 52400, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.933825, exp_rate<=1 : 0.975066, exp_rate<=2 : 0.994971, exp_rate<=3 : 0.998271, loss: 0.001715, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19423 s, avg_samples: 56.0, ips: 288.31598 samples/s, eta: 1 day, 2:30:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:41:15] ppocr INFO: epoch: [320/500], global_step: 52500, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.932757, exp_rate<=1 : 0.974493, exp_rate<=2 : 0.994715, exp_rate<=3 : 0.998117, loss: 0.001413, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19973 s, avg_samples: 56.0, ips: 280.37673 samples/s, eta: 1 day, 2:30:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:41:33] ppocr INFO: epoch: [320/500], global_step: 52600, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.934406, exp_rate<=1 : 0.975396, exp_rate<=2 : 0.994996, exp_rate<=3 : 0.998176, loss: 0.001811, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18036 s, avg_samples: 56.0, ips: 310.49158 samples/s, eta: 1 day, 2:30:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:41:52] ppocr INFO: epoch: [320/500], global_step: 52700, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.934651, exp_rate<=1 : 0.975206, exp_rate<=2 : 0.994690, exp_rate<=3 : 0.998090, loss: 0.001831, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18848 s, avg_samples: 56.0, ips: 297.11443 samples/s, eta: 1 day, 2:29:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:42:11] ppocr INFO: epoch: [320/500], global_step: 52800, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.934487, exp_rate<=1 : 0.975298, exp_rate<=2 : 0.994749, exp_rate<=3 : 0.998077, loss: 0.001633, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18654 s, avg_samples: 56.0, ips: 300.20992 samples/s, eta: 1 day, 2:29:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:42:29] ppocr INFO: epoch: [320/500], global_step: 52900, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.934227, exp_rate<=1 : 0.975324, exp_rate<=2 : 0.994776, exp_rate<=3 : 0.998129, loss: 0.001652, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18064 s, avg_samples: 56.0, ips: 310.01331 samples/s, eta: 1 day, 2:28:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:42:48] ppocr INFO: epoch: [320/500], global_step: 53000, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.933638, exp_rate<=1 : 0.975066, exp_rate<=2 : 0.994698, exp_rate<=3 : 0.998073, loss: 0.002172, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19403 s, avg_samples: 56.0, ips: 288.60962 samples/s, eta: 1 day, 2:28:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:43:09] ppocr INFO: epoch: [320/500], global_step: 53100, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.932799, exp_rate<=1 : 0.974479, exp_rate<=2 : 0.994441, exp_rate<=3 : 0.998015, loss: 0.001428, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20857 s, avg_samples: 56.0, ips: 268.49243 samples/s, eta: 1 day, 2:28:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:43:28] ppocr INFO: epoch: [320/500], global_step: 53200, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.933462, exp_rate<=1 : 0.974753, exp_rate<=2 : 0.994480, exp_rate<=3 : 0.998018, loss: 0.002165, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18748 s, avg_samples: 56.0, ips: 298.69866 samples/s, eta: 1 day, 2:27:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:43:47] ppocr INFO: epoch: [320/500], global_step: 53300, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.933516, exp_rate<=1 : 0.974436, exp_rate<=2 : 0.994371, exp_rate<=3 : 0.998036, loss: 0.001613, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19077 s, avg_samples: 56.0, ips: 293.54054 samples/s, eta: 1 day, 2:27:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:44:06] ppocr INFO: epoch: [320/500], global_step: 53400, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.933312, exp_rate<=1 : 0.974301, exp_rate<=2 : 0.994358, exp_rate<=3 : 0.998011, loss: 0.002030, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19230 s, avg_samples: 56.0, ips: 291.21866 samples/s, eta: 1 day, 2:27:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:44:25] ppocr INFO: epoch: [320/500], global_step: 53500, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.933153, exp_rate<=1 : 0.974193, exp_rate<=2 : 0.994383, exp_rate<=3 : 0.998032, loss: 0.001332, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18610 s, avg_samples: 56.0, ips: 300.91480 samples/s, eta: 1 day, 2:26:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:44:44] ppocr INFO: epoch: [320/500], global_step: 53600, lr: 0.000100, edit distance: 0.001448, exp_rate: 0.932850, exp_rate<=1 : 0.973861, exp_rate<=2 : 0.994329, exp_rate<=3 : 0.998094, loss: 0.001375, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18621 s, avg_samples: 56.0, ips: 300.73005 samples/s, eta: 1 day, 2:26:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:45:02] ppocr INFO: epoch: [320/500], global_step: 53700, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.932971, exp_rate<=1 : 0.974041, exp_rate<=2 : 0.994347, exp_rate<=3 : 0.998127, loss: 0.001182, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18656 s, avg_samples: 56.0, ips: 300.17899 samples/s, eta: 1 day, 2:26:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:45:23] ppocr INFO: epoch: [320/500], global_step: 53800, lr: 0.000100, edit distance: 0.001471, exp_rate: 0.932469, exp_rate<=1 : 0.973666, exp_rate<=2 : 0.994226, exp_rate<=3 : 0.998070, loss: 0.002227, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20923 s, avg_samples: 56.0, ips: 267.65104 samples/s, eta: 1 day, 2:25:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:45:43] ppocr INFO: epoch: [320/500], global_step: 53900, lr: 0.000100, edit distance: 0.001475, exp_rate: 0.932046, exp_rate<=1 : 0.973353, exp_rate<=2 : 0.994051, exp_rate<=3 : 0.997927, loss: 0.001638, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20156 s, avg_samples: 56.0, ips: 277.83100 samples/s, eta: 1 day, 2:25:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:46:03] ppocr INFO: epoch: [320/500], global_step: 54000, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.931847, exp_rate<=1 : 0.973126, exp_rate<=2 : 0.993829, exp_rate<=3 : 0.997837, loss: 0.001327, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19663 s, avg_samples: 56.0, ips: 284.79647 samples/s, eta: 1 day, 2:25:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:46:21] ppocr INFO: epoch: [320/500], global_step: 54100, lr: 0.000100, edit distance: 0.001463, exp_rate: 0.932360, exp_rate<=1 : 0.973296, exp_rate<=2 : 0.993932, exp_rate<=3 : 0.997900, loss: 0.001601, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18451 s, avg_samples: 56.0, ips: 303.50099 samples/s, eta: 1 day, 2:25:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:46:41] ppocr INFO: epoch: [320/500], global_step: 54200, lr: 0.000100, edit distance: 0.001479, exp_rate: 0.931847, exp_rate<=1 : 0.973101, exp_rate<=2 : 0.993790, exp_rate<=3 : 0.997810, loss: 0.001657, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19242 s, avg_samples: 56.0, ips: 291.03131 samples/s, eta: 1 day, 2:24:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:47:00] ppocr INFO: epoch: [320/500], global_step: 54300, lr: 0.000100, edit distance: 0.001503, exp_rate: 0.931223, exp_rate<=1 : 0.972818, exp_rate<=2 : 0.993683, exp_rate<=3 : 0.997709, loss: 0.001990, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18979 s, avg_samples: 56.0, ips: 295.06327 samples/s, eta: 1 day, 2:24:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:47:19] ppocr INFO: epoch: [320/500], global_step: 54400, lr: 0.000100, edit distance: 0.001510, exp_rate: 0.930950, exp_rate<=1 : 0.972625, exp_rate<=2 : 0.993643, exp_rate<=3 : 0.997685, loss: 0.002107, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19385 s, avg_samples: 56.0, ips: 288.88348 samples/s, eta: 1 day, 2:24:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:47:39] ppocr INFO: epoch: [320/500], global_step: 54500, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.930671, exp_rate<=1 : 0.972512, exp_rate<=2 : 0.993513, exp_rate<=3 : 0.997612, loss: 0.002207, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19908 s, avg_samples: 56.0, ips: 281.30047 samples/s, eta: 1 day, 2:23:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:47:58] ppocr INFO: epoch: [320/500], global_step: 54600, lr: 0.000100, edit distance: 0.001536, exp_rate: 0.930211, exp_rate<=1 : 0.972289, exp_rate<=2 : 0.993443, exp_rate<=3 : 0.997580, loss: 0.002864, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19048 s, avg_samples: 56.0, ips: 293.99173 samples/s, eta: 1 day, 2:23:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:48:17] ppocr INFO: epoch: [320/500], global_step: 54700, lr: 0.000100, edit distance: 0.001547, exp_rate: 0.929924, exp_rate<=1 : 0.972221, exp_rate<=2 : 0.993411, exp_rate<=3 : 0.997564, loss: 0.002351, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18625 s, avg_samples: 56.0, ips: 300.67597 samples/s, eta: 1 day, 2:23:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:48:36] ppocr INFO: epoch: [320/500], global_step: 54800, lr: 0.000100, edit distance: 0.001571, exp_rate: 0.929398, exp_rate<=1 : 0.972086, exp_rate<=2 : 0.993334, exp_rate<=3 : 0.997521, loss: 0.002719, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19089 s, avg_samples: 56.0, ips: 293.36990 samples/s, eta: 1 day, 2:22:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:48:38] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 05:48:39] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_320 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 05:48:57] ppocr INFO: epoch: [321/500], global_step: 54900, lr: 0.000100, edit distance: 0.001717, exp_rate: 0.930938, exp_rate<=1 : 0.974746, exp_rate<=2 : 0.994683, exp_rate<=3 : 0.998295, loss: 0.001171, avg_reader_cost: 0.03246 s, avg_batch_cost: 0.21447 s, avg_samples: 56.0, ips: 261.10950 samples/s, eta: 1 day, 2:22:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:49:15] ppocr INFO: epoch: [321/500], global_step: 55000, lr: 0.000100, edit distance: 0.001754, exp_rate: 0.933774, exp_rate<=1 : 0.976386, exp_rate<=2 : 0.995156, exp_rate<=3 : 0.998191, loss: 0.001754, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17878 s, avg_samples: 56.0, ips: 313.23288 samples/s, eta: 1 day, 2:22:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:49:34] ppocr INFO: epoch: [321/500], global_step: 55100, lr: 0.000100, edit distance: 0.001603, exp_rate: 0.937561, exp_rate<=1 : 0.977656, exp_rate<=2 : 0.995815, exp_rate<=3 : 0.998382, loss: 0.002325, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18544 s, avg_samples: 56.0, ips: 301.98678 samples/s, eta: 1 day, 2:21:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:49:52] ppocr INFO: epoch: [321/500], global_step: 55200, lr: 0.000100, edit distance: 0.001572, exp_rate: 0.936533, exp_rate<=1 : 0.976030, exp_rate<=2 : 0.995484, exp_rate<=3 : 0.998279, loss: 0.001465, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18512 s, avg_samples: 56.0, ips: 302.49954 samples/s, eta: 1 day, 2:21:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:50:11] ppocr INFO: epoch: [321/500], global_step: 55300, lr: 0.000100, edit distance: 0.001579, exp_rate: 0.936639, exp_rate<=1 : 0.975624, exp_rate<=2 : 0.995160, exp_rate<=3 : 0.998112, loss: 0.001490, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19046 s, avg_samples: 56.0, ips: 294.02437 samples/s, eta: 1 day, 2:21:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:50:31] ppocr INFO: epoch: [321/500], global_step: 55400, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.937017, exp_rate<=1 : 0.975123, exp_rate<=2 : 0.994853, exp_rate<=3 : 0.997952, loss: 0.001466, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19430 s, avg_samples: 56.0, ips: 288.21548 samples/s, eta: 1 day, 2:20:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:50:51] ppocr INFO: epoch: [321/500], global_step: 55500, lr: 0.000100, edit distance: 0.001563, exp_rate: 0.935476, exp_rate<=1 : 0.974756, exp_rate<=2 : 0.994837, exp_rate<=3 : 0.998030, loss: 0.002009, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19815 s, avg_samples: 56.0, ips: 282.61405 samples/s, eta: 1 day, 2:20:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:51:09] ppocr INFO: epoch: [321/500], global_step: 55600, lr: 0.000100, edit distance: 0.001603, exp_rate: 0.934665, exp_rate<=1 : 0.974506, exp_rate<=2 : 0.994583, exp_rate<=3 : 0.997885, loss: 0.001807, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18288 s, avg_samples: 56.0, ips: 306.20462 samples/s, eta: 1 day, 2:20:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:51:29] ppocr INFO: epoch: [321/500], global_step: 55700, lr: 0.000100, edit distance: 0.001600, exp_rate: 0.933298, exp_rate<=1 : 0.973702, exp_rate<=2 : 0.994256, exp_rate<=3 : 0.997764, loss: 0.002110, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19584 s, avg_samples: 56.0, ips: 285.95187 samples/s, eta: 1 day, 2:19:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:51:47] ppocr INFO: epoch: [321/500], global_step: 55800, lr: 0.000100, edit distance: 0.001559, exp_rate: 0.934165, exp_rate<=1 : 0.974052, exp_rate<=2 : 0.994329, exp_rate<=3 : 0.997842, loss: 0.001504, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18749 s, avg_samples: 56.0, ips: 298.67552 samples/s, eta: 1 day, 2:19:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:52:06] ppocr INFO: epoch: [321/500], global_step: 55900, lr: 0.000100, edit distance: 0.001551, exp_rate: 0.934477, exp_rate<=1 : 0.974085, exp_rate<=2 : 0.994370, exp_rate<=3 : 0.997885, loss: 0.002153, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18589 s, avg_samples: 56.0, ips: 301.26043 samples/s, eta: 1 day, 2:18:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:52:25] ppocr INFO: epoch: [321/500], global_step: 56000, lr: 0.000100, edit distance: 0.001541, exp_rate: 0.933787, exp_rate<=1 : 0.973777, exp_rate<=2 : 0.994361, exp_rate<=3 : 0.997944, loss: 0.001682, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19115 s, avg_samples: 56.0, ips: 292.95654 samples/s, eta: 1 day, 2:18:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:52:44] ppocr INFO: epoch: [321/500], global_step: 56100, lr: 0.000100, edit distance: 0.001549, exp_rate: 0.933300, exp_rate<=1 : 0.973630, exp_rate<=2 : 0.994257, exp_rate<=3 : 0.997841, loss: 0.001902, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19224 s, avg_samples: 56.0, ips: 291.30533 samples/s, eta: 1 day, 2:18:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:53:04] ppocr INFO: epoch: [321/500], global_step: 56200, lr: 0.000100, edit distance: 0.001548, exp_rate: 0.932909, exp_rate<=1 : 0.973408, exp_rate<=2 : 0.994243, exp_rate<=3 : 0.997893, loss: 0.001590, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19176 s, avg_samples: 56.0, ips: 292.02584 samples/s, eta: 1 day, 2:17:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:53:22] ppocr INFO: epoch: [321/500], global_step: 56300, lr: 0.000100, edit distance: 0.001538, exp_rate: 0.933139, exp_rate<=1 : 0.973614, exp_rate<=2 : 0.994205, exp_rate<=3 : 0.997851, loss: 0.001761, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18863 s, avg_samples: 56.0, ips: 296.87433 samples/s, eta: 1 day, 2:17:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:53:41] ppocr INFO: epoch: [321/500], global_step: 56400, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.933202, exp_rate<=1 : 0.973688, exp_rate<=2 : 0.994195, exp_rate<=3 : 0.997844, loss: 0.001664, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18150 s, avg_samples: 56.0, ips: 308.53769 samples/s, eta: 1 day, 2:17:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:54:01] ppocr INFO: epoch: [321/500], global_step: 56500, lr: 0.000100, edit distance: 0.001542, exp_rate: 0.932569, exp_rate<=1 : 0.973343, exp_rate<=2 : 0.994130, exp_rate<=3 : 0.997833, loss: 0.001799, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20155 s, avg_samples: 56.0, ips: 277.84230 samples/s, eta: 1 day, 2:16:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:54:20] ppocr INFO: epoch: [321/500], global_step: 56600, lr: 0.000100, edit distance: 0.001527, exp_rate: 0.932824, exp_rate<=1 : 0.973329, exp_rate<=2 : 0.994096, exp_rate<=3 : 0.997866, loss: 0.001200, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19206 s, avg_samples: 56.0, ips: 291.56837 samples/s, eta: 1 day, 2:16:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:54:40] ppocr INFO: epoch: [321/500], global_step: 56700, lr: 0.000100, edit distance: 0.001532, exp_rate: 0.932620, exp_rate<=1 : 0.973257, exp_rate<=2 : 0.994151, exp_rate<=3 : 0.997885, loss: 0.002045, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19796 s, avg_samples: 56.0, ips: 282.88299 samples/s, eta: 1 day, 2:16:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:55:00] ppocr INFO: epoch: [321/500], global_step: 56800, lr: 0.000100, edit distance: 0.001529, exp_rate: 0.931746, exp_rate<=1 : 0.972668, exp_rate<=2 : 0.993907, exp_rate<=3 : 0.997716, loss: 0.001205, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20198 s, avg_samples: 56.0, ips: 277.25041 samples/s, eta: 1 day, 2:16:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:55:21] ppocr INFO: epoch: [321/500], global_step: 56900, lr: 0.000100, edit distance: 0.001528, exp_rate: 0.931551, exp_rate<=1 : 0.972598, exp_rate<=2 : 0.993867, exp_rate<=3 : 0.997766, loss: 0.001564, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20824 s, avg_samples: 56.0, ips: 268.92376 samples/s, eta: 1 day, 2:16:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:55:39] ppocr INFO: epoch: [321/500], global_step: 57000, lr: 0.000100, edit distance: 0.001526, exp_rate: 0.931926, exp_rate<=1 : 0.972698, exp_rate<=2 : 0.993850, exp_rate<=3 : 0.997700, loss: 0.000963, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18275 s, avg_samples: 56.0, ips: 306.42243 samples/s, eta: 1 day, 2:15:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:55:58] ppocr INFO: epoch: [321/500], global_step: 57100, lr: 0.000100, edit distance: 0.001528, exp_rate: 0.931851, exp_rate<=1 : 0.972663, exp_rate<=2 : 0.993732, exp_rate<=3 : 0.997685, loss: 0.001314, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18927 s, avg_samples: 56.0, ips: 295.87390 samples/s, eta: 1 day, 2:15:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:56:17] ppocr INFO: epoch: [321/500], global_step: 57200, lr: 0.000100, edit distance: 0.001524, exp_rate: 0.932056, exp_rate<=1 : 0.972770, exp_rate<=2 : 0.993775, exp_rate<=3 : 0.997713, loss: 0.002254, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19213 s, avg_samples: 56.0, ips: 291.47032 samples/s, eta: 1 day, 2:14:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:56:37] ppocr INFO: epoch: [321/500], global_step: 57300, lr: 0.000100, edit distance: 0.001521, exp_rate: 0.931897, exp_rate<=1 : 0.972850, exp_rate<=2 : 0.993773, exp_rate<=3 : 0.997736, loss: 0.001721, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19648 s, avg_samples: 56.0, ips: 285.02032 samples/s, eta: 1 day, 2:14:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:56:57] ppocr INFO: epoch: [321/500], global_step: 57400, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.931536, exp_rate<=1 : 0.972696, exp_rate<=2 : 0.993761, exp_rate<=3 : 0.997736, loss: 0.002308, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19998 s, avg_samples: 56.0, ips: 280.02918 samples/s, eta: 1 day, 2:14:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:57:16] ppocr INFO: epoch: [321/500], global_step: 57500, lr: 0.000100, edit distance: 0.001530, exp_rate: 0.931194, exp_rate<=1 : 0.972538, exp_rate<=2 : 0.993661, exp_rate<=3 : 0.997684, loss: 0.002080, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18724 s, avg_samples: 56.0, ips: 299.08444 samples/s, eta: 1 day, 2:13:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:57:23] ppocr INFO: epoch: [321/500], global_step: 57540, lr: 0.000100, edit distance: 0.001530, exp_rate: 0.931215, exp_rate<=1 : 0.972597, exp_rate<=2 : 0.993676, exp_rate<=3 : 0.997707, loss: 0.002506, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.06913 s, avg_samples: 22.4, ips: 324.01609 samples/s, eta: 1 day, 2:13:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:57:25] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 05:57:38] ppocr INFO: epoch: [322/500], global_step: 57600, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.932297, exp_rate<=1 : 0.974026, exp_rate<=2 : 0.991237, exp_rate<=3 : 0.996429, loss: 0.001746, avg_reader_cost: 0.02772 s, avg_batch_cost: 0.14969 s, avg_samples: 33.6, ips: 224.46953 samples/s, eta: 1 day, 2:14:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:57:58] ppocr INFO: epoch: [322/500], global_step: 57700, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.932565, exp_rate<=1 : 0.975482, exp_rate<=2 : 0.993208, exp_rate<=3 : 0.997551, loss: 0.001763, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20021 s, avg_samples: 56.0, ips: 279.70620 samples/s, eta: 1 day, 2:13:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:58:18] ppocr INFO: epoch: [322/500], global_step: 57800, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.934476, exp_rate<=1 : 0.976607, exp_rate<=2 : 0.993955, exp_rate<=3 : 0.997861, loss: 0.001285, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19776 s, avg_samples: 56.0, ips: 283.17218 samples/s, eta: 1 day, 2:13:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:58:35] ppocr INFO: epoch: [322/500], global_step: 57900, lr: 0.000100, edit distance: 0.001343, exp_rate: 0.935714, exp_rate<=1 : 0.976613, exp_rate<=2 : 0.993671, exp_rate<=3 : 0.997846, loss: 0.002523, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17566 s, avg_samples: 56.0, ips: 318.79455 samples/s, eta: 1 day, 2:13:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:58:54] ppocr INFO: epoch: [322/500], global_step: 58000, lr: 0.000100, edit distance: 0.001383, exp_rate: 0.933726, exp_rate<=1 : 0.975474, exp_rate<=2 : 0.993671, exp_rate<=3 : 0.997819, loss: 0.001792, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18942 s, avg_samples: 56.0, ips: 295.63361 samples/s, eta: 1 day, 2:12:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:59:15] ppocr INFO: epoch: [322/500], global_step: 58100, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.932946, exp_rate<=1 : 0.975172, exp_rate<=2 : 0.993515, exp_rate<=3 : 0.997774, loss: 0.001729, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21112 s, avg_samples: 56.0, ips: 265.25593 samples/s, eta: 1 day, 2:12:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:59:35] ppocr INFO: epoch: [322/500], global_step: 58200, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.931739, exp_rate<=1 : 0.974347, exp_rate<=2 : 0.993225, exp_rate<=3 : 0.997749, loss: 0.001566, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19913 s, avg_samples: 56.0, ips: 281.22347 samples/s, eta: 1 day, 2:12:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 05:59:54] ppocr INFO: epoch: [322/500], global_step: 58300, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.931466, exp_rate<=1 : 0.973789, exp_rate<=2 : 0.993204, exp_rate<=3 : 0.997656, loss: 0.001649, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19201 s, avg_samples: 56.0, ips: 291.65151 samples/s, eta: 1 day, 2:12:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:00:13] ppocr INFO: epoch: [322/500], global_step: 58400, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.931822, exp_rate<=1 : 0.973882, exp_rate<=2 : 0.993258, exp_rate<=3 : 0.997690, loss: 0.001456, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18238 s, avg_samples: 56.0, ips: 307.05001 samples/s, eta: 1 day, 2:11:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:00:30] ppocr INFO: epoch: [322/500], global_step: 58500, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.933277, exp_rate<=1 : 0.974323, exp_rate<=2 : 0.993471, exp_rate<=3 : 0.997774, loss: 0.001609, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17008 s, avg_samples: 56.0, ips: 329.26527 samples/s, eta: 1 day, 2:10:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:00:48] ppocr INFO: epoch: [322/500], global_step: 58600, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.934028, exp_rate<=1 : 0.974457, exp_rate<=2 : 0.993566, exp_rate<=3 : 0.997892, loss: 0.001145, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18705 s, avg_samples: 56.0, ips: 299.39067 samples/s, eta: 1 day, 2:10:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:01:10] ppocr INFO: epoch: [322/500], global_step: 58700, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.933457, exp_rate<=1 : 0.974169, exp_rate<=2 : 0.993406, exp_rate<=3 : 0.997861, loss: 0.001686, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21062 s, avg_samples: 56.0, ips: 265.87911 samples/s, eta: 1 day, 2:10:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:01:29] ppocr INFO: epoch: [322/500], global_step: 58800, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.933093, exp_rate<=1 : 0.974073, exp_rate<=2 : 0.993524, exp_rate<=3 : 0.997929, loss: 0.001186, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19165 s, avg_samples: 56.0, ips: 292.19736 samples/s, eta: 1 day, 2:10:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:01:48] ppocr INFO: epoch: [322/500], global_step: 58900, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.933126, exp_rate<=1 : 0.974005, exp_rate<=2 : 0.993546, exp_rate<=3 : 0.997929, loss: 0.001402, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19520 s, avg_samples: 56.0, ips: 286.87814 samples/s, eta: 1 day, 2:09:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:02:06] ppocr INFO: epoch: [322/500], global_step: 59000, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.933573, exp_rate<=1 : 0.974287, exp_rate<=2 : 0.993652, exp_rate<=3 : 0.997981, loss: 0.001337, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17871 s, avg_samples: 56.0, ips: 313.35300 samples/s, eta: 1 day, 2:09:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:02:26] ppocr INFO: epoch: [322/500], global_step: 59100, lr: 0.000100, edit distance: 0.001454, exp_rate: 0.933039, exp_rate<=1 : 0.974079, exp_rate<=2 : 0.993656, exp_rate<=3 : 0.997950, loss: 0.001807, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.19941 s, avg_samples: 56.0, ips: 280.82631 samples/s, eta: 1 day, 2:09:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:02:45] ppocr INFO: epoch: [322/500], global_step: 59200, lr: 0.000100, edit distance: 0.001454, exp_rate: 0.932388, exp_rate<=1 : 0.973601, exp_rate<=2 : 0.993714, exp_rate<=3 : 0.997934, loss: 0.001115, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19104 s, avg_samples: 56.0, ips: 293.12623 samples/s, eta: 1 day, 2:08:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:03:05] ppocr INFO: epoch: [322/500], global_step: 59300, lr: 0.000100, edit distance: 0.001458, exp_rate: 0.931947, exp_rate<=1 : 0.973255, exp_rate<=2 : 0.993514, exp_rate<=3 : 0.997872, loss: 0.001945, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19414 s, avg_samples: 56.0, ips: 288.45279 samples/s, eta: 1 day, 2:08:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:03:23] ppocr INFO: epoch: [322/500], global_step: 59400, lr: 0.000100, edit distance: 0.001463, exp_rate: 0.932013, exp_rate<=1 : 0.973137, exp_rate<=2 : 0.993357, exp_rate<=3 : 0.997771, loss: 0.001748, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18157 s, avg_samples: 56.0, ips: 308.42098 samples/s, eta: 1 day, 2:08:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:03:43] ppocr INFO: epoch: [322/500], global_step: 59500, lr: 0.000100, edit distance: 0.001480, exp_rate: 0.931314, exp_rate<=1 : 0.972674, exp_rate<=2 : 0.993148, exp_rate<=3 : 0.997674, loss: 0.001664, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19762 s, avg_samples: 56.0, ips: 283.37277 samples/s, eta: 1 day, 2:07:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:04:02] ppocr INFO: epoch: [322/500], global_step: 59600, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.931242, exp_rate<=1 : 0.972727, exp_rate<=2 : 0.993064, exp_rate<=3 : 0.997616, loss: 0.002036, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19472 s, avg_samples: 56.0, ips: 287.58713 samples/s, eta: 1 day, 2:07:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:04:21] ppocr INFO: epoch: [322/500], global_step: 59700, lr: 0.000100, edit distance: 0.001493, exp_rate: 0.931144, exp_rate<=1 : 0.972538, exp_rate<=2 : 0.993007, exp_rate<=3 : 0.997572, loss: 0.001524, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19126 s, avg_samples: 56.0, ips: 292.79118 samples/s, eta: 1 day, 2:07:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:04:40] ppocr INFO: epoch: [322/500], global_step: 59800, lr: 0.000100, edit distance: 0.001484, exp_rate: 0.931100, exp_rate<=1 : 0.972537, exp_rate<=2 : 0.993070, exp_rate<=3 : 0.997610, loss: 0.001735, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18707 s, avg_samples: 56.0, ips: 299.35518 samples/s, eta: 1 day, 2:06:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:04:58] ppocr INFO: epoch: [322/500], global_step: 59900, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.931215, exp_rate<=1 : 0.972711, exp_rate<=2 : 0.993103, exp_rate<=3 : 0.997604, loss: 0.002245, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18147 s, avg_samples: 56.0, ips: 308.59287 samples/s, eta: 1 day, 2:06:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:05:19] ppocr INFO: epoch: [322/500], global_step: 60000, lr: 0.000100, edit distance: 0.001489, exp_rate: 0.930224, exp_rate<=1 : 0.972253, exp_rate<=2 : 0.992929, exp_rate<=3 : 0.997525, loss: 0.002482, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20648 s, avg_samples: 56.0, ips: 271.20701 samples/s, eta: 1 day, 2:06:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:05:38] ppocr INFO: epoch: [322/500], global_step: 60100, lr: 0.000100, edit distance: 0.001501, exp_rate: 0.929680, exp_rate<=1 : 0.971953, exp_rate<=2 : 0.992919, exp_rate<=3 : 0.997505, loss: 0.001567, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19131 s, avg_samples: 56.0, ips: 292.72267 samples/s, eta: 1 day, 2:05:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:05:56] ppocr INFO: epoch: [322/500], global_step: 60200, lr: 0.000100, edit distance: 0.001502, exp_rate: 0.929702, exp_rate<=1 : 0.972032, exp_rate<=2 : 0.992932, exp_rate<=3 : 0.997521, loss: 0.001718, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18090 s, avg_samples: 56.0, ips: 309.55683 samples/s, eta: 1 day, 2:05:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:06:13] ppocr INFO: epoch: [322/500], global_step: 60280, lr: 0.000100, edit distance: 0.001503, exp_rate: 0.929343, exp_rate<=1 : 0.971889, exp_rate<=2 : 0.992852, exp_rate<=3 : 0.997537, loss: 0.001932, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17083 s, avg_samples: 44.8, ips: 262.24675 samples/s, eta: 1 day, 2:05:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:06:15] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 06:06:20] ppocr INFO: epoch: [323/500], global_step: 60300, lr: 0.000100, edit distance: 0.001132, exp_rate: 0.940901, exp_rate<=1 : 0.971726, exp_rate<=2 : 0.991780, exp_rate<=3 : 0.998115, loss: 0.001617, avg_reader_cost: 0.02513 s, avg_batch_cost: 0.06373 s, avg_samples: 11.2, ips: 175.74174 samples/s, eta: 1 day, 2:05:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:06:38] ppocr INFO: epoch: [323/500], global_step: 60400, lr: 0.000100, edit distance: 0.001414, exp_rate: 0.938244, exp_rate<=1 : 0.972770, exp_rate<=2 : 0.993293, exp_rate<=3 : 0.997657, loss: 0.001834, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18777 s, avg_samples: 56.0, ips: 298.23952 samples/s, eta: 1 day, 2:05:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:06:58] ppocr INFO: epoch: [323/500], global_step: 60500, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.937478, exp_rate<=1 : 0.973214, exp_rate<=2 : 0.994334, exp_rate<=3 : 0.998049, loss: 0.001682, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19950 s, avg_samples: 56.0, ips: 280.70599 samples/s, eta: 1 day, 2:04:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:07:17] ppocr INFO: epoch: [323/500], global_step: 60600, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.938291, exp_rate<=1 : 0.975121, exp_rate<=2 : 0.994915, exp_rate<=3 : 0.998401, loss: 0.001676, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18249 s, avg_samples: 56.0, ips: 306.86636 samples/s, eta: 1 day, 2:04:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:07:36] ppocr INFO: epoch: [323/500], global_step: 60700, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.937197, exp_rate<=1 : 0.974577, exp_rate<=2 : 0.994185, exp_rate<=3 : 0.998112, loss: 0.001282, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19434 s, avg_samples: 56.0, ips: 288.15614 samples/s, eta: 1 day, 2:04:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:07:55] ppocr INFO: epoch: [323/500], global_step: 60800, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.936987, exp_rate<=1 : 0.975028, exp_rate<=2 : 0.994357, exp_rate<=3 : 0.998302, loss: 0.001798, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18976 s, avg_samples: 56.0, ips: 295.10963 samples/s, eta: 1 day, 2:03:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:08:13] ppocr INFO: epoch: [323/500], global_step: 60900, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.937493, exp_rate<=1 : 0.975359, exp_rate<=2 : 0.994531, exp_rate<=3 : 0.998369, loss: 0.001450, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18059 s, avg_samples: 56.0, ips: 310.09530 samples/s, eta: 1 day, 2:03:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:08:32] ppocr INFO: epoch: [323/500], global_step: 61000, lr: 0.000100, edit distance: 0.001348, exp_rate: 0.937337, exp_rate<=1 : 0.975562, exp_rate<=2 : 0.994797, exp_rate<=3 : 0.998370, loss: 0.001467, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19090 s, avg_samples: 56.0, ips: 293.34703 samples/s, eta: 1 day, 2:03:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:08:52] ppocr INFO: epoch: [323/500], global_step: 61100, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.935697, exp_rate<=1 : 0.974771, exp_rate<=2 : 0.994657, exp_rate<=3 : 0.998438, loss: 0.001246, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19496 s, avg_samples: 56.0, ips: 287.23854 samples/s, eta: 1 day, 2:02:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:09:12] ppocr INFO: epoch: [323/500], global_step: 61200, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.934699, exp_rate<=1 : 0.974438, exp_rate<=2 : 0.994467, exp_rate<=3 : 0.998270, loss: 0.001669, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20241 s, avg_samples: 56.0, ips: 276.67287 samples/s, eta: 1 day, 2:02:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:09:31] ppocr INFO: epoch: [323/500], global_step: 61300, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.935456, exp_rate<=1 : 0.974570, exp_rate<=2 : 0.994289, exp_rate<=3 : 0.998133, loss: 0.001770, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19137 s, avg_samples: 56.0, ips: 292.63179 samples/s, eta: 1 day, 2:02:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:09:50] ppocr INFO: epoch: [323/500], global_step: 61400, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.935076, exp_rate<=1 : 0.974209, exp_rate<=2 : 0.994018, exp_rate<=3 : 0.998042, loss: 0.002135, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18941 s, avg_samples: 56.0, ips: 295.64874 samples/s, eta: 1 day, 2:01:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:10:10] ppocr INFO: epoch: [323/500], global_step: 61500, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.934241, exp_rate<=1 : 0.973709, exp_rate<=2 : 0.993676, exp_rate<=3 : 0.997899, loss: 0.001734, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20146 s, avg_samples: 56.0, ips: 277.97375 samples/s, eta: 1 day, 2:01:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:10:31] ppocr INFO: epoch: [323/500], global_step: 61600, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.934067, exp_rate<=1 : 0.973667, exp_rate<=2 : 0.993680, exp_rate<=3 : 0.997992, loss: 0.001575, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20238 s, avg_samples: 56.0, ips: 276.71098 samples/s, eta: 1 day, 2:01:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:10:50] ppocr INFO: epoch: [323/500], global_step: 61700, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.933886, exp_rate<=1 : 0.973702, exp_rate<=2 : 0.993679, exp_rate<=3 : 0.997978, loss: 0.002003, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19191 s, avg_samples: 56.0, ips: 291.80126 samples/s, eta: 1 day, 2:01:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:11:09] ppocr INFO: epoch: [323/500], global_step: 61800, lr: 0.000100, edit distance: 0.001446, exp_rate: 0.933631, exp_rate<=1 : 0.973606, exp_rate<=2 : 0.993699, exp_rate<=3 : 0.997978, loss: 0.001304, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19629 s, avg_samples: 56.0, ips: 285.29541 samples/s, eta: 1 day, 2:00:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:11:28] ppocr INFO: epoch: [323/500], global_step: 61900, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.933145, exp_rate<=1 : 0.973600, exp_rate<=2 : 0.993666, exp_rate<=3 : 0.997956, loss: 0.001966, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18346 s, avg_samples: 56.0, ips: 305.25151 samples/s, eta: 1 day, 2:00:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:11:46] ppocr INFO: epoch: [323/500], global_step: 62000, lr: 0.000100, edit distance: 0.001495, exp_rate: 0.932635, exp_rate<=1 : 0.973193, exp_rate<=2 : 0.993576, exp_rate<=3 : 0.997946, loss: 0.001703, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18198 s, avg_samples: 56.0, ips: 307.73415 samples/s, eta: 1 day, 1:59:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:12:06] ppocr INFO: epoch: [323/500], global_step: 62100, lr: 0.000100, edit distance: 0.001499, exp_rate: 0.932094, exp_rate<=1 : 0.972862, exp_rate<=2 : 0.993342, exp_rate<=3 : 0.997814, loss: 0.001745, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19751 s, avg_samples: 56.0, ips: 283.53154 samples/s, eta: 1 day, 1:59:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:12:25] ppocr INFO: epoch: [323/500], global_step: 62200, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.931383, exp_rate<=1 : 0.972567, exp_rate<=2 : 0.993243, exp_rate<=3 : 0.997785, loss: 0.001392, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19259 s, avg_samples: 56.0, ips: 290.76574 samples/s, eta: 1 day, 1:59:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:12:45] ppocr INFO: epoch: [323/500], global_step: 62300, lr: 0.000100, edit distance: 0.001517, exp_rate: 0.930897, exp_rate<=1 : 0.972480, exp_rate<=2 : 0.993161, exp_rate<=3 : 0.997727, loss: 0.001872, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19921 s, avg_samples: 56.0, ips: 281.11508 samples/s, eta: 1 day, 1:59:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:13:03] ppocr INFO: epoch: [323/500], global_step: 62400, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.930795, exp_rate<=1 : 0.972547, exp_rate<=2 : 0.993209, exp_rate<=3 : 0.997765, loss: 0.001239, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18434 s, avg_samples: 56.0, ips: 303.79375 samples/s, eta: 1 day, 1:58:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:13:23] ppocr INFO: epoch: [323/500], global_step: 62500, lr: 0.000100, edit distance: 0.001517, exp_rate: 0.930858, exp_rate<=1 : 0.972705, exp_rate<=2 : 0.993288, exp_rate<=3 : 0.997796, loss: 0.002161, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19082 s, avg_samples: 56.0, ips: 293.46384 samples/s, eta: 1 day, 1:58:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:13:42] ppocr INFO: epoch: [323/500], global_step: 62600, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.930691, exp_rate<=1 : 0.972667, exp_rate<=2 : 0.993342, exp_rate<=3 : 0.997781, loss: 0.002121, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19595 s, avg_samples: 56.0, ips: 285.78897 samples/s, eta: 1 day, 1:58:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:14:01] ppocr INFO: epoch: [323/500], global_step: 62700, lr: 0.000100, edit distance: 0.001537, exp_rate: 0.930458, exp_rate<=1 : 0.972610, exp_rate<=2 : 0.993251, exp_rate<=3 : 0.997711, loss: 0.001538, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18305 s, avg_samples: 56.0, ips: 305.92833 samples/s, eta: 1 day, 1:57:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:14:19] ppocr INFO: epoch: [323/500], global_step: 62800, lr: 0.000100, edit distance: 0.001547, exp_rate: 0.930399, exp_rate<=1 : 0.972603, exp_rate<=2 : 0.993329, exp_rate<=3 : 0.997719, loss: 0.001890, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18039 s, avg_samples: 56.0, ips: 310.44388 samples/s, eta: 1 day, 1:57:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:14:37] ppocr INFO: epoch: [323/500], global_step: 62900, lr: 0.000100, edit distance: 0.001551, exp_rate: 0.930149, exp_rate<=1 : 0.972507, exp_rate<=2 : 0.993322, exp_rate<=3 : 0.997732, loss: 0.001884, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18801 s, avg_samples: 56.0, ips: 297.85772 samples/s, eta: 1 day, 1:56:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:14:56] ppocr INFO: epoch: [323/500], global_step: 63000, lr: 0.000100, edit distance: 0.001545, exp_rate: 0.930325, exp_rate<=1 : 0.972565, exp_rate<=2 : 0.993310, exp_rate<=3 : 0.997740, loss: 0.001227, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19032 s, avg_samples: 56.0, ips: 294.24675 samples/s, eta: 1 day, 1:56:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:15:00] ppocr INFO: epoch: [323/500], global_step: 63020, lr: 0.000100, edit distance: 0.001544, exp_rate: 0.930425, exp_rate<=1 : 0.972635, exp_rate<=2 : 0.993303, exp_rate<=3 : 0.997741, loss: 0.002986, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03679 s, avg_samples: 11.2, ips: 304.44465 samples/s, eta: 1 day, 1:56:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:15:02] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 06:15:17] ppocr INFO: epoch: [324/500], global_step: 63100, lr: 0.000100, edit distance: 0.001488, exp_rate: 0.939395, exp_rate<=1 : 0.977832, exp_rate<=2 : 0.993198, exp_rate<=3 : 0.997688, loss: 0.001312, avg_reader_cost: 0.02575 s, avg_batch_cost: 0.17024 s, avg_samples: 44.8, ips: 263.15868 samples/s, eta: 1 day, 1:56:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:15:38] ppocr INFO: epoch: [324/500], global_step: 63200, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.937905, exp_rate<=1 : 0.975659, exp_rate<=2 : 0.993736, exp_rate<=3 : 0.997834, loss: 0.001209, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20662 s, avg_samples: 56.0, ips: 271.03125 samples/s, eta: 1 day, 1:56:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:15:58] ppocr INFO: epoch: [324/500], global_step: 63300, lr: 0.000100, edit distance: 0.001383, exp_rate: 0.935769, exp_rate<=1 : 0.974920, exp_rate<=2 : 0.993283, exp_rate<=3 : 0.997951, loss: 0.002154, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19842 s, avg_samples: 56.0, ips: 282.22648 samples/s, eta: 1 day, 1:55:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:16:17] ppocr INFO: epoch: [324/500], global_step: 63400, lr: 0.000100, edit distance: 0.001386, exp_rate: 0.935374, exp_rate<=1 : 0.975801, exp_rate<=2 : 0.993734, exp_rate<=3 : 0.997795, loss: 0.001354, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19558 s, avg_samples: 56.0, ips: 286.32571 samples/s, eta: 1 day, 1:55:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:16:37] ppocr INFO: epoch: [324/500], global_step: 63500, lr: 0.000100, edit distance: 0.001367, exp_rate: 0.936460, exp_rate<=1 : 0.975113, exp_rate<=2 : 0.993447, exp_rate<=3 : 0.997462, loss: 0.001521, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19508 s, avg_samples: 56.0, ips: 287.06227 samples/s, eta: 1 day, 1:55:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:16:55] ppocr INFO: epoch: [324/500], global_step: 63600, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.938433, exp_rate<=1 : 0.976103, exp_rate<=2 : 0.993986, exp_rate<=3 : 0.997848, loss: 0.001384, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17707 s, avg_samples: 56.0, ips: 316.26742 samples/s, eta: 1 day, 1:54:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:17:15] ppocr INFO: epoch: [324/500], global_step: 63700, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.938229, exp_rate<=1 : 0.976284, exp_rate<=2 : 0.994171, exp_rate<=3 : 0.997963, loss: 0.001428, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20461 s, avg_samples: 56.0, ips: 273.69293 samples/s, eta: 1 day, 1:54:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:17:34] ppocr INFO: epoch: [324/500], global_step: 63800, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.939087, exp_rate<=1 : 0.976705, exp_rate<=2 : 0.994279, exp_rate<=3 : 0.997984, loss: 0.001987, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19029 s, avg_samples: 56.0, ips: 294.28458 samples/s, eta: 1 day, 1:54:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:17:53] ppocr INFO: epoch: [324/500], global_step: 63900, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.938899, exp_rate<=1 : 0.976700, exp_rate<=2 : 0.994411, exp_rate<=3 : 0.998089, loss: 0.001368, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18571 s, avg_samples: 56.0, ips: 301.54058 samples/s, eta: 1 day, 1:53:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:18:12] ppocr INFO: epoch: [324/500], global_step: 64000, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.938106, exp_rate<=1 : 0.976043, exp_rate<=2 : 0.994158, exp_rate<=3 : 0.998007, loss: 0.001648, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19028 s, avg_samples: 56.0, ips: 294.29815 samples/s, eta: 1 day, 1:53:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:18:32] ppocr INFO: epoch: [324/500], global_step: 64100, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.937588, exp_rate<=1 : 0.975576, exp_rate<=2 : 0.994215, exp_rate<=3 : 0.997988, loss: 0.002389, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20411 s, avg_samples: 56.0, ips: 274.35948 samples/s, eta: 1 day, 1:53:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:18:53] ppocr INFO: epoch: [324/500], global_step: 64200, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.935986, exp_rate<=1 : 0.974858, exp_rate<=2 : 0.994045, exp_rate<=3 : 0.997911, loss: 0.002062, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20756 s, avg_samples: 56.0, ips: 269.80728 samples/s, eta: 1 day, 1:53:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:19:12] ppocr INFO: epoch: [324/500], global_step: 64300, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.935644, exp_rate<=1 : 0.974841, exp_rate<=2 : 0.994036, exp_rate<=3 : 0.997980, loss: 0.001409, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19421 s, avg_samples: 56.0, ips: 288.35502 samples/s, eta: 1 day, 1:52:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:19:31] ppocr INFO: epoch: [324/500], global_step: 64400, lr: 0.000100, edit distance: 0.001405, exp_rate: 0.936103, exp_rate<=1 : 0.975127, exp_rate<=2 : 0.994022, exp_rate<=3 : 0.997943, loss: 0.001922, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18161 s, avg_samples: 56.0, ips: 308.34600 samples/s, eta: 1 day, 1:52:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:19:50] ppocr INFO: epoch: [324/500], global_step: 64500, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.935584, exp_rate<=1 : 0.975039, exp_rate<=2 : 0.993995, exp_rate<=3 : 0.997942, loss: 0.002120, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19396 s, avg_samples: 56.0, ips: 288.71469 samples/s, eta: 1 day, 1:52:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:20:10] ppocr INFO: epoch: [324/500], global_step: 64600, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.934734, exp_rate<=1 : 0.974799, exp_rate<=2 : 0.993883, exp_rate<=3 : 0.997949, loss: 0.001519, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19558 s, avg_samples: 56.0, ips: 286.33143 samples/s, eta: 1 day, 1:51:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:20:29] ppocr INFO: epoch: [324/500], global_step: 64700, lr: 0.000100, edit distance: 0.001451, exp_rate: 0.934087, exp_rate<=1 : 0.974372, exp_rate<=2 : 0.993741, exp_rate<=3 : 0.997873, loss: 0.001417, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18923 s, avg_samples: 56.0, ips: 295.93956 samples/s, eta: 1 day, 1:51:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:20:47] ppocr INFO: epoch: [324/500], global_step: 64800, lr: 0.000100, edit distance: 0.001464, exp_rate: 0.933996, exp_rate<=1 : 0.974431, exp_rate<=2 : 0.993747, exp_rate<=3 : 0.997854, loss: 0.002048, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18547 s, avg_samples: 56.0, ips: 301.93250 samples/s, eta: 1 day, 1:51:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:21:07] ppocr INFO: epoch: [324/500], global_step: 64900, lr: 0.000100, edit distance: 0.001484, exp_rate: 0.933579, exp_rate<=1 : 0.974152, exp_rate<=2 : 0.993667, exp_rate<=3 : 0.997785, loss: 0.002557, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19893 s, avg_samples: 56.0, ips: 281.50720 samples/s, eta: 1 day, 1:50:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:21:26] ppocr INFO: epoch: [324/500], global_step: 65000, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.932819, exp_rate<=1 : 0.973848, exp_rate<=2 : 0.993591, exp_rate<=3 : 0.997789, loss: 0.001829, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18740 s, avg_samples: 56.0, ips: 298.82637 samples/s, eta: 1 day, 1:50:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:21:44] ppocr INFO: epoch: [324/500], global_step: 65100, lr: 0.000100, edit distance: 0.001521, exp_rate: 0.932686, exp_rate<=1 : 0.973926, exp_rate<=2 : 0.993612, exp_rate<=3 : 0.997819, loss: 0.002534, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18510 s, avg_samples: 56.0, ips: 302.54155 samples/s, eta: 1 day, 1:50:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:22:03] ppocr INFO: epoch: [324/500], global_step: 65200, lr: 0.000100, edit distance: 0.001534, exp_rate: 0.932376, exp_rate<=1 : 0.973936, exp_rate<=2 : 0.993668, exp_rate<=3 : 0.997788, loss: 0.002584, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19049 s, avg_samples: 56.0, ips: 293.97924 samples/s, eta: 1 day, 1:49:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:22:23] ppocr INFO: epoch: [324/500], global_step: 65300, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.931529, exp_rate<=1 : 0.973602, exp_rate<=2 : 0.993566, exp_rate<=3 : 0.997729, loss: 0.002442, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19120 s, avg_samples: 56.0, ips: 292.88258 samples/s, eta: 1 day, 1:49:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:22:42] ppocr INFO: epoch: [324/500], global_step: 65400, lr: 0.000100, edit distance: 0.001574, exp_rate: 0.930965, exp_rate<=1 : 0.973376, exp_rate<=2 : 0.993481, exp_rate<=3 : 0.997678, loss: 0.001566, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18952 s, avg_samples: 56.0, ips: 295.48141 samples/s, eta: 1 day, 1:49:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:23:00] ppocr INFO: epoch: [324/500], global_step: 65500, lr: 0.000100, edit distance: 0.001579, exp_rate: 0.931091, exp_rate<=1 : 0.973449, exp_rate<=2 : 0.993467, exp_rate<=3 : 0.997682, loss: 0.001564, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18105 s, avg_samples: 56.0, ips: 309.31129 samples/s, eta: 1 day, 1:48:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:23:18] ppocr INFO: epoch: [324/500], global_step: 65600, lr: 0.000100, edit distance: 0.001576, exp_rate: 0.931051, exp_rate<=1 : 0.973355, exp_rate<=2 : 0.993439, exp_rate<=3 : 0.997631, loss: 0.001583, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18450 s, avg_samples: 56.0, ips: 303.51827 samples/s, eta: 1 day, 1:48:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:23:37] ppocr INFO: epoch: [324/500], global_step: 65700, lr: 0.000100, edit distance: 0.001577, exp_rate: 0.931145, exp_rate<=1 : 0.973430, exp_rate<=2 : 0.993470, exp_rate<=3 : 0.997638, loss: 0.001763, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18733 s, avg_samples: 56.0, ips: 298.93180 samples/s, eta: 1 day, 1:47:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:23:49] ppocr INFO: epoch: [324/500], global_step: 65760, lr: 0.000100, edit distance: 0.001583, exp_rate: 0.930735, exp_rate<=1 : 0.973200, exp_rate<=2 : 0.993414, exp_rate<=3 : 0.997606, loss: 0.001578, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.12348 s, avg_samples: 33.6, ips: 272.10307 samples/s, eta: 1 day, 1:47:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:23:51] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 06:23:59] ppocr INFO: epoch: [325/500], global_step: 65800, lr: 0.000100, edit distance: 0.000969, exp_rate: 0.954613, exp_rate<=1 : 0.982639, exp_rate<=2 : 0.997162, exp_rate<=3 : 1.000000, loss: 0.001209, avg_reader_cost: 0.02488 s, avg_batch_cost: 0.09725 s, avg_samples: 22.4, ips: 230.32726 samples/s, eta: 1 day, 1:47:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:24:17] ppocr INFO: epoch: [325/500], global_step: 65900, lr: 0.000100, edit distance: 0.001084, exp_rate: 0.952588, exp_rate<=1 : 0.982546, exp_rate<=2 : 0.996582, exp_rate<=3 : 0.998529, loss: 0.001488, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18281 s, avg_samples: 56.0, ips: 306.32758 samples/s, eta: 1 day, 1:47:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:24:36] ppocr INFO: epoch: [325/500], global_step: 66000, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.946240, exp_rate<=1 : 0.979775, exp_rate<=2 : 0.996193, exp_rate<=3 : 0.998387, loss: 0.001123, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18450 s, avg_samples: 56.0, ips: 303.52561 samples/s, eta: 1 day, 1:46:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:24:55] ppocr INFO: epoch: [325/500], global_step: 66100, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.941699, exp_rate<=1 : 0.977759, exp_rate<=2 : 0.995083, exp_rate<=3 : 0.997805, loss: 0.001310, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19374 s, avg_samples: 56.0, ips: 289.04146 samples/s, eta: 1 day, 1:46:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:25:15] ppocr INFO: epoch: [325/500], global_step: 66200, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.940449, exp_rate<=1 : 0.976916, exp_rate<=2 : 0.994807, exp_rate<=3 : 0.998018, loss: 0.001204, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19633 s, avg_samples: 56.0, ips: 285.23307 samples/s, eta: 1 day, 1:46:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:25:33] ppocr INFO: epoch: [325/500], global_step: 66300, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.939804, exp_rate<=1 : 0.976603, exp_rate<=2 : 0.994789, exp_rate<=3 : 0.997889, loss: 0.002296, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18579 s, avg_samples: 56.0, ips: 301.42180 samples/s, eta: 1 day, 1:45:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:25:53] ppocr INFO: epoch: [325/500], global_step: 66400, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.937181, exp_rate<=1 : 0.975373, exp_rate<=2 : 0.994318, exp_rate<=3 : 0.997912, loss: 0.001893, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19497 s, avg_samples: 56.0, ips: 287.22277 samples/s, eta: 1 day, 1:45:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:26:12] ppocr INFO: epoch: [325/500], global_step: 66500, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.937050, exp_rate<=1 : 0.975416, exp_rate<=2 : 0.994304, exp_rate<=3 : 0.997869, loss: 0.001669, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19399 s, avg_samples: 56.0, ips: 288.67969 samples/s, eta: 1 day, 1:45:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:26:32] ppocr INFO: epoch: [325/500], global_step: 66600, lr: 0.000100, edit distance: 0.001446, exp_rate: 0.935994, exp_rate<=1 : 0.975112, exp_rate<=2 : 0.994005, exp_rate<=3 : 0.997862, loss: 0.003056, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19635 s, avg_samples: 56.0, ips: 285.20464 samples/s, eta: 1 day, 1:45:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:26:50] ppocr INFO: epoch: [325/500], global_step: 66700, lr: 0.000100, edit distance: 0.001470, exp_rate: 0.935131, exp_rate<=1 : 0.975054, exp_rate<=2 : 0.994080, exp_rate<=3 : 0.997891, loss: 0.002202, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17870 s, avg_samples: 56.0, ips: 313.37306 samples/s, eta: 1 day, 1:44:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:27:09] ppocr INFO: epoch: [325/500], global_step: 66800, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.934125, exp_rate<=1 : 0.974676, exp_rate<=2 : 0.994077, exp_rate<=3 : 0.997973, loss: 0.001781, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18631 s, avg_samples: 56.0, ips: 300.56880 samples/s, eta: 1 day, 1:44:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:27:27] ppocr INFO: epoch: [325/500], global_step: 66900, lr: 0.000100, edit distance: 0.001508, exp_rate: 0.933307, exp_rate<=1 : 0.974334, exp_rate<=2 : 0.994063, exp_rate<=3 : 0.998024, loss: 0.001731, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18764 s, avg_samples: 56.0, ips: 298.44940 samples/s, eta: 1 day, 1:43:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:27:46] ppocr INFO: epoch: [325/500], global_step: 67000, lr: 0.000100, edit distance: 0.001551, exp_rate: 0.932345, exp_rate<=1 : 0.974009, exp_rate<=2 : 0.993897, exp_rate<=3 : 0.997923, loss: 0.001545, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18259 s, avg_samples: 56.0, ips: 306.70169 samples/s, eta: 1 day, 1:43:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:28:04] ppocr INFO: epoch: [325/500], global_step: 67100, lr: 0.000100, edit distance: 0.001547, exp_rate: 0.932273, exp_rate<=1 : 0.973764, exp_rate<=2 : 0.993882, exp_rate<=3 : 0.997849, loss: 0.001924, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18761 s, avg_samples: 56.0, ips: 298.49295 samples/s, eta: 1 day, 1:42:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:28:24] ppocr INFO: epoch: [325/500], global_step: 67200, lr: 0.000100, edit distance: 0.001558, exp_rate: 0.931134, exp_rate<=1 : 0.973077, exp_rate<=2 : 0.993735, exp_rate<=3 : 0.997790, loss: 0.002384, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19313 s, avg_samples: 56.0, ips: 289.95316 samples/s, eta: 1 day, 1:42:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:28:45] ppocr INFO: epoch: [325/500], global_step: 67300, lr: 0.000100, edit distance: 0.001579, exp_rate: 0.929702, exp_rate<=1 : 0.972505, exp_rate<=2 : 0.993583, exp_rate<=3 : 0.997722, loss: 0.001585, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20992 s, avg_samples: 56.0, ips: 266.76525 samples/s, eta: 1 day, 1:42:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:29:04] ppocr INFO: epoch: [325/500], global_step: 67400, lr: 0.000100, edit distance: 0.001567, exp_rate: 0.929527, exp_rate<=1 : 0.972472, exp_rate<=2 : 0.993552, exp_rate<=3 : 0.997702, loss: 0.002696, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19051 s, avg_samples: 56.0, ips: 293.95180 samples/s, eta: 1 day, 1:42:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:29:22] ppocr INFO: epoch: [325/500], global_step: 67500, lr: 0.000100, edit distance: 0.001562, exp_rate: 0.929981, exp_rate<=1 : 0.972625, exp_rate<=2 : 0.993577, exp_rate<=3 : 0.997732, loss: 0.001671, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18200 s, avg_samples: 56.0, ips: 307.69138 samples/s, eta: 1 day, 1:41:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:29:41] ppocr INFO: epoch: [325/500], global_step: 67600, lr: 0.000100, edit distance: 0.001563, exp_rate: 0.930063, exp_rate<=1 : 0.972746, exp_rate<=2 : 0.993625, exp_rate<=3 : 0.997778, loss: 0.002035, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.25212 samples/s, eta: 1 day, 1:41:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:30:00] ppocr INFO: epoch: [325/500], global_step: 67700, lr: 0.000100, edit distance: 0.001587, exp_rate: 0.929606, exp_rate<=1 : 0.972746, exp_rate<=2 : 0.993670, exp_rate<=3 : 0.997817, loss: 0.001637, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19133 s, avg_samples: 56.0, ips: 292.69324 samples/s, eta: 1 day, 1:41:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:30:20] ppocr INFO: epoch: [325/500], global_step: 67800, lr: 0.000100, edit distance: 0.001577, exp_rate: 0.929452, exp_rate<=1 : 0.972807, exp_rate<=2 : 0.993711, exp_rate<=3 : 0.997827, loss: 0.001724, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20293 s, avg_samples: 56.0, ips: 275.95703 samples/s, eta: 1 day, 1:40:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:30:40] ppocr INFO: epoch: [325/500], global_step: 67900, lr: 0.000100, edit distance: 0.001563, exp_rate: 0.929869, exp_rate<=1 : 0.972896, exp_rate<=2 : 0.993688, exp_rate<=3 : 0.997782, loss: 0.001040, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19620 s, avg_samples: 56.0, ips: 285.41981 samples/s, eta: 1 day, 1:40:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:31:00] ppocr INFO: epoch: [325/500], global_step: 68000, lr: 0.000100, edit distance: 0.001546, exp_rate: 0.930269, exp_rate<=1 : 0.972976, exp_rate<=2 : 0.993678, exp_rate<=3 : 0.997769, loss: 0.001983, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20408 s, avg_samples: 56.0, ips: 274.39569 samples/s, eta: 1 day, 1:40:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:31:20] ppocr INFO: epoch: [325/500], global_step: 68100, lr: 0.000100, edit distance: 0.001534, exp_rate: 0.930449, exp_rate<=1 : 0.972880, exp_rate<=2 : 0.993650, exp_rate<=3 : 0.997751, loss: 0.001263, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19648 s, avg_samples: 56.0, ips: 285.02246 samples/s, eta: 1 day, 1:40:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:31:40] ppocr INFO: epoch: [325/500], global_step: 68200, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.930515, exp_rate<=1 : 0.972924, exp_rate<=2 : 0.993615, exp_rate<=3 : 0.997738, loss: 0.001969, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19458 s, avg_samples: 56.0, ips: 287.79470 samples/s, eta: 1 day, 1:39:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:31:58] ppocr INFO: epoch: [325/500], global_step: 68300, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.930307, exp_rate<=1 : 0.972735, exp_rate<=2 : 0.993635, exp_rate<=3 : 0.997749, loss: 0.001378, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18773 s, avg_samples: 56.0, ips: 298.30289 samples/s, eta: 1 day, 1:39:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:32:18] ppocr INFO: epoch: [325/500], global_step: 68400, lr: 0.000100, edit distance: 0.001541, exp_rate: 0.930295, exp_rate<=1 : 0.972738, exp_rate<=2 : 0.993552, exp_rate<=3 : 0.997740, loss: 0.001713, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19356 s, avg_samples: 56.0, ips: 289.31439 samples/s, eta: 1 day, 1:39:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:32:37] ppocr INFO: epoch: [325/500], global_step: 68500, lr: 0.000100, edit distance: 0.001544, exp_rate: 0.930189, exp_rate<=1 : 0.972683, exp_rate<=2 : 0.993571, exp_rate<=3 : 0.997763, loss: 0.001890, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18946 s, avg_samples: 56.0, ips: 295.57374 samples/s, eta: 1 day, 1:38:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:32:39] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 06:32:39] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_325 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 06:33:00] ppocr INFO: epoch: [326/500], global_step: 68600, lr: 0.000100, edit distance: 0.001651, exp_rate: 0.927430, exp_rate<=1 : 0.968720, exp_rate<=2 : 0.992388, exp_rate<=3 : 0.997237, loss: 0.001808, avg_reader_cost: 0.03153 s, avg_batch_cost: 0.23460 s, avg_samples: 56.0, ips: 238.70441 samples/s, eta: 1 day, 1:38:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:33:18] ppocr INFO: epoch: [326/500], global_step: 68700, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.935245, exp_rate<=1 : 0.974149, exp_rate<=2 : 0.993907, exp_rate<=3 : 0.997938, loss: 0.001242, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18047 s, avg_samples: 56.0, ips: 310.30235 samples/s, eta: 1 day, 1:38:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:33:37] ppocr INFO: epoch: [326/500], global_step: 68800, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.939137, exp_rate<=1 : 0.975963, exp_rate<=2 : 0.994817, exp_rate<=3 : 0.998304, loss: 0.001784, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18371 s, avg_samples: 56.0, ips: 304.83552 samples/s, eta: 1 day, 1:38:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:33:57] ppocr INFO: epoch: [326/500], global_step: 68900, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.934379, exp_rate<=1 : 0.973683, exp_rate<=2 : 0.993783, exp_rate<=3 : 0.997730, loss: 0.001830, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20662 s, avg_samples: 56.0, ips: 271.03376 samples/s, eta: 1 day, 1:37:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:34:16] ppocr INFO: epoch: [326/500], global_step: 69000, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.936123, exp_rate<=1 : 0.974382, exp_rate<=2 : 0.994162, exp_rate<=3 : 0.997894, loss: 0.001722, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18745 s, avg_samples: 56.0, ips: 298.74144 samples/s, eta: 1 day, 1:37:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:34:35] ppocr INFO: epoch: [326/500], global_step: 69100, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.936424, exp_rate<=1 : 0.974593, exp_rate<=2 : 0.994311, exp_rate<=3 : 0.997944, loss: 0.001633, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18657 s, avg_samples: 56.0, ips: 300.15358 samples/s, eta: 1 day, 1:37:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:34:54] ppocr INFO: epoch: [326/500], global_step: 69200, lr: 0.000100, edit distance: 0.001406, exp_rate: 0.935042, exp_rate<=1 : 0.973620, exp_rate<=2 : 0.994276, exp_rate<=3 : 0.997931, loss: 0.001630, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19495 s, avg_samples: 56.0, ips: 287.25967 samples/s, eta: 1 day, 1:36:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:35:13] ppocr INFO: epoch: [326/500], global_step: 69300, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.934897, exp_rate<=1 : 0.973573, exp_rate<=2 : 0.993888, exp_rate<=3 : 0.997611, loss: 0.001992, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19102 s, avg_samples: 56.0, ips: 293.16335 samples/s, eta: 1 day, 1:36:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:35:34] ppocr INFO: epoch: [326/500], global_step: 69400, lr: 0.000100, edit distance: 0.001492, exp_rate: 0.933533, exp_rate<=1 : 0.973028, exp_rate<=2 : 0.993838, exp_rate<=3 : 0.997634, loss: 0.001871, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20349 s, avg_samples: 56.0, ips: 275.20446 samples/s, eta: 1 day, 1:36:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:35:54] ppocr INFO: epoch: [326/500], global_step: 69500, lr: 0.000100, edit distance: 0.001475, exp_rate: 0.933622, exp_rate<=1 : 0.973002, exp_rate<=2 : 0.993882, exp_rate<=3 : 0.997636, loss: 0.001277, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20627 s, avg_samples: 56.0, ips: 271.48404 samples/s, eta: 1 day, 1:36:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:36:14] ppocr INFO: epoch: [326/500], global_step: 69600, lr: 0.000100, edit distance: 0.001451, exp_rate: 0.934217, exp_rate<=1 : 0.973423, exp_rate<=2 : 0.993993, exp_rate<=3 : 0.997721, loss: 0.001749, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19776 s, avg_samples: 56.0, ips: 283.17632 samples/s, eta: 1 day, 1:35:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:36:32] ppocr INFO: epoch: [326/500], global_step: 69700, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.935124, exp_rate<=1 : 0.973721, exp_rate<=2 : 0.994060, exp_rate<=3 : 0.997795, loss: 0.001800, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17768 s, avg_samples: 56.0, ips: 315.18169 samples/s, eta: 1 day, 1:35:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:36:52] ppocr INFO: epoch: [326/500], global_step: 69800, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.934054, exp_rate<=1 : 0.973297, exp_rate<=2 : 0.993930, exp_rate<=3 : 0.997758, loss: 0.002265, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19685 s, avg_samples: 56.0, ips: 284.47417 samples/s, eta: 1 day, 1:35:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:37:10] ppocr INFO: epoch: [326/500], global_step: 69900, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.934594, exp_rate<=1 : 0.973645, exp_rate<=2 : 0.994054, exp_rate<=3 : 0.997804, loss: 0.001813, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18522 s, avg_samples: 56.0, ips: 302.34612 samples/s, eta: 1 day, 1:34:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:37:29] ppocr INFO: epoch: [326/500], global_step: 70000, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.933705, exp_rate<=1 : 0.973331, exp_rate<=2 : 0.994010, exp_rate<=3 : 0.997830, loss: 0.001612, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18438 s, avg_samples: 56.0, ips: 303.71660 samples/s, eta: 1 day, 1:34:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:37:50] ppocr INFO: epoch: [326/500], global_step: 70100, lr: 0.000100, edit distance: 0.001490, exp_rate: 0.932508, exp_rate<=1 : 0.972608, exp_rate<=2 : 0.993720, exp_rate<=3 : 0.997668, loss: 0.001619, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21228 s, avg_samples: 56.0, ips: 263.80247 samples/s, eta: 1 day, 1:34:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:38:10] ppocr INFO: epoch: [326/500], global_step: 70200, lr: 0.000100, edit distance: 0.001500, exp_rate: 0.931860, exp_rate<=1 : 0.972497, exp_rate<=2 : 0.993619, exp_rate<=3 : 0.997666, loss: 0.001500, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19837 s, avg_samples: 56.0, ips: 282.29617 samples/s, eta: 1 day, 1:33:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:38:28] ppocr INFO: epoch: [326/500], global_step: 70300, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.931692, exp_rate<=1 : 0.972730, exp_rate<=2 : 0.993755, exp_rate<=3 : 0.997733, loss: 0.002295, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18578 s, avg_samples: 56.0, ips: 301.43877 samples/s, eta: 1 day, 1:33:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:38:48] ppocr INFO: epoch: [326/500], global_step: 70400, lr: 0.000100, edit distance: 0.001534, exp_rate: 0.930893, exp_rate<=1 : 0.972412, exp_rate<=2 : 0.993481, exp_rate<=3 : 0.997568, loss: 0.002195, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19667 s, avg_samples: 56.0, ips: 284.73544 samples/s, eta: 1 day, 1:33:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:39:07] ppocr INFO: epoch: [326/500], global_step: 70500, lr: 0.000100, edit distance: 0.001538, exp_rate: 0.930898, exp_rate<=1 : 0.972315, exp_rate<=2 : 0.993409, exp_rate<=3 : 0.997541, loss: 0.002548, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19163 s, avg_samples: 56.0, ips: 292.23532 samples/s, eta: 1 day, 1:32:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:39:25] ppocr INFO: epoch: [326/500], global_step: 70600, lr: 0.000100, edit distance: 0.001564, exp_rate: 0.930499, exp_rate<=1 : 0.972217, exp_rate<=2 : 0.993255, exp_rate<=3 : 0.997500, loss: 0.002049, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18099 s, avg_samples: 56.0, ips: 309.40982 samples/s, eta: 1 day, 1:32:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:39:45] ppocr INFO: epoch: [326/500], global_step: 70700, lr: 0.000100, edit distance: 0.001615, exp_rate: 0.928433, exp_rate<=1 : 0.971390, exp_rate<=2 : 0.992822, exp_rate<=3 : 0.997241, loss: 0.002200, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19573 s, avg_samples: 56.0, ips: 286.11179 samples/s, eta: 1 day, 1:32:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:40:04] ppocr INFO: epoch: [326/500], global_step: 70800, lr: 0.000100, edit distance: 0.001641, exp_rate: 0.927510, exp_rate<=1 : 0.970780, exp_rate<=2 : 0.992384, exp_rate<=3 : 0.996954, loss: 0.001363, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19363 s, avg_samples: 56.0, ips: 289.21201 samples/s, eta: 1 day, 1:31:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:40:24] ppocr INFO: epoch: [326/500], global_step: 70900, lr: 0.000100, edit distance: 0.001638, exp_rate: 0.927152, exp_rate<=1 : 0.970572, exp_rate<=2 : 0.992299, exp_rate<=3 : 0.996969, loss: 0.001580, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19251 s, avg_samples: 56.0, ips: 290.89301 samples/s, eta: 1 day, 1:31:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:40:42] ppocr INFO: epoch: [326/500], global_step: 71000, lr: 0.000100, edit distance: 0.001635, exp_rate: 0.927421, exp_rate<=1 : 0.970823, exp_rate<=2 : 0.992389, exp_rate<=3 : 0.997035, loss: 0.001590, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17876 s, avg_samples: 56.0, ips: 313.26790 samples/s, eta: 1 day, 1:31:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:41:01] ppocr INFO: epoch: [326/500], global_step: 71100, lr: 0.000100, edit distance: 0.001643, exp_rate: 0.927217, exp_rate<=1 : 0.970834, exp_rate<=2 : 0.992428, exp_rate<=3 : 0.997050, loss: 0.002538, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19322 s, avg_samples: 56.0, ips: 289.83223 samples/s, eta: 1 day, 1:30:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:41:21] ppocr INFO: epoch: [326/500], global_step: 71200, lr: 0.000100, edit distance: 0.001658, exp_rate: 0.926335, exp_rate<=1 : 0.970447, exp_rate<=2 : 0.992239, exp_rate<=3 : 0.996972, loss: 0.002225, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20489 s, avg_samples: 56.0, ips: 273.32364 samples/s, eta: 1 day, 1:30:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:41:28] ppocr INFO: epoch: [326/500], global_step: 71240, lr: 0.000100, edit distance: 0.001658, exp_rate: 0.926374, exp_rate<=1 : 0.970492, exp_rate<=2 : 0.992270, exp_rate<=3 : 0.996988, loss: 0.001943, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.06993 s, avg_samples: 22.4, ips: 320.29860 samples/s, eta: 1 day, 1:30:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:41:30] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 06:41:42] ppocr INFO: epoch: [327/500], global_step: 71300, lr: 0.000100, edit distance: 0.001923, exp_rate: 0.923085, exp_rate<=1 : 0.972667, exp_rate<=2 : 0.990805, exp_rate<=3 : 0.995789, loss: 0.001524, avg_reader_cost: 0.02530 s, avg_batch_cost: 0.13960 s, avg_samples: 33.6, ips: 240.69035 samples/s, eta: 1 day, 1:30:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:42:02] ppocr INFO: epoch: [327/500], global_step: 71400, lr: 0.000100, edit distance: 0.001693, exp_rate: 0.930774, exp_rate<=1 : 0.974892, exp_rate<=2 : 0.993026, exp_rate<=3 : 0.997152, loss: 0.001681, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19440 s, avg_samples: 56.0, ips: 288.07183 samples/s, eta: 1 day, 1:30:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:42:21] ppocr INFO: epoch: [327/500], global_step: 71500, lr: 0.000100, edit distance: 0.001647, exp_rate: 0.932338, exp_rate<=1 : 0.975000, exp_rate<=2 : 0.994036, exp_rate<=3 : 0.997821, loss: 0.001570, avg_reader_cost: 0.00019 s, avg_batch_cost: 0.18965 s, avg_samples: 56.0, ips: 295.27747 samples/s, eta: 1 day, 1:29:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:42:38] ppocr INFO: epoch: [327/500], global_step: 71600, lr: 0.000100, edit distance: 0.001561, exp_rate: 0.935463, exp_rate<=1 : 0.976735, exp_rate<=2 : 0.995209, exp_rate<=3 : 0.998265, loss: 0.001724, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17435 s, avg_samples: 56.0, ips: 321.18407 samples/s, eta: 1 day, 1:29:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:42:56] ppocr INFO: epoch: [327/500], global_step: 71700, lr: 0.000100, edit distance: 0.001460, exp_rate: 0.937685, exp_rate<=1 : 0.976741, exp_rate<=2 : 0.995331, exp_rate<=3 : 0.998448, loss: 0.001758, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18054 s, avg_samples: 56.0, ips: 310.18063 samples/s, eta: 1 day, 1:28:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:43:15] ppocr INFO: epoch: [327/500], global_step: 71800, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.937854, exp_rate<=1 : 0.976747, exp_rate<=2 : 0.995283, exp_rate<=3 : 0.998372, loss: 0.001528, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18870 s, avg_samples: 56.0, ips: 296.77228 samples/s, eta: 1 day, 1:28:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:43:34] ppocr INFO: epoch: [327/500], global_step: 71900, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.937699, exp_rate<=1 : 0.976537, exp_rate<=2 : 0.994949, exp_rate<=3 : 0.998192, loss: 0.001832, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19173 s, avg_samples: 56.0, ips: 292.07124 samples/s, eta: 1 day, 1:28:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:43:53] ppocr INFO: epoch: [327/500], global_step: 72000, lr: 0.000100, edit distance: 0.001444, exp_rate: 0.937649, exp_rate<=1 : 0.976043, exp_rate<=2 : 0.994658, exp_rate<=3 : 0.998250, loss: 0.001836, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18315 s, avg_samples: 56.0, ips: 305.76589 samples/s, eta: 1 day, 1:27:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:44:12] ppocr INFO: epoch: [327/500], global_step: 72100, lr: 0.000100, edit distance: 0.001423, exp_rate: 0.937731, exp_rate<=1 : 0.976271, exp_rate<=2 : 0.994683, exp_rate<=3 : 0.998285, loss: 0.001175, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18701 s, avg_samples: 56.0, ips: 299.45109 samples/s, eta: 1 day, 1:27:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:44:30] ppocr INFO: epoch: [327/500], global_step: 72200, lr: 0.000100, edit distance: 0.001416, exp_rate: 0.937279, exp_rate<=1 : 0.976066, exp_rate<=2 : 0.994558, exp_rate<=3 : 0.998149, loss: 0.001802, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18812 s, avg_samples: 56.0, ips: 297.67448 samples/s, eta: 1 day, 1:26:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:44:49] ppocr INFO: epoch: [327/500], global_step: 72300, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.937071, exp_rate<=1 : 0.975735, exp_rate<=2 : 0.994278, exp_rate<=3 : 0.997960, loss: 0.002709, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18965 s, avg_samples: 56.0, ips: 295.28488 samples/s, eta: 1 day, 1:26:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:45:08] ppocr INFO: epoch: [327/500], global_step: 72400, lr: 0.000100, edit distance: 0.001443, exp_rate: 0.936604, exp_rate<=1 : 0.975502, exp_rate<=2 : 0.994133, exp_rate<=3 : 0.997954, loss: 0.001090, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18863 s, avg_samples: 56.0, ips: 296.88273 samples/s, eta: 1 day, 1:26:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:45:28] ppocr INFO: epoch: [327/500], global_step: 72500, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.936735, exp_rate<=1 : 0.975539, exp_rate<=2 : 0.994071, exp_rate<=3 : 0.997977, loss: 0.001576, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19573 s, avg_samples: 56.0, ips: 286.11288 samples/s, eta: 1 day, 1:25:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:45:47] ppocr INFO: epoch: [327/500], global_step: 72600, lr: 0.000100, edit distance: 0.001463, exp_rate: 0.936279, exp_rate<=1 : 0.975485, exp_rate<=2 : 0.994136, exp_rate<=3 : 0.998025, loss: 0.001950, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19644 s, avg_samples: 56.0, ips: 285.07408 samples/s, eta: 1 day, 1:25:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:46:08] ppocr INFO: epoch: [327/500], global_step: 72700, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.935492, exp_rate<=1 : 0.975044, exp_rate<=2 : 0.994080, exp_rate<=3 : 0.998018, loss: 0.001597, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20208 s, avg_samples: 56.0, ips: 277.12289 samples/s, eta: 1 day, 1:25:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:46:28] ppocr INFO: epoch: [327/500], global_step: 72800, lr: 0.000100, edit distance: 0.001479, exp_rate: 0.935315, exp_rate<=1 : 0.974926, exp_rate<=2 : 0.994073, exp_rate<=3 : 0.998055, loss: 0.001871, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19803 s, avg_samples: 56.0, ips: 282.79041 samples/s, eta: 1 day, 1:25:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:46:46] ppocr INFO: epoch: [327/500], global_step: 72900, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.935063, exp_rate<=1 : 0.974526, exp_rate<=2 : 0.993985, exp_rate<=3 : 0.998031, loss: 0.001325, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18121 s, avg_samples: 56.0, ips: 309.03204 samples/s, eta: 1 day, 1:24:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:47:05] ppocr INFO: epoch: [327/500], global_step: 73000, lr: 0.000100, edit distance: 0.001498, exp_rate: 0.934949, exp_rate<=1 : 0.974373, exp_rate<=2 : 0.993804, exp_rate<=3 : 0.997963, loss: 0.001331, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18868 s, avg_samples: 56.0, ips: 296.80419 samples/s, eta: 1 day, 1:24:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:47:23] ppocr INFO: epoch: [327/500], global_step: 73100, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.935016, exp_rate<=1 : 0.974324, exp_rate<=2 : 0.993848, exp_rate<=3 : 0.997964, loss: 0.002156, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18543 s, avg_samples: 56.0, ips: 301.99283 samples/s, eta: 1 day, 1:23:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:47:43] ppocr INFO: epoch: [327/500], global_step: 73200, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.934165, exp_rate<=1 : 0.974006, exp_rate<=2 : 0.993796, exp_rate<=3 : 0.997954, loss: 0.001905, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20002 s, avg_samples: 56.0, ips: 279.97306 samples/s, eta: 1 day, 1:23:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:48:03] ppocr INFO: epoch: [327/500], global_step: 73300, lr: 0.000100, edit distance: 0.001526, exp_rate: 0.933835, exp_rate<=1 : 0.973843, exp_rate<=2 : 0.993771, exp_rate<=3 : 0.997949, loss: 0.001281, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19397 s, avg_samples: 56.0, ips: 288.70090 samples/s, eta: 1 day, 1:23:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:48:22] ppocr INFO: epoch: [327/500], global_step: 73400, lr: 0.000100, edit distance: 0.001519, exp_rate: 0.933533, exp_rate<=1 : 0.973785, exp_rate<=2 : 0.993724, exp_rate<=3 : 0.997915, loss: 0.001896, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19862 s, avg_samples: 56.0, ips: 281.94203 samples/s, eta: 1 day, 1:23:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:48:43] ppocr INFO: epoch: [327/500], global_step: 73500, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.932811, exp_rate<=1 : 0.973468, exp_rate<=2 : 0.993620, exp_rate<=3 : 0.997865, loss: 0.001639, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20591 s, avg_samples: 56.0, ips: 271.95792 samples/s, eta: 1 day, 1:22:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:49:02] ppocr INFO: epoch: [327/500], global_step: 73600, lr: 0.000100, edit distance: 0.001522, exp_rate: 0.933036, exp_rate<=1 : 0.973474, exp_rate<=2 : 0.993619, exp_rate<=3 : 0.997855, loss: 0.001613, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18979 s, avg_samples: 56.0, ips: 295.06088 samples/s, eta: 1 day, 1:22:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:49:20] ppocr INFO: epoch: [327/500], global_step: 73700, lr: 0.000100, edit distance: 0.001534, exp_rate: 0.932926, exp_rate<=1 : 0.973393, exp_rate<=2 : 0.993551, exp_rate<=3 : 0.997841, loss: 0.002435, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17782 s, avg_samples: 56.0, ips: 314.92563 samples/s, eta: 1 day, 1:22:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:49:39] ppocr INFO: epoch: [327/500], global_step: 73800, lr: 0.000100, edit distance: 0.001538, exp_rate: 0.932618, exp_rate<=1 : 0.973260, exp_rate<=2 : 0.993499, exp_rate<=3 : 0.997808, loss: 0.002368, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19586 s, avg_samples: 56.0, ips: 285.91696 samples/s, eta: 1 day, 1:21:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:50:00] ppocr INFO: epoch: [327/500], global_step: 73900, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.932071, exp_rate<=1 : 0.972945, exp_rate<=2 : 0.993380, exp_rate<=3 : 0.997777, loss: 0.001607, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20701 s, avg_samples: 56.0, ips: 270.52455 samples/s, eta: 1 day, 1:21:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:50:16] ppocr INFO: epoch: [327/500], global_step: 73980, lr: 0.000100, edit distance: 0.001545, exp_rate: 0.931914, exp_rate<=1 : 0.972931, exp_rate<=2 : 0.993350, exp_rate<=3 : 0.997765, loss: 0.001795, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.15661 s, avg_samples: 44.8, ips: 286.06900 samples/s, eta: 1 day, 1:21:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:50:18] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 06:50:23] ppocr INFO: epoch: [328/500], global_step: 74000, lr: 0.000100, edit distance: 0.000941, exp_rate: 0.947545, exp_rate<=1 : 0.981548, exp_rate<=2 : 0.996429, exp_rate<=3 : 0.998767, loss: 0.001332, avg_reader_cost: 0.02537 s, avg_batch_cost: 0.06627 s, avg_samples: 11.2, ips: 169.00115 samples/s, eta: 1 day, 1:21:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:50:42] ppocr INFO: epoch: [328/500], global_step: 74100, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.941026, exp_rate<=1 : 0.976568, exp_rate<=2 : 0.995616, exp_rate<=3 : 0.999354, loss: 0.001438, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19236 s, avg_samples: 56.0, ips: 291.11584 samples/s, eta: 1 day, 1:21:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:51:01] ppocr INFO: epoch: [328/500], global_step: 74200, lr: 0.000100, edit distance: 0.001405, exp_rate: 0.940806, exp_rate<=1 : 0.977281, exp_rate<=2 : 0.994608, exp_rate<=3 : 0.998834, loss: 0.001376, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19629 s, avg_samples: 56.0, ips: 285.28629 samples/s, eta: 1 day, 1:21:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:51:21] ppocr INFO: epoch: [328/500], global_step: 74300, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.938463, exp_rate<=1 : 0.975789, exp_rate<=2 : 0.993304, exp_rate<=3 : 0.997926, loss: 0.001303, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19898 s, avg_samples: 56.0, ips: 281.44063 samples/s, eta: 1 day, 1:20:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:51:42] ppocr INFO: epoch: [328/500], global_step: 74400, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.937164, exp_rate<=1 : 0.975053, exp_rate<=2 : 0.993041, exp_rate<=3 : 0.997804, loss: 0.001710, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20891 s, avg_samples: 56.0, ips: 268.06137 samples/s, eta: 1 day, 1:20:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:52:02] ppocr INFO: epoch: [328/500], global_step: 74500, lr: 0.000100, edit distance: 0.001519, exp_rate: 0.935065, exp_rate<=1 : 0.974113, exp_rate<=2 : 0.992970, exp_rate<=3 : 0.997809, loss: 0.001958, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20194 s, avg_samples: 56.0, ips: 277.31657 samples/s, eta: 1 day, 1:20:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:52:21] ppocr INFO: epoch: [328/500], global_step: 74600, lr: 0.000100, edit distance: 0.001486, exp_rate: 0.935732, exp_rate<=1 : 0.974599, exp_rate<=2 : 0.993154, exp_rate<=3 : 0.997920, loss: 0.001625, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18485 s, avg_samples: 56.0, ips: 302.95218 samples/s, eta: 1 day, 1:20:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:52:41] ppocr INFO: epoch: [328/500], global_step: 74700, lr: 0.000100, edit distance: 0.001498, exp_rate: 0.934327, exp_rate<=1 : 0.973628, exp_rate<=2 : 0.992887, exp_rate<=3 : 0.997832, loss: 0.001349, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19883 s, avg_samples: 56.0, ips: 281.64691 samples/s, eta: 1 day, 1:19:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:53:00] ppocr INFO: epoch: [328/500], global_step: 74800, lr: 0.000100, edit distance: 0.001506, exp_rate: 0.934786, exp_rate<=1 : 0.974118, exp_rate<=2 : 0.993071, exp_rate<=3 : 0.997839, loss: 0.001472, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18650 s, avg_samples: 56.0, ips: 300.26729 samples/s, eta: 1 day, 1:19:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:53:18] ppocr INFO: epoch: [328/500], global_step: 74900, lr: 0.000100, edit distance: 0.001474, exp_rate: 0.935184, exp_rate<=1 : 0.974502, exp_rate<=2 : 0.993175, exp_rate<=3 : 0.997921, loss: 0.001176, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18480 s, avg_samples: 56.0, ips: 303.02289 samples/s, eta: 1 day, 1:19:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:53:38] ppocr INFO: epoch: [328/500], global_step: 75000, lr: 0.000100, edit distance: 0.001480, exp_rate: 0.934412, exp_rate<=1 : 0.974491, exp_rate<=2 : 0.993267, exp_rate<=3 : 0.997882, loss: 0.001356, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20174 s, avg_samples: 56.0, ips: 277.58788 samples/s, eta: 1 day, 1:18:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:53:57] ppocr INFO: epoch: [328/500], global_step: 75100, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.934809, exp_rate<=1 : 0.974532, exp_rate<=2 : 0.993161, exp_rate<=3 : 0.997678, loss: 0.002725, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18472 s, avg_samples: 56.0, ips: 303.15617 samples/s, eta: 1 day, 1:18:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:54:14] ppocr INFO: epoch: [328/500], global_step: 75200, lr: 0.000100, edit distance: 0.001516, exp_rate: 0.934310, exp_rate<=1 : 0.974673, exp_rate<=2 : 0.993403, exp_rate<=3 : 0.997767, loss: 0.002738, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17575 s, avg_samples: 56.0, ips: 318.64038 samples/s, eta: 1 day, 1:17:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:54:34] ppocr INFO: epoch: [328/500], global_step: 75300, lr: 0.000100, edit distance: 0.001533, exp_rate: 0.933177, exp_rate<=1 : 0.974288, exp_rate<=2 : 0.993298, exp_rate<=3 : 0.997617, loss: 0.002113, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19576 s, avg_samples: 56.0, ips: 286.05923 samples/s, eta: 1 day, 1:17:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:54:53] ppocr INFO: epoch: [328/500], global_step: 75400, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.933066, exp_rate<=1 : 0.974118, exp_rate<=2 : 0.993266, exp_rate<=3 : 0.997645, loss: 0.001440, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18557 s, avg_samples: 56.0, ips: 301.77468 samples/s, eta: 1 day, 1:17:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:55:11] ppocr INFO: epoch: [328/500], global_step: 75500, lr: 0.000100, edit distance: 0.001546, exp_rate: 0.932323, exp_rate<=1 : 0.973704, exp_rate<=2 : 0.993233, exp_rate<=3 : 0.997661, loss: 0.001835, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18508 s, avg_samples: 56.0, ips: 302.56747 samples/s, eta: 1 day, 1:16:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:55:29] ppocr INFO: epoch: [328/500], global_step: 75600, lr: 0.000100, edit distance: 0.001563, exp_rate: 0.932045, exp_rate<=1 : 0.973534, exp_rate<=2 : 0.993221, exp_rate<=3 : 0.997614, loss: 0.002617, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18016 s, avg_samples: 56.0, ips: 310.83516 samples/s, eta: 1 day, 1:16:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:55:49] ppocr INFO: epoch: [328/500], global_step: 75700, lr: 0.000100, edit distance: 0.001570, exp_rate: 0.931447, exp_rate<=1 : 0.973301, exp_rate<=2 : 0.993346, exp_rate<=3 : 0.997651, loss: 0.001762, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20071 s, avg_samples: 56.0, ips: 279.01550 samples/s, eta: 1 day, 1:16:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:56:08] ppocr INFO: epoch: [328/500], global_step: 75800, lr: 0.000100, edit distance: 0.001583, exp_rate: 0.931298, exp_rate<=1 : 0.973303, exp_rate<=2 : 0.993334, exp_rate<=3 : 0.997638, loss: 0.001894, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19146 s, avg_samples: 56.0, ips: 292.49475 samples/s, eta: 1 day, 1:15:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:56:27] ppocr INFO: epoch: [328/500], global_step: 75900, lr: 0.000100, edit distance: 0.001603, exp_rate: 0.930922, exp_rate<=1 : 0.973240, exp_rate<=2 : 0.993402, exp_rate<=3 : 0.997682, loss: 0.001591, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18675 s, avg_samples: 56.0, ips: 299.86282 samples/s, eta: 1 day, 1:15:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:56:46] ppocr INFO: epoch: [328/500], global_step: 76000, lr: 0.000100, edit distance: 0.001623, exp_rate: 0.930633, exp_rate<=1 : 0.973039, exp_rate<=2 : 0.993400, exp_rate<=3 : 0.997682, loss: 0.001774, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19373 s, avg_samples: 56.0, ips: 289.05538 samples/s, eta: 1 day, 1:15:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:57:07] ppocr INFO: epoch: [328/500], global_step: 76100, lr: 0.000100, edit distance: 0.001631, exp_rate: 0.929821, exp_rate<=1 : 0.972763, exp_rate<=2 : 0.993370, exp_rate<=3 : 0.997649, loss: 0.001558, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20053 s, avg_samples: 56.0, ips: 279.26033 samples/s, eta: 1 day, 1:14:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:57:26] ppocr INFO: epoch: [328/500], global_step: 76200, lr: 0.000100, edit distance: 0.001623, exp_rate: 0.929986, exp_rate<=1 : 0.972792, exp_rate<=2 : 0.993409, exp_rate<=3 : 0.997656, loss: 0.001917, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19355 s, avg_samples: 56.0, ips: 289.32866 samples/s, eta: 1 day, 1:14:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:57:46] ppocr INFO: epoch: [328/500], global_step: 76300, lr: 0.000100, edit distance: 0.001636, exp_rate: 0.929616, exp_rate<=1 : 0.972566, exp_rate<=2 : 0.993325, exp_rate<=3 : 0.997655, loss: 0.001845, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20413 s, avg_samples: 56.0, ips: 274.33231 samples/s, eta: 1 day, 1:14:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:58:05] ppocr INFO: epoch: [328/500], global_step: 76400, lr: 0.000100, edit distance: 0.001652, exp_rate: 0.929422, exp_rate<=1 : 0.972605, exp_rate<=2 : 0.993308, exp_rate<=3 : 0.997652, loss: 0.001797, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18762 s, avg_samples: 56.0, ips: 298.46922 samples/s, eta: 1 day, 1:13:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:58:25] ppocr INFO: epoch: [328/500], global_step: 76500, lr: 0.000100, edit distance: 0.001655, exp_rate: 0.929423, exp_rate<=1 : 0.972616, exp_rate<=2 : 0.993354, exp_rate<=3 : 0.997612, loss: 0.002070, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.36746 samples/s, eta: 1 day, 1:13:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:58:43] ppocr INFO: epoch: [328/500], global_step: 76600, lr: 0.000100, edit distance: 0.001666, exp_rate: 0.929269, exp_rate<=1 : 0.972562, exp_rate<=2 : 0.993370, exp_rate<=3 : 0.997579, loss: 0.002318, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18779 s, avg_samples: 56.0, ips: 298.20330 samples/s, eta: 1 day, 1:13:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:59:03] ppocr INFO: epoch: [328/500], global_step: 76700, lr: 0.000100, edit distance: 0.001683, exp_rate: 0.928545, exp_rate<=1 : 0.972208, exp_rate<=2 : 0.993185, exp_rate<=3 : 0.997482, loss: 0.002343, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19672 s, avg_samples: 56.0, ips: 284.66265 samples/s, eta: 1 day, 1:13:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:59:06] ppocr INFO: epoch: [328/500], global_step: 76720, lr: 0.000100, edit distance: 0.001685, exp_rate: 0.928549, exp_rate<=1 : 0.972228, exp_rate<=2 : 0.993208, exp_rate<=3 : 0.997489, loss: 0.002208, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03363 s, avg_samples: 11.2, ips: 333.06161 samples/s, eta: 1 day, 1:12:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:59:09] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 06:59:24] ppocr INFO: epoch: [329/500], global_step: 76800, lr: 0.000100, edit distance: 0.001588, exp_rate: 0.938139, exp_rate<=1 : 0.972966, exp_rate<=2 : 0.993198, exp_rate<=3 : 0.997214, loss: 0.001763, avg_reader_cost: 0.02691 s, avg_batch_cost: 0.17892 s, avg_samples: 44.8, ips: 250.38688 samples/s, eta: 1 day, 1:12:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 06:59:44] ppocr INFO: epoch: [329/500], global_step: 76900, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.935392, exp_rate<=1 : 0.974281, exp_rate<=2 : 0.993317, exp_rate<=3 : 0.997420, loss: 0.001743, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.20121 s, avg_samples: 56.0, ips: 278.31481 samples/s, eta: 1 day, 1:12:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:00:03] ppocr INFO: epoch: [329/500], global_step: 77000, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.938719, exp_rate<=1 : 0.976331, exp_rate<=2 : 0.994345, exp_rate<=3 : 0.997998, loss: 0.001519, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18487 s, avg_samples: 56.0, ips: 302.91412 samples/s, eta: 1 day, 1:12:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:00:23] ppocr INFO: epoch: [329/500], global_step: 77100, lr: 0.000100, edit distance: 0.001387, exp_rate: 0.941958, exp_rate<=1 : 0.977787, exp_rate<=2 : 0.994669, exp_rate<=3 : 0.998135, loss: 0.001482, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19514 s, avg_samples: 56.0, ips: 286.96687 samples/s, eta: 1 day, 1:11:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:00:42] ppocr INFO: epoch: [329/500], global_step: 77200, lr: 0.000100, edit distance: 0.001398, exp_rate: 0.941260, exp_rate<=1 : 0.977635, exp_rate<=2 : 0.994694, exp_rate<=3 : 0.998157, loss: 0.001700, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19923 s, avg_samples: 56.0, ips: 281.08605 samples/s, eta: 1 day, 1:11:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:01:01] ppocr INFO: epoch: [329/500], global_step: 77300, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.941039, exp_rate<=1 : 0.977261, exp_rate<=2 : 0.994750, exp_rate<=3 : 0.998195, loss: 0.001455, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18664 s, avg_samples: 56.0, ips: 300.04387 samples/s, eta: 1 day, 1:11:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:01:20] ppocr INFO: epoch: [329/500], global_step: 77400, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.940503, exp_rate<=1 : 0.976734, exp_rate<=2 : 0.994619, exp_rate<=3 : 0.998020, loss: 0.001403, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19135 s, avg_samples: 56.0, ips: 292.65483 samples/s, eta: 1 day, 1:11:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:01:39] ppocr INFO: epoch: [329/500], global_step: 77500, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.939843, exp_rate<=1 : 0.976781, exp_rate<=2 : 0.994478, exp_rate<=3 : 0.997891, loss: 0.002077, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18593 s, avg_samples: 56.0, ips: 301.19466 samples/s, eta: 1 day, 1:10:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:01:57] ppocr INFO: epoch: [329/500], global_step: 77600, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.939701, exp_rate<=1 : 0.976477, exp_rate<=2 : 0.994440, exp_rate<=3 : 0.997887, loss: 0.001452, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18499 s, avg_samples: 56.0, ips: 302.72690 samples/s, eta: 1 day, 1:10:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:02:16] ppocr INFO: epoch: [329/500], global_step: 77700, lr: 0.000100, edit distance: 0.001456, exp_rate: 0.938309, exp_rate<=1 : 0.975956, exp_rate<=2 : 0.994485, exp_rate<=3 : 0.997992, loss: 0.002101, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18519 s, avg_samples: 56.0, ips: 302.38548 samples/s, eta: 1 day, 1:09:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:02:35] ppocr INFO: epoch: [329/500], global_step: 77800, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.937912, exp_rate<=1 : 0.975716, exp_rate<=2 : 0.994561, exp_rate<=3 : 0.997982, loss: 0.001464, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18976 s, avg_samples: 56.0, ips: 295.11504 samples/s, eta: 1 day, 1:09:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:02:54] ppocr INFO: epoch: [329/500], global_step: 77900, lr: 0.000100, edit distance: 0.001508, exp_rate: 0.937028, exp_rate<=1 : 0.975604, exp_rate<=2 : 0.994493, exp_rate<=3 : 0.997987, loss: 0.001728, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18573 s, avg_samples: 56.0, ips: 301.50910 samples/s, eta: 1 day, 1:09:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:03:13] ppocr INFO: epoch: [329/500], global_step: 78000, lr: 0.000100, edit distance: 0.001506, exp_rate: 0.936391, exp_rate<=1 : 0.975474, exp_rate<=2 : 0.994535, exp_rate<=3 : 0.998079, loss: 0.001577, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19369 s, avg_samples: 56.0, ips: 289.12026 samples/s, eta: 1 day, 1:08:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:03:32] ppocr INFO: epoch: [329/500], global_step: 78100, lr: 0.000100, edit distance: 0.001529, exp_rate: 0.935540, exp_rate<=1 : 0.975056, exp_rate<=2 : 0.994295, exp_rate<=3 : 0.997920, loss: 0.001261, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.35685 samples/s, eta: 1 day, 1:08:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:03:51] ppocr INFO: epoch: [329/500], global_step: 78200, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.935653, exp_rate<=1 : 0.975062, exp_rate<=2 : 0.994403, exp_rate<=3 : 0.997934, loss: 0.001551, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18261 s, avg_samples: 56.0, ips: 306.66569 samples/s, eta: 1 day, 1:08:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:04:10] ppocr INFO: epoch: [329/500], global_step: 78300, lr: 0.000100, edit distance: 0.001512, exp_rate: 0.935377, exp_rate<=1 : 0.974749, exp_rate<=2 : 0.994184, exp_rate<=3 : 0.997820, loss: 0.001451, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19333 s, avg_samples: 56.0, ips: 289.66030 samples/s, eta: 1 day, 1:07:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:04:28] ppocr INFO: epoch: [329/500], global_step: 78400, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.935227, exp_rate<=1 : 0.974655, exp_rate<=2 : 0.994146, exp_rate<=3 : 0.997830, loss: 0.001752, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18156 s, avg_samples: 56.0, ips: 308.43108 samples/s, eta: 1 day, 1:07:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:04:47] ppocr INFO: epoch: [329/500], global_step: 78500, lr: 0.000100, edit distance: 0.001564, exp_rate: 0.934761, exp_rate<=1 : 0.974623, exp_rate<=2 : 0.994241, exp_rate<=3 : 0.997872, loss: 0.001324, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18589 s, avg_samples: 56.0, ips: 301.25918 samples/s, eta: 1 day, 1:06:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:05:05] ppocr INFO: epoch: [329/500], global_step: 78600, lr: 0.000100, edit distance: 0.001573, exp_rate: 0.934822, exp_rate<=1 : 0.974541, exp_rate<=2 : 0.994126, exp_rate<=3 : 0.997808, loss: 0.001214, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18445 s, avg_samples: 56.0, ips: 303.60487 samples/s, eta: 1 day, 1:06:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:05:24] ppocr INFO: epoch: [329/500], global_step: 78700, lr: 0.000100, edit distance: 0.001586, exp_rate: 0.934530, exp_rate<=1 : 0.974378, exp_rate<=2 : 0.994048, exp_rate<=3 : 0.997817, loss: 0.002009, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18544 s, avg_samples: 56.0, ips: 301.98839 samples/s, eta: 1 day, 1:06:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:05:43] ppocr INFO: epoch: [329/500], global_step: 78800, lr: 0.000100, edit distance: 0.001581, exp_rate: 0.934153, exp_rate<=1 : 0.974168, exp_rate<=2 : 0.993935, exp_rate<=3 : 0.997749, loss: 0.001491, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19072 s, avg_samples: 56.0, ips: 293.63095 samples/s, eta: 1 day, 1:05:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:06:03] ppocr INFO: epoch: [329/500], global_step: 78900, lr: 0.000100, edit distance: 0.001575, exp_rate: 0.934002, exp_rate<=1 : 0.974081, exp_rate<=2 : 0.993891, exp_rate<=3 : 0.997765, loss: 0.001953, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20131 s, avg_samples: 56.0, ips: 278.17726 samples/s, eta: 1 day, 1:05:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:06:24] ppocr INFO: epoch: [329/500], global_step: 79000, lr: 0.000100, edit distance: 0.001568, exp_rate: 0.933604, exp_rate<=1 : 0.973920, exp_rate<=2 : 0.993858, exp_rate<=3 : 0.997765, loss: 0.001661, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20393 s, avg_samples: 56.0, ips: 274.60841 samples/s, eta: 1 day, 1:05:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:06:42] ppocr INFO: epoch: [329/500], global_step: 79100, lr: 0.000100, edit distance: 0.001552, exp_rate: 0.933878, exp_rate<=1 : 0.974079, exp_rate<=2 : 0.993943, exp_rate<=3 : 0.997815, loss: 0.001228, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18566 s, avg_samples: 56.0, ips: 301.63341 samples/s, eta: 1 day, 1:04:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:07:02] ppocr INFO: epoch: [329/500], global_step: 79200, lr: 0.000100, edit distance: 0.001548, exp_rate: 0.933956, exp_rate<=1 : 0.974040, exp_rate<=2 : 0.993913, exp_rate<=3 : 0.997803, loss: 0.001448, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19525 s, avg_samples: 56.0, ips: 286.80845 samples/s, eta: 1 day, 1:04:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:07:22] ppocr INFO: epoch: [329/500], global_step: 79300, lr: 0.000100, edit distance: 0.001562, exp_rate: 0.933410, exp_rate<=1 : 0.973843, exp_rate<=2 : 0.993865, exp_rate<=3 : 0.997800, loss: 0.002101, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20138 s, avg_samples: 56.0, ips: 278.08127 samples/s, eta: 1 day, 1:04:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:07:42] ppocr INFO: epoch: [329/500], global_step: 79400, lr: 0.000100, edit distance: 0.001566, exp_rate: 0.933123, exp_rate<=1 : 0.973744, exp_rate<=2 : 0.993815, exp_rate<=3 : 0.997764, loss: 0.001897, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20032 s, avg_samples: 56.0, ips: 279.55082 samples/s, eta: 1 day, 1:04:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:07:54] ppocr INFO: epoch: [329/500], global_step: 79460, lr: 0.000100, edit distance: 0.001569, exp_rate: 0.932801, exp_rate<=1 : 0.973574, exp_rate<=2 : 0.993787, exp_rate<=3 : 0.997749, loss: 0.001632, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.12315 s, avg_samples: 33.6, ips: 272.84823 samples/s, eta: 1 day, 1:04:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:07:56] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 07:08:05] ppocr INFO: epoch: [330/500], global_step: 79500, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.939678, exp_rate<=1 : 0.975537, exp_rate<=2 : 0.992717, exp_rate<=3 : 0.997919, loss: 0.001784, avg_reader_cost: 0.02614 s, avg_batch_cost: 0.10308 s, avg_samples: 22.4, ips: 217.29811 samples/s, eta: 1 day, 1:04:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:08:23] ppocr INFO: epoch: [330/500], global_step: 79600, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.939546, exp_rate<=1 : 0.974206, exp_rate<=2 : 0.994315, exp_rate<=3 : 0.998495, loss: 0.001770, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18693 s, avg_samples: 56.0, ips: 299.57657 samples/s, eta: 1 day, 1:03:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:08:42] ppocr INFO: epoch: [330/500], global_step: 79700, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.938187, exp_rate<=1 : 0.973596, exp_rate<=2 : 0.993704, exp_rate<=3 : 0.998134, loss: 0.002555, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18712 s, avg_samples: 56.0, ips: 299.27913 samples/s, eta: 1 day, 1:03:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:09:00] ppocr INFO: epoch: [330/500], global_step: 79800, lr: 0.000100, edit distance: 0.001384, exp_rate: 0.938228, exp_rate<=1 : 0.974642, exp_rate<=2 : 0.994101, exp_rate<=3 : 0.998347, loss: 0.000977, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18453 s, avg_samples: 56.0, ips: 303.46995 samples/s, eta: 1 day, 1:03:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:09:21] ppocr INFO: epoch: [330/500], global_step: 79900, lr: 0.000100, edit distance: 0.001393, exp_rate: 0.936726, exp_rate<=1 : 0.974280, exp_rate<=2 : 0.994331, exp_rate<=3 : 0.998258, loss: 0.001936, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20640 s, avg_samples: 56.0, ips: 271.31405 samples/s, eta: 1 day, 1:02:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:09:40] ppocr INFO: epoch: [330/500], global_step: 80000, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.937096, exp_rate<=1 : 0.974495, exp_rate<=2 : 0.994319, exp_rate<=3 : 0.998317, loss: 0.001178, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19245 s, avg_samples: 56.0, ips: 290.98749 samples/s, eta: 1 day, 1:02:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:10:00] ppocr INFO: epoch: [330/500], global_step: 80100, lr: 0.000100, edit distance: 0.001428, exp_rate: 0.937116, exp_rate<=1 : 0.974500, exp_rate<=2 : 0.994124, exp_rate<=3 : 0.998108, loss: 0.001626, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19718 s, avg_samples: 56.0, ips: 284.00586 samples/s, eta: 1 day, 1:02:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:10:20] ppocr INFO: epoch: [330/500], global_step: 80200, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.935858, exp_rate<=1 : 0.973746, exp_rate<=2 : 0.993905, exp_rate<=3 : 0.997944, loss: 0.001478, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19658 s, avg_samples: 56.0, ips: 284.86455 samples/s, eta: 1 day, 1:01:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:10:38] ppocr INFO: epoch: [330/500], global_step: 80300, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.936271, exp_rate<=1 : 0.974304, exp_rate<=2 : 0.994159, exp_rate<=3 : 0.998019, loss: 0.001152, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18382 s, avg_samples: 56.0, ips: 304.64531 samples/s, eta: 1 day, 1:01:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:10:56] ppocr INFO: epoch: [330/500], global_step: 80400, lr: 0.000100, edit distance: 0.001474, exp_rate: 0.937049, exp_rate<=1 : 0.974685, exp_rate<=2 : 0.994272, exp_rate<=3 : 0.998064, loss: 0.001512, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17569 s, avg_samples: 56.0, ips: 318.74335 samples/s, eta: 1 day, 1:01:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:11:17] ppocr INFO: epoch: [330/500], global_step: 80500, lr: 0.000100, edit distance: 0.001486, exp_rate: 0.935166, exp_rate<=1 : 0.973588, exp_rate<=2 : 0.993866, exp_rate<=3 : 0.997869, loss: 0.001563, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.21106 s, avg_samples: 56.0, ips: 265.32831 samples/s, eta: 1 day, 1:00:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:11:36] ppocr INFO: epoch: [330/500], global_step: 80600, lr: 0.000100, edit distance: 0.001506, exp_rate: 0.934817, exp_rate<=1 : 0.973553, exp_rate<=2 : 0.993932, exp_rate<=3 : 0.997883, loss: 0.001228, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18706 s, avg_samples: 56.0, ips: 299.36720 samples/s, eta: 1 day, 1:00:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:11:54] ppocr INFO: epoch: [330/500], global_step: 80700, lr: 0.000100, edit distance: 0.001504, exp_rate: 0.934565, exp_rate<=1 : 0.973768, exp_rate<=2 : 0.993951, exp_rate<=3 : 0.997964, loss: 0.002038, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18788 s, avg_samples: 56.0, ips: 298.05848 samples/s, eta: 1 day, 1:00:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:12:14] ppocr INFO: epoch: [330/500], global_step: 80800, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.933685, exp_rate<=1 : 0.973362, exp_rate<=2 : 0.993856, exp_rate<=3 : 0.997937, loss: 0.001466, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19079 s, avg_samples: 56.0, ips: 293.51467 samples/s, eta: 1 day, 0:59:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:12:34] ppocr INFO: epoch: [330/500], global_step: 80900, lr: 0.000100, edit distance: 0.001505, exp_rate: 0.933145, exp_rate<=1 : 0.973099, exp_rate<=2 : 0.993762, exp_rate<=3 : 0.997944, loss: 0.001513, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20827 s, avg_samples: 56.0, ips: 268.87939 samples/s, eta: 1 day, 0:59:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:12:54] ppocr INFO: epoch: [330/500], global_step: 81000, lr: 0.000100, edit distance: 0.001501, exp_rate: 0.933064, exp_rate<=1 : 0.973249, exp_rate<=2 : 0.993707, exp_rate<=3 : 0.997930, loss: 0.002166, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19511 s, avg_samples: 56.0, ips: 287.01119 samples/s, eta: 1 day, 0:59:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:13:14] ppocr INFO: epoch: [330/500], global_step: 81100, lr: 0.000100, edit distance: 0.001542, exp_rate: 0.932039, exp_rate<=1 : 0.972833, exp_rate<=2 : 0.993578, exp_rate<=3 : 0.997926, loss: 0.002331, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19619 s, avg_samples: 56.0, ips: 285.43761 samples/s, eta: 1 day, 0:59:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:13:32] ppocr INFO: epoch: [330/500], global_step: 81200, lr: 0.000100, edit distance: 0.001588, exp_rate: 0.931175, exp_rate<=1 : 0.972514, exp_rate<=2 : 0.993530, exp_rate<=3 : 0.997885, loss: 0.002350, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18259 s, avg_samples: 56.0, ips: 306.70645 samples/s, eta: 1 day, 0:58:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:13:51] ppocr INFO: epoch: [330/500], global_step: 81300, lr: 0.000100, edit distance: 0.001640, exp_rate: 0.930861, exp_rate<=1 : 0.972440, exp_rate<=2 : 0.993535, exp_rate<=3 : 0.997856, loss: 0.001905, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18839 s, avg_samples: 56.0, ips: 297.26241 samples/s, eta: 1 day, 0:58:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:14:10] ppocr INFO: epoch: [330/500], global_step: 81400, lr: 0.000100, edit distance: 0.001647, exp_rate: 0.930359, exp_rate<=1 : 0.972431, exp_rate<=2 : 0.993559, exp_rate<=3 : 0.997922, loss: 0.001660, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19172 s, avg_samples: 56.0, ips: 292.08589 samples/s, eta: 1 day, 0:57:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:14:29] ppocr INFO: epoch: [330/500], global_step: 81500, lr: 0.000100, edit distance: 0.001657, exp_rate: 0.929886, exp_rate<=1 : 0.972289, exp_rate<=2 : 0.993589, exp_rate<=3 : 0.997942, loss: 0.001677, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19445 s, avg_samples: 56.0, ips: 287.98919 samples/s, eta: 1 day, 0:57:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:14:49] ppocr INFO: epoch: [330/500], global_step: 81600, lr: 0.000100, edit distance: 0.001656, exp_rate: 0.929836, exp_rate<=1 : 0.972421, exp_rate<=2 : 0.993611, exp_rate<=3 : 0.997949, loss: 0.002192, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19633 s, avg_samples: 56.0, ips: 285.23520 samples/s, eta: 1 day, 0:57:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:15:07] ppocr INFO: epoch: [330/500], global_step: 81700, lr: 0.000100, edit distance: 0.001656, exp_rate: 0.929795, exp_rate<=1 : 0.972454, exp_rate<=2 : 0.993626, exp_rate<=3 : 0.997922, loss: 0.001980, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17949 s, avg_samples: 56.0, ips: 312.00318 samples/s, eta: 1 day, 0:56:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:15:26] ppocr INFO: epoch: [330/500], global_step: 81800, lr: 0.000100, edit distance: 0.001655, exp_rate: 0.929847, exp_rate<=1 : 0.972573, exp_rate<=2 : 0.993644, exp_rate<=3 : 0.997940, loss: 0.002078, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18811 s, avg_samples: 56.0, ips: 297.69369 samples/s, eta: 1 day, 0:56:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:15:45] ppocr INFO: epoch: [330/500], global_step: 81900, lr: 0.000100, edit distance: 0.001652, exp_rate: 0.929861, exp_rate<=1 : 0.972516, exp_rate<=2 : 0.993565, exp_rate<=3 : 0.997899, loss: 0.001533, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19271 s, avg_samples: 56.0, ips: 290.59469 samples/s, eta: 1 day, 0:56:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:16:05] ppocr INFO: epoch: [330/500], global_step: 82000, lr: 0.000100, edit distance: 0.001646, exp_rate: 0.929841, exp_rate<=1 : 0.972453, exp_rate<=2 : 0.993597, exp_rate<=3 : 0.997927, loss: 0.001737, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20332 s, avg_samples: 56.0, ips: 275.43452 samples/s, eta: 1 day, 0:56:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:16:25] ppocr INFO: epoch: [330/500], global_step: 82100, lr: 0.000100, edit distance: 0.001648, exp_rate: 0.929594, exp_rate<=1 : 0.972312, exp_rate<=2 : 0.993531, exp_rate<=3 : 0.997917, loss: 0.001820, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19979 s, avg_samples: 56.0, ips: 280.29210 samples/s, eta: 1 day, 0:55:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:16:44] ppocr INFO: epoch: [330/500], global_step: 82200, lr: 0.000100, edit distance: 0.001647, exp_rate: 0.929627, exp_rate<=1 : 0.972309, exp_rate<=2 : 0.993543, exp_rate<=3 : 0.997920, loss: 0.001354, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18621 s, avg_samples: 56.0, ips: 300.72942 samples/s, eta: 1 day, 0:55:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:16:46] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 07:16:47] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_330 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 07:17:07] ppocr INFO: epoch: [331/500], global_step: 82300, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.943087, exp_rate<=1 : 0.977298, exp_rate<=2 : 0.992876, exp_rate<=3 : 0.997351, loss: 0.001172, avg_reader_cost: 0.03327 s, avg_batch_cost: 0.22384 s, avg_samples: 56.0, ips: 250.17623 samples/s, eta: 1 day, 0:55:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:17:25] ppocr INFO: epoch: [331/500], global_step: 82400, lr: 0.000100, edit distance: 0.001214, exp_rate: 0.945811, exp_rate<=1 : 0.978759, exp_rate<=2 : 0.995407, exp_rate<=3 : 0.998229, loss: 0.001300, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18276 s, avg_samples: 56.0, ips: 306.40930 samples/s, eta: 1 day, 0:54:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:17:42] ppocr INFO: epoch: [331/500], global_step: 82500, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.946098, exp_rate<=1 : 0.979328, exp_rate<=2 : 0.995286, exp_rate<=3 : 0.998586, loss: 0.001724, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17586 s, avg_samples: 56.0, ips: 318.43584 samples/s, eta: 1 day, 0:54:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:18:03] ppocr INFO: epoch: [331/500], global_step: 82600, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.942152, exp_rate<=1 : 0.977390, exp_rate<=2 : 0.994717, exp_rate<=3 : 0.998317, loss: 0.001576, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20600 s, avg_samples: 56.0, ips: 271.84120 samples/s, eta: 1 day, 0:54:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:18:22] ppocr INFO: epoch: [331/500], global_step: 82700, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.940188, exp_rate<=1 : 0.975879, exp_rate<=2 : 0.993891, exp_rate<=3 : 0.997681, loss: 0.001465, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19109 s, avg_samples: 56.0, ips: 293.05751 samples/s, eta: 1 day, 0:53:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:18:44] ppocr INFO: epoch: [331/500], global_step: 82800, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.937674, exp_rate<=1 : 0.975395, exp_rate<=2 : 0.993809, exp_rate<=3 : 0.997669, loss: 0.001465, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21746 s, avg_samples: 56.0, ips: 257.51613 samples/s, eta: 1 day, 0:53:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:19:05] ppocr INFO: epoch: [331/500], global_step: 82900, lr: 0.000100, edit distance: 0.001405, exp_rate: 0.936137, exp_rate<=1 : 0.974238, exp_rate<=2 : 0.993540, exp_rate<=3 : 0.997581, loss: 0.001310, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21072 s, avg_samples: 56.0, ips: 265.75275 samples/s, eta: 1 day, 0:53:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:19:26] ppocr INFO: epoch: [331/500], global_step: 83000, lr: 0.000100, edit distance: 0.001398, exp_rate: 0.935212, exp_rate<=1 : 0.973885, exp_rate<=2 : 0.993611, exp_rate<=3 : 0.997663, loss: 0.001834, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20683 s, avg_samples: 56.0, ips: 270.74803 samples/s, eta: 1 day, 0:53:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:19:46] ppocr INFO: epoch: [331/500], global_step: 83100, lr: 0.000100, edit distance: 0.001446, exp_rate: 0.932826, exp_rate<=1 : 0.972769, exp_rate<=2 : 0.992951, exp_rate<=3 : 0.997080, loss: 0.002162, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20523 s, avg_samples: 56.0, ips: 272.86276 samples/s, eta: 1 day, 0:53:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:20:05] ppocr INFO: epoch: [331/500], global_step: 83200, lr: 0.000100, edit distance: 0.001435, exp_rate: 0.933112, exp_rate<=1 : 0.972981, exp_rate<=2 : 0.993038, exp_rate<=3 : 0.997097, loss: 0.001490, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19051 s, avg_samples: 56.0, ips: 293.94252 samples/s, eta: 1 day, 0:52:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:20:24] ppocr INFO: epoch: [331/500], global_step: 83300, lr: 0.000100, edit distance: 0.001443, exp_rate: 0.933111, exp_rate<=1 : 0.973190, exp_rate<=2 : 0.993278, exp_rate<=3 : 0.997205, loss: 0.001749, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19045 s, avg_samples: 56.0, ips: 294.03740 samples/s, eta: 1 day, 0:52:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:20:43] ppocr INFO: epoch: [331/500], global_step: 83400, lr: 0.000100, edit distance: 0.001446, exp_rate: 0.933211, exp_rate<=1 : 0.973477, exp_rate<=2 : 0.993290, exp_rate<=3 : 0.997301, loss: 0.001678, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18812 s, avg_samples: 56.0, ips: 297.68630 samples/s, eta: 1 day, 0:52:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:21:02] ppocr INFO: epoch: [331/500], global_step: 83500, lr: 0.000100, edit distance: 0.001451, exp_rate: 0.933214, exp_rate<=1 : 0.973391, exp_rate<=2 : 0.993178, exp_rate<=3 : 0.997193, loss: 0.001041, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18950 s, avg_samples: 56.0, ips: 295.50902 samples/s, eta: 1 day, 0:51:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:21:21] ppocr INFO: epoch: [331/500], global_step: 83600, lr: 0.000100, edit distance: 0.001473, exp_rate: 0.933033, exp_rate<=1 : 0.973324, exp_rate<=2 : 0.993153, exp_rate<=3 : 0.997244, loss: 0.002214, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18465 s, avg_samples: 56.0, ips: 303.28306 samples/s, eta: 1 day, 0:51:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:21:40] ppocr INFO: epoch: [331/500], global_step: 83700, lr: 0.000100, edit distance: 0.001486, exp_rate: 0.932416, exp_rate<=1 : 0.973070, exp_rate<=2 : 0.993142, exp_rate<=3 : 0.997256, loss: 0.002158, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18788 s, avg_samples: 56.0, ips: 298.06828 samples/s, eta: 1 day, 0:51:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:21:58] ppocr INFO: epoch: [331/500], global_step: 83800, lr: 0.000100, edit distance: 0.001504, exp_rate: 0.932169, exp_rate<=1 : 0.972976, exp_rate<=2 : 0.993092, exp_rate<=3 : 0.997265, loss: 0.002919, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18408 s, avg_samples: 56.0, ips: 304.21571 samples/s, eta: 1 day, 0:50:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:22:18] ppocr INFO: epoch: [331/500], global_step: 83900, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.931657, exp_rate<=1 : 0.972647, exp_rate<=2 : 0.993009, exp_rate<=3 : 0.997285, loss: 0.001271, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19570 s, avg_samples: 56.0, ips: 286.14769 samples/s, eta: 1 day, 0:50:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:22:36] ppocr INFO: epoch: [331/500], global_step: 84000, lr: 0.000100, edit distance: 0.001532, exp_rate: 0.931655, exp_rate<=1 : 0.972797, exp_rate<=2 : 0.993008, exp_rate<=3 : 0.997324, loss: 0.001905, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18312 s, avg_samples: 56.0, ips: 305.80649 samples/s, eta: 1 day, 0:50:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:22:56] ppocr INFO: epoch: [331/500], global_step: 84100, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.931231, exp_rate<=1 : 0.972531, exp_rate<=2 : 0.992992, exp_rate<=3 : 0.997348, loss: 0.001313, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19640 s, avg_samples: 56.0, ips: 285.13361 samples/s, eta: 1 day, 0:49:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:23:16] ppocr INFO: epoch: [331/500], global_step: 84200, lr: 0.000100, edit distance: 0.001545, exp_rate: 0.930385, exp_rate<=1 : 0.972098, exp_rate<=2 : 0.992949, exp_rate<=3 : 0.997362, loss: 0.001584, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20142 s, avg_samples: 56.0, ips: 278.02754 samples/s, eta: 1 day, 0:49:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:23:34] ppocr INFO: epoch: [331/500], global_step: 84300, lr: 0.000100, edit distance: 0.001560, exp_rate: 0.930639, exp_rate<=1 : 0.972366, exp_rate<=2 : 0.993062, exp_rate<=3 : 0.997411, loss: 0.001508, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17928 s, avg_samples: 56.0, ips: 312.35682 samples/s, eta: 1 day, 0:49:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:23:52] ppocr INFO: epoch: [331/500], global_step: 84400, lr: 0.000100, edit distance: 0.001563, exp_rate: 0.930825, exp_rate<=1 : 0.972446, exp_rate<=2 : 0.993092, exp_rate<=3 : 0.997408, loss: 0.002200, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18480 s, avg_samples: 56.0, ips: 303.03100 samples/s, eta: 1 day, 0:48:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:24:12] ppocr INFO: epoch: [331/500], global_step: 84500, lr: 0.000100, edit distance: 0.001567, exp_rate: 0.930660, exp_rate<=1 : 0.972392, exp_rate<=2 : 0.993099, exp_rate<=3 : 0.997432, loss: 0.001486, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.19401 s, avg_samples: 56.0, ips: 288.63819 samples/s, eta: 1 day, 0:48:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:24:29] ppocr INFO: epoch: [331/500], global_step: 84600, lr: 0.000100, edit distance: 0.001567, exp_rate: 0.931024, exp_rate<=1 : 0.972679, exp_rate<=2 : 0.993210, exp_rate<=3 : 0.997520, loss: 0.001757, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17490 s, avg_samples: 56.0, ips: 320.17874 samples/s, eta: 1 day, 0:47:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:24:48] ppocr INFO: epoch: [331/500], global_step: 84700, lr: 0.000100, edit distance: 0.001562, exp_rate: 0.931012, exp_rate<=1 : 0.972669, exp_rate<=2 : 0.993196, exp_rate<=3 : 0.997556, loss: 0.002435, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18977 s, avg_samples: 56.0, ips: 295.09091 samples/s, eta: 1 day, 0:47:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:25:06] ppocr INFO: epoch: [331/500], global_step: 84800, lr: 0.000100, edit distance: 0.001573, exp_rate: 0.930772, exp_rate<=1 : 0.972532, exp_rate<=2 : 0.993214, exp_rate<=3 : 0.997565, loss: 0.002121, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18041 s, avg_samples: 56.0, ips: 310.39967 samples/s, eta: 1 day, 0:47:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:25:25] ppocr INFO: epoch: [331/500], global_step: 84900, lr: 0.000100, edit distance: 0.001592, exp_rate: 0.930504, exp_rate<=1 : 0.972373, exp_rate<=2 : 0.993204, exp_rate<=3 : 0.997550, loss: 0.001772, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19237 s, avg_samples: 56.0, ips: 291.10267 samples/s, eta: 1 day, 0:46:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:25:34] ppocr INFO: epoch: [331/500], global_step: 84940, lr: 0.000100, edit distance: 0.001593, exp_rate: 0.930260, exp_rate<=1 : 0.972287, exp_rate<=2 : 0.993194, exp_rate<=3 : 0.997514, loss: 0.002508, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.08267 s, avg_samples: 22.4, ips: 270.94655 samples/s, eta: 1 day, 0:46:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:25:36] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 07:25:47] ppocr INFO: epoch: [332/500], global_step: 85000, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.945773, exp_rate<=1 : 0.980023, exp_rate<=2 : 0.995351, exp_rate<=3 : 0.999118, loss: 0.001343, avg_reader_cost: 0.02561 s, avg_batch_cost: 0.13436 s, avg_samples: 33.6, ips: 250.08026 samples/s, eta: 1 day, 0:46:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:26:06] ppocr INFO: epoch: [332/500], global_step: 85100, lr: 0.000100, edit distance: 0.001114, exp_rate: 0.944649, exp_rate<=1 : 0.980152, exp_rate<=2 : 0.994856, exp_rate<=3 : 0.998686, loss: 0.001827, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18764 s, avg_samples: 56.0, ips: 298.44666 samples/s, eta: 1 day, 0:46:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:26:25] ppocr INFO: epoch: [332/500], global_step: 85200, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.939293, exp_rate<=1 : 0.976167, exp_rate<=2 : 0.993871, exp_rate<=3 : 0.998211, loss: 0.002459, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19322 s, avg_samples: 56.0, ips: 289.82115 samples/s, eta: 1 day, 0:46:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:26:44] ppocr INFO: epoch: [332/500], global_step: 85300, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.939584, exp_rate<=1 : 0.976880, exp_rate<=2 : 0.993937, exp_rate<=3 : 0.998225, loss: 0.001259, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19172 s, avg_samples: 56.0, ips: 292.09966 samples/s, eta: 1 day, 0:45:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:27:03] ppocr INFO: epoch: [332/500], global_step: 85400, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.940503, exp_rate<=1 : 0.977141, exp_rate<=2 : 0.994141, exp_rate<=3 : 0.998134, loss: 0.001670, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18126 s, avg_samples: 56.0, ips: 308.95623 samples/s, eta: 1 day, 0:45:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:27:23] ppocr INFO: epoch: [332/500], global_step: 85500, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.938319, exp_rate<=1 : 0.976468, exp_rate<=2 : 0.993711, exp_rate<=3 : 0.998001, loss: 0.001146, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19916 s, avg_samples: 56.0, ips: 281.17852 samples/s, eta: 1 day, 0:45:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:27:41] ppocr INFO: epoch: [332/500], global_step: 85600, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.938687, exp_rate<=1 : 0.976560, exp_rate<=2 : 0.993733, exp_rate<=3 : 0.998134, loss: 0.001711, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18621 s, avg_samples: 56.0, ips: 300.72797 samples/s, eta: 1 day, 0:44:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:28:00] ppocr INFO: epoch: [332/500], global_step: 85700, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.938554, exp_rate<=1 : 0.976987, exp_rate<=2 : 0.993844, exp_rate<=3 : 0.998219, loss: 0.001679, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18799 s, avg_samples: 56.0, ips: 297.89281 samples/s, eta: 1 day, 0:44:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:28:20] ppocr INFO: epoch: [332/500], global_step: 85800, lr: 0.000100, edit distance: 0.001407, exp_rate: 0.937630, exp_rate<=1 : 0.976401, exp_rate<=2 : 0.993689, exp_rate<=3 : 0.998234, loss: 0.001230, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19716 s, avg_samples: 56.0, ips: 284.03172 samples/s, eta: 1 day, 0:43:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:28:38] ppocr INFO: epoch: [332/500], global_step: 85900, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.937823, exp_rate<=1 : 0.976297, exp_rate<=2 : 0.993634, exp_rate<=3 : 0.998140, loss: 0.001224, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18644 s, avg_samples: 56.0, ips: 300.36073 samples/s, eta: 1 day, 0:43:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:28:57] ppocr INFO: epoch: [332/500], global_step: 86000, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.937975, exp_rate<=1 : 0.976225, exp_rate<=2 : 0.993629, exp_rate<=3 : 0.998172, loss: 0.001435, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18695 s, avg_samples: 56.0, ips: 299.54423 samples/s, eta: 1 day, 0:43:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:29:17] ppocr INFO: epoch: [332/500], global_step: 86100, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.937582, exp_rate<=1 : 0.975819, exp_rate<=2 : 0.993682, exp_rate<=3 : 0.998136, loss: 0.001717, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19805 s, avg_samples: 56.0, ips: 282.76388 samples/s, eta: 1 day, 0:42:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:29:36] ppocr INFO: epoch: [332/500], global_step: 86200, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.937982, exp_rate<=1 : 0.976016, exp_rate<=2 : 0.993774, exp_rate<=3 : 0.998156, loss: 0.001802, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19118 s, avg_samples: 56.0, ips: 292.91415 samples/s, eta: 1 day, 0:42:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:29:56] ppocr INFO: epoch: [332/500], global_step: 86300, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.937414, exp_rate<=1 : 0.975821, exp_rate<=2 : 0.993756, exp_rate<=3 : 0.998028, loss: 0.001915, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19718 s, avg_samples: 56.0, ips: 284.00769 samples/s, eta: 1 day, 0:42:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:30:15] ppocr INFO: epoch: [332/500], global_step: 86400, lr: 0.000100, edit distance: 0.001386, exp_rate: 0.937509, exp_rate<=1 : 0.975869, exp_rate<=2 : 0.993802, exp_rate<=3 : 0.998045, loss: 0.001651, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.25660 samples/s, eta: 1 day, 0:41:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:30:35] ppocr INFO: epoch: [332/500], global_step: 86500, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.936474, exp_rate<=1 : 0.975301, exp_rate<=2 : 0.993576, exp_rate<=3 : 0.997963, loss: 0.001872, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20710 s, avg_samples: 56.0, ips: 270.39498 samples/s, eta: 1 day, 0:41:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:30:56] ppocr INFO: epoch: [332/500], global_step: 86600, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.935539, exp_rate<=1 : 0.974742, exp_rate<=2 : 0.993416, exp_rate<=3 : 0.997899, loss: 0.001788, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20486 s, avg_samples: 56.0, ips: 273.36159 samples/s, eta: 1 day, 0:41:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:31:15] ppocr INFO: epoch: [332/500], global_step: 86700, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.935593, exp_rate<=1 : 0.974933, exp_rate<=2 : 0.993475, exp_rate<=3 : 0.997951, loss: 0.001780, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18988 s, avg_samples: 56.0, ips: 294.92998 samples/s, eta: 1 day, 0:41:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:31:35] ppocr INFO: epoch: [332/500], global_step: 86800, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.935171, exp_rate<=1 : 0.974633, exp_rate<=2 : 0.993387, exp_rate<=3 : 0.997916, loss: 0.001918, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20395 s, avg_samples: 56.0, ips: 274.57281 samples/s, eta: 1 day, 0:41:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:31:54] ppocr INFO: epoch: [332/500], global_step: 86900, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.934827, exp_rate<=1 : 0.974583, exp_rate<=2 : 0.993507, exp_rate<=3 : 0.997958, loss: 0.002291, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18923 s, avg_samples: 56.0, ips: 295.93005 samples/s, eta: 1 day, 0:40:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:32:12] ppocr INFO: epoch: [332/500], global_step: 87000, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.934135, exp_rate<=1 : 0.974376, exp_rate<=2 : 0.993402, exp_rate<=3 : 0.997879, loss: 0.002125, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17962 s, avg_samples: 56.0, ips: 311.76864 samples/s, eta: 1 day, 0:40:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:32:31] ppocr INFO: epoch: [332/500], global_step: 87100, lr: 0.000100, edit distance: 0.001477, exp_rate: 0.933275, exp_rate<=1 : 0.974055, exp_rate<=2 : 0.993347, exp_rate<=3 : 0.997864, loss: 0.001781, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18565 s, avg_samples: 56.0, ips: 301.63522 samples/s, eta: 1 day, 0:39:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:32:51] ppocr INFO: epoch: [332/500], global_step: 87200, lr: 0.000100, edit distance: 0.001486, exp_rate: 0.932810, exp_rate<=1 : 0.973744, exp_rate<=2 : 0.993389, exp_rate<=3 : 0.997864, loss: 0.001532, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20445 s, avg_samples: 56.0, ips: 273.90581 samples/s, eta: 1 day, 0:39:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:33:10] ppocr INFO: epoch: [332/500], global_step: 87300, lr: 0.000100, edit distance: 0.001492, exp_rate: 0.932514, exp_rate<=1 : 0.973825, exp_rate<=2 : 0.993467, exp_rate<=3 : 0.997889, loss: 0.001863, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.24931 samples/s, eta: 1 day, 0:39:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:33:29] ppocr INFO: epoch: [332/500], global_step: 87400, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.931807, exp_rate<=1 : 0.973457, exp_rate<=2 : 0.993353, exp_rate<=3 : 0.997818, loss: 0.002741, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18930 s, avg_samples: 56.0, ips: 295.82756 samples/s, eta: 1 day, 0:38:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:33:48] ppocr INFO: epoch: [332/500], global_step: 87500, lr: 0.000100, edit distance: 0.001530, exp_rate: 0.931300, exp_rate<=1 : 0.973272, exp_rate<=2 : 0.993232, exp_rate<=3 : 0.997779, loss: 0.002094, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19205 s, avg_samples: 56.0, ips: 291.58973 samples/s, eta: 1 day, 0:38:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:34:08] ppocr INFO: epoch: [332/500], global_step: 87600, lr: 0.000100, edit distance: 0.001547, exp_rate: 0.930861, exp_rate<=1 : 0.973052, exp_rate<=2 : 0.993217, exp_rate<=3 : 0.997762, loss: 0.001658, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19414 s, avg_samples: 56.0, ips: 288.45841 samples/s, eta: 1 day, 0:38:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:34:23] ppocr INFO: epoch: [332/500], global_step: 87680, lr: 0.000100, edit distance: 0.001549, exp_rate: 0.930832, exp_rate<=1 : 0.973049, exp_rate<=2 : 0.993191, exp_rate<=3 : 0.997691, loss: 0.001846, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.15114 s, avg_samples: 44.8, ips: 296.42049 samples/s, eta: 1 day, 0:37:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:34:25] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 07:34:30] ppocr INFO: epoch: [333/500], global_step: 87700, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.938799, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.996154, exp_rate<=3 : 1.000000, loss: 0.001274, avg_reader_cost: 0.02625 s, avg_batch_cost: 0.06685 s, avg_samples: 11.2, ips: 167.54804 samples/s, eta: 1 day, 0:38:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:34:48] ppocr INFO: epoch: [333/500], global_step: 87800, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.938553, exp_rate<=1 : 0.978446, exp_rate<=2 : 0.995313, exp_rate<=3 : 0.998848, loss: 0.001829, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18713 s, avg_samples: 56.0, ips: 299.25249 samples/s, eta: 1 day, 0:37:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:35:05] ppocr INFO: epoch: [333/500], global_step: 87900, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.942806, exp_rate<=1 : 0.978121, exp_rate<=2 : 0.995751, exp_rate<=3 : 0.998751, loss: 0.001438, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17198 s, avg_samples: 56.0, ips: 325.61329 samples/s, eta: 1 day, 0:37:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:35:24] ppocr INFO: epoch: [333/500], global_step: 88000, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.942633, exp_rate<=1 : 0.978438, exp_rate<=2 : 0.995859, exp_rate<=3 : 0.998637, loss: 0.001846, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18504 s, avg_samples: 56.0, ips: 302.63379 samples/s, eta: 1 day, 0:36:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:35:43] ppocr INFO: epoch: [333/500], global_step: 88100, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.941389, exp_rate<=1 : 0.978329, exp_rate<=2 : 0.995613, exp_rate<=3 : 0.998488, loss: 0.001424, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18813 s, avg_samples: 56.0, ips: 297.66691 samples/s, eta: 1 day, 0:36:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:36:03] ppocr INFO: epoch: [333/500], global_step: 88200, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.940388, exp_rate<=1 : 0.978368, exp_rate<=2 : 0.995694, exp_rate<=3 : 0.998446, loss: 0.002254, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19940 s, avg_samples: 56.0, ips: 280.84700 samples/s, eta: 1 day, 0:36:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:36:23] ppocr INFO: epoch: [333/500], global_step: 88300, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.938684, exp_rate<=1 : 0.977602, exp_rate<=2 : 0.994992, exp_rate<=3 : 0.998277, loss: 0.001690, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19982 s, avg_samples: 56.0, ips: 280.24923 samples/s, eta: 1 day, 0:36:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:36:41] ppocr INFO: epoch: [333/500], global_step: 88400, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.938657, exp_rate<=1 : 0.977055, exp_rate<=2 : 0.994893, exp_rate<=3 : 0.998088, loss: 0.001888, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18654 s, avg_samples: 56.0, ips: 300.19930 samples/s, eta: 1 day, 0:35:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:37:02] ppocr INFO: epoch: [333/500], global_step: 88500, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.936755, exp_rate<=1 : 0.976014, exp_rate<=2 : 0.994727, exp_rate<=3 : 0.997957, loss: 0.001553, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20635 s, avg_samples: 56.0, ips: 271.38229 samples/s, eta: 1 day, 0:35:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:37:21] ppocr INFO: epoch: [333/500], global_step: 88600, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.937210, exp_rate<=1 : 0.975822, exp_rate<=2 : 0.994441, exp_rate<=3 : 0.997829, loss: 0.001281, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19242 s, avg_samples: 56.0, ips: 291.02860 samples/s, eta: 1 day, 0:35:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:37:40] ppocr INFO: epoch: [333/500], global_step: 88700, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.937654, exp_rate<=1 : 0.975846, exp_rate<=2 : 0.994416, exp_rate<=3 : 0.997879, loss: 0.002005, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18164 s, avg_samples: 56.0, ips: 308.30043 samples/s, eta: 1 day, 0:34:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:37:59] ppocr INFO: epoch: [333/500], global_step: 88800, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.936521, exp_rate<=1 : 0.975335, exp_rate<=2 : 0.994415, exp_rate<=3 : 0.997861, loss: 0.001166, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19746 s, avg_samples: 56.0, ips: 283.60072 samples/s, eta: 1 day, 0:34:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:38:19] ppocr INFO: epoch: [333/500], global_step: 88900, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.935456, exp_rate<=1 : 0.974850, exp_rate<=2 : 0.994160, exp_rate<=3 : 0.997894, loss: 0.001339, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19534 s, avg_samples: 56.0, ips: 286.68433 samples/s, eta: 1 day, 0:34:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:38:38] ppocr INFO: epoch: [333/500], global_step: 89000, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.935468, exp_rate<=1 : 0.974739, exp_rate<=2 : 0.994161, exp_rate<=3 : 0.997896, loss: 0.001816, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19122 s, avg_samples: 56.0, ips: 292.85372 samples/s, eta: 1 day, 0:33:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:38:57] ppocr INFO: epoch: [333/500], global_step: 89100, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.935685, exp_rate<=1 : 0.974765, exp_rate<=2 : 0.994100, exp_rate<=3 : 0.997876, loss: 0.001537, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18536 s, avg_samples: 56.0, ips: 302.10668 samples/s, eta: 1 day, 0:33:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:39:15] ppocr INFO: epoch: [333/500], global_step: 89200, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.935075, exp_rate<=1 : 0.974680, exp_rate<=2 : 0.994127, exp_rate<=3 : 0.997860, loss: 0.002074, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18070 s, avg_samples: 56.0, ips: 309.91075 samples/s, eta: 1 day, 0:33:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:39:34] ppocr INFO: epoch: [333/500], global_step: 89300, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.935076, exp_rate<=1 : 0.974662, exp_rate<=2 : 0.994200, exp_rate<=3 : 0.997915, loss: 0.001632, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18940 s, avg_samples: 56.0, ips: 295.67229 samples/s, eta: 1 day, 0:32:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:39:54] ppocr INFO: epoch: [333/500], global_step: 89400, lr: 0.000100, edit distance: 0.001428, exp_rate: 0.934700, exp_rate<=1 : 0.974494, exp_rate<=2 : 0.994149, exp_rate<=3 : 0.997927, loss: 0.001890, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20225 s, avg_samples: 56.0, ips: 276.88875 samples/s, eta: 1 day, 0:32:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:40:14] ppocr INFO: epoch: [333/500], global_step: 89500, lr: 0.000100, edit distance: 0.001431, exp_rate: 0.934058, exp_rate<=1 : 0.974268, exp_rate<=2 : 0.994123, exp_rate<=3 : 0.997930, loss: 0.001240, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19709 s, avg_samples: 56.0, ips: 284.13710 samples/s, eta: 1 day, 0:32:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:40:34] ppocr INFO: epoch: [333/500], global_step: 89600, lr: 0.000100, edit distance: 0.001443, exp_rate: 0.933517, exp_rate<=1 : 0.973993, exp_rate<=2 : 0.993990, exp_rate<=3 : 0.997874, loss: 0.001427, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19890 s, avg_samples: 56.0, ips: 281.55186 samples/s, eta: 1 day, 0:31:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:40:52] ppocr INFO: epoch: [333/500], global_step: 89700, lr: 0.000100, edit distance: 0.001441, exp_rate: 0.933636, exp_rate<=1 : 0.974168, exp_rate<=2 : 0.994033, exp_rate<=3 : 0.997882, loss: 0.001299, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18395 s, avg_samples: 56.0, ips: 304.43782 samples/s, eta: 1 day, 0:31:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:41:11] ppocr INFO: epoch: [333/500], global_step: 89800, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.933359, exp_rate<=1 : 0.974148, exp_rate<=2 : 0.994077, exp_rate<=3 : 0.997951, loss: 0.001823, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19335 s, avg_samples: 56.0, ips: 289.62472 samples/s, eta: 1 day, 0:31:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:41:31] ppocr INFO: epoch: [333/500], global_step: 89900, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.933190, exp_rate<=1 : 0.974178, exp_rate<=2 : 0.994106, exp_rate<=3 : 0.997961, loss: 0.002085, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19771 s, avg_samples: 56.0, ips: 283.23953 samples/s, eta: 1 day, 0:30:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:41:51] ppocr INFO: epoch: [333/500], global_step: 90000, lr: 0.000100, edit distance: 0.001444, exp_rate: 0.932411, exp_rate<=1 : 0.973906, exp_rate<=2 : 0.994041, exp_rate<=3 : 0.997928, loss: 0.001632, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19825 s, avg_samples: 56.0, ips: 282.47367 samples/s, eta: 1 day, 0:30:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:42:10] ppocr INFO: epoch: [333/500], global_step: 90100, lr: 0.000100, edit distance: 0.001456, exp_rate: 0.932080, exp_rate<=1 : 0.973651, exp_rate<=2 : 0.993998, exp_rate<=3 : 0.997896, loss: 0.001814, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18660 s, avg_samples: 56.0, ips: 300.10377 samples/s, eta: 1 day, 0:30:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:42:29] ppocr INFO: epoch: [333/500], global_step: 90200, lr: 0.000100, edit distance: 0.001467, exp_rate: 0.931822, exp_rate<=1 : 0.973519, exp_rate<=2 : 0.993964, exp_rate<=3 : 0.997887, loss: 0.001799, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18863 s, avg_samples: 56.0, ips: 296.86978 samples/s, eta: 1 day, 0:29:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:42:48] ppocr INFO: epoch: [333/500], global_step: 90300, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.931646, exp_rate<=1 : 0.973434, exp_rate<=2 : 0.993944, exp_rate<=3 : 0.997863, loss: 0.001580, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19539 s, avg_samples: 56.0, ips: 286.60236 samples/s, eta: 1 day, 0:29:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:43:06] ppocr INFO: epoch: [333/500], global_step: 90400, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.931494, exp_rate<=1 : 0.973284, exp_rate<=2 : 0.993896, exp_rate<=3 : 0.997870, loss: 0.001763, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18030 s, avg_samples: 56.0, ips: 310.59390 samples/s, eta: 1 day, 0:29:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:43:10] ppocr INFO: epoch: [333/500], global_step: 90420, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.931468, exp_rate<=1 : 0.973268, exp_rate<=2 : 0.993895, exp_rate<=3 : 0.997861, loss: 0.001436, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.03766 s, avg_samples: 11.2, ips: 297.40647 samples/s, eta: 1 day, 0:29:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:43:12] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 07:43:29] ppocr INFO: epoch: [334/500], global_step: 90500, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.930872, exp_rate<=1 : 0.974345, exp_rate<=2 : 0.992431, exp_rate<=3 : 0.997355, loss: 0.002131, avg_reader_cost: 0.02780 s, avg_batch_cost: 0.19245 s, avg_samples: 44.8, ips: 232.78840 samples/s, eta: 1 day, 0:29:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:43:48] ppocr INFO: epoch: [334/500], global_step: 90600, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.939183, exp_rate<=1 : 0.978305, exp_rate<=2 : 0.993831, exp_rate<=3 : 0.998033, loss: 0.002015, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19076 s, avg_samples: 56.0, ips: 293.55988 samples/s, eta: 1 day, 0:28:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:44:08] ppocr INFO: epoch: [334/500], global_step: 90700, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.933184, exp_rate<=1 : 0.975178, exp_rate<=2 : 0.993099, exp_rate<=3 : 0.997909, loss: 0.001710, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19321 s, avg_samples: 56.0, ips: 289.83537 samples/s, eta: 1 day, 0:28:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:44:27] ppocr INFO: epoch: [334/500], global_step: 90800, lr: 0.000100, edit distance: 0.001620, exp_rate: 0.927315, exp_rate<=1 : 0.971003, exp_rate<=2 : 0.989875, exp_rate<=3 : 0.995015, loss: 0.001688, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18995 s, avg_samples: 56.0, ips: 294.82102 samples/s, eta: 1 day, 0:28:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:44:45] ppocr INFO: epoch: [334/500], global_step: 90900, lr: 0.000100, edit distance: 0.001617, exp_rate: 0.928725, exp_rate<=1 : 0.971942, exp_rate<=2 : 0.990977, exp_rate<=3 : 0.995725, loss: 0.001922, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18233 s, avg_samples: 56.0, ips: 307.13093 samples/s, eta: 1 day, 0:27:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:45:04] ppocr INFO: epoch: [334/500], global_step: 91000, lr: 0.000100, edit distance: 0.001591, exp_rate: 0.928290, exp_rate<=1 : 0.971532, exp_rate<=2 : 0.991397, exp_rate<=3 : 0.995937, loss: 0.002085, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19441 s, avg_samples: 56.0, ips: 288.05381 samples/s, eta: 1 day, 0:27:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:45:25] ppocr INFO: epoch: [334/500], global_step: 91100, lr: 0.000100, edit distance: 0.001577, exp_rate: 0.927535, exp_rate<=1 : 0.970824, exp_rate<=2 : 0.991051, exp_rate<=3 : 0.995872, loss: 0.001340, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20530 s, avg_samples: 56.0, ips: 272.77814 samples/s, eta: 1 day, 0:27:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:45:46] ppocr INFO: epoch: [334/500], global_step: 91200, lr: 0.000100, edit distance: 0.001543, exp_rate: 0.927586, exp_rate<=1 : 0.971272, exp_rate<=2 : 0.991480, exp_rate<=3 : 0.996046, loss: 0.001865, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20929 s, avg_samples: 56.0, ips: 267.56640 samples/s, eta: 1 day, 0:27:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:46:06] ppocr INFO: epoch: [334/500], global_step: 91300, lr: 0.000100, edit distance: 0.001519, exp_rate: 0.928043, exp_rate<=1 : 0.971596, exp_rate<=2 : 0.991729, exp_rate<=3 : 0.996222, loss: 0.001910, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19810 s, avg_samples: 56.0, ips: 282.68508 samples/s, eta: 1 day, 0:26:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:46:26] ppocr INFO: epoch: [334/500], global_step: 91400, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.927891, exp_rate<=1 : 0.971291, exp_rate<=2 : 0.991624, exp_rate<=3 : 0.996196, loss: 0.001708, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20475 s, avg_samples: 56.0, ips: 273.50860 samples/s, eta: 1 day, 0:26:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:46:45] ppocr INFO: epoch: [334/500], global_step: 91500, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.929190, exp_rate<=1 : 0.971840, exp_rate<=2 : 0.991969, exp_rate<=3 : 0.996377, loss: 0.001236, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18461 s, avg_samples: 56.0, ips: 303.33709 samples/s, eta: 1 day, 0:26:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:47:05] ppocr INFO: epoch: [334/500], global_step: 91600, lr: 0.000100, edit distance: 0.001485, exp_rate: 0.928886, exp_rate<=1 : 0.971658, exp_rate<=2 : 0.991913, exp_rate<=3 : 0.996461, loss: 0.002232, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20500 s, avg_samples: 56.0, ips: 273.16640 samples/s, eta: 1 day, 0:25:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:47:25] ppocr INFO: epoch: [334/500], global_step: 91700, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.928972, exp_rate<=1 : 0.971620, exp_rate<=2 : 0.992004, exp_rate<=3 : 0.996574, loss: 0.001539, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19517 s, avg_samples: 56.0, ips: 286.92844 samples/s, eta: 1 day, 0:25:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:47:44] ppocr INFO: epoch: [334/500], global_step: 91800, lr: 0.000100, edit distance: 0.001489, exp_rate: 0.929405, exp_rate<=1 : 0.971768, exp_rate<=2 : 0.992205, exp_rate<=3 : 0.996737, loss: 0.002011, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19294 s, avg_samples: 56.0, ips: 290.24310 samples/s, eta: 1 day, 0:25:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:48:04] ppocr INFO: epoch: [334/500], global_step: 91900, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.930132, exp_rate<=1 : 0.972137, exp_rate<=2 : 0.992451, exp_rate<=3 : 0.996867, loss: 0.001194, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19806 s, avg_samples: 56.0, ips: 282.74146 samples/s, eta: 1 day, 0:25:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:48:22] ppocr INFO: epoch: [334/500], global_step: 92000, lr: 0.000100, edit distance: 0.001467, exp_rate: 0.930487, exp_rate<=1 : 0.972467, exp_rate<=2 : 0.992697, exp_rate<=3 : 0.996987, loss: 0.001720, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18575 s, avg_samples: 56.0, ips: 301.47964 samples/s, eta: 1 day, 0:24:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:48:43] ppocr INFO: epoch: [334/500], global_step: 92100, lr: 0.000100, edit distance: 0.001474, exp_rate: 0.930682, exp_rate<=1 : 0.972624, exp_rate<=2 : 0.992763, exp_rate<=3 : 0.997028, loss: 0.001816, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20166 s, avg_samples: 56.0, ips: 277.69183 samples/s, eta: 1 day, 0:24:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:49:01] ppocr INFO: epoch: [334/500], global_step: 92200, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.931014, exp_rate<=1 : 0.972824, exp_rate<=2 : 0.992878, exp_rate<=3 : 0.997089, loss: 0.002478, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18330 s, avg_samples: 56.0, ips: 305.51132 samples/s, eta: 1 day, 0:24:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:49:21] ppocr INFO: epoch: [334/500], global_step: 92300, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.930863, exp_rate<=1 : 0.972856, exp_rate<=2 : 0.992951, exp_rate<=3 : 0.997081, loss: 0.001659, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19544 s, avg_samples: 56.0, ips: 286.52949 samples/s, eta: 1 day, 0:23:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:49:41] ppocr INFO: epoch: [334/500], global_step: 92400, lr: 0.000100, edit distance: 0.001498, exp_rate: 0.930695, exp_rate<=1 : 0.972681, exp_rate<=2 : 0.992949, exp_rate<=3 : 0.997077, loss: 0.002056, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19892 s, avg_samples: 56.0, ips: 281.52401 samples/s, eta: 1 day, 0:23:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:49:58] ppocr INFO: epoch: [334/500], global_step: 92500, lr: 0.000100, edit distance: 0.001499, exp_rate: 0.931010, exp_rate<=1 : 0.972770, exp_rate<=2 : 0.992988, exp_rate<=3 : 0.997154, loss: 0.001254, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17604 s, avg_samples: 56.0, ips: 318.10830 samples/s, eta: 1 day, 0:22:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:50:16] ppocr INFO: epoch: [334/500], global_step: 92600, lr: 0.000100, edit distance: 0.001509, exp_rate: 0.931085, exp_rate<=1 : 0.972858, exp_rate<=2 : 0.993081, exp_rate<=3 : 0.997186, loss: 0.001742, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18209 s, avg_samples: 56.0, ips: 307.53624 samples/s, eta: 1 day, 0:22:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:50:36] ppocr INFO: epoch: [334/500], global_step: 92700, lr: 0.000100, edit distance: 0.001525, exp_rate: 0.930711, exp_rate<=1 : 0.972755, exp_rate<=2 : 0.993043, exp_rate<=3 : 0.997197, loss: 0.001305, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19581 s, avg_samples: 56.0, ips: 285.98939 samples/s, eta: 1 day, 0:22:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:50:54] ppocr INFO: epoch: [334/500], global_step: 92800, lr: 0.000100, edit distance: 0.001521, exp_rate: 0.931015, exp_rate<=1 : 0.972997, exp_rate<=2 : 0.993124, exp_rate<=3 : 0.997274, loss: 0.001376, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18307 s, avg_samples: 56.0, ips: 305.89495 samples/s, eta: 1 day, 0:21:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:51:13] ppocr INFO: epoch: [334/500], global_step: 92900, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.930633, exp_rate<=1 : 0.972875, exp_rate<=2 : 0.993114, exp_rate<=3 : 0.997286, loss: 0.001988, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19064 s, avg_samples: 56.0, ips: 293.75089 samples/s, eta: 1 day, 0:21:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:51:30] ppocr INFO: epoch: [334/500], global_step: 93000, lr: 0.000100, edit distance: 0.001539, exp_rate: 0.930652, exp_rate<=1 : 0.972914, exp_rate<=2 : 0.993141, exp_rate<=3 : 0.997293, loss: 0.001695, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16859 s, avg_samples: 56.0, ips: 332.16133 samples/s, eta: 1 day, 0:21:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:51:48] ppocr INFO: epoch: [334/500], global_step: 93100, lr: 0.000100, edit distance: 0.001546, exp_rate: 0.930497, exp_rate<=1 : 0.972824, exp_rate<=2 : 0.993134, exp_rate<=3 : 0.997270, loss: 0.001744, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17876 s, avg_samples: 56.0, ips: 313.26676 samples/s, eta: 1 day, 0:20:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:52:00] ppocr INFO: epoch: [334/500], global_step: 93160, lr: 0.000100, edit distance: 0.001552, exp_rate: 0.930191, exp_rate<=1 : 0.972690, exp_rate<=2 : 0.993118, exp_rate<=3 : 0.997291, loss: 0.002394, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.11816 s, avg_samples: 33.6, ips: 284.36349 samples/s, eta: 1 day, 0:20:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:52:02] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 07:52:10] ppocr INFO: epoch: [335/500], global_step: 93200, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.943461, exp_rate<=1 : 0.976492, exp_rate<=2 : 0.994739, exp_rate<=3 : 0.997739, loss: 0.001317, avg_reader_cost: 0.02577 s, avg_batch_cost: 0.09828 s, avg_samples: 22.4, ips: 227.93169 samples/s, eta: 1 day, 0:20:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:52:29] ppocr INFO: epoch: [335/500], global_step: 93300, lr: 0.000100, edit distance: 0.001367, exp_rate: 0.933368, exp_rate<=1 : 0.973558, exp_rate<=2 : 0.991653, exp_rate<=3 : 0.996039, loss: 0.001775, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19486 s, avg_samples: 56.0, ips: 287.38857 samples/s, eta: 1 day, 0:20:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:52:49] ppocr INFO: epoch: [335/500], global_step: 93400, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.934109, exp_rate<=1 : 0.972226, exp_rate<=2 : 0.993070, exp_rate<=3 : 0.996540, loss: 0.002038, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19903 s, avg_samples: 56.0, ips: 281.36190 samples/s, eta: 1 day, 0:19:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:53:08] ppocr INFO: epoch: [335/500], global_step: 93500, lr: 0.000100, edit distance: 0.001461, exp_rate: 0.934470, exp_rate<=1 : 0.972985, exp_rate<=2 : 0.993084, exp_rate<=3 : 0.996758, loss: 0.001275, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18837 s, avg_samples: 56.0, ips: 297.29331 samples/s, eta: 1 day, 0:19:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:53:27] ppocr INFO: epoch: [335/500], global_step: 93600, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.936522, exp_rate<=1 : 0.974342, exp_rate<=2 : 0.993533, exp_rate<=3 : 0.997151, loss: 0.001747, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18501 s, avg_samples: 56.0, ips: 302.68256 samples/s, eta: 1 day, 0:19:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:53:46] ppocr INFO: epoch: [335/500], global_step: 93700, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.937678, exp_rate<=1 : 0.974324, exp_rate<=2 : 0.993747, exp_rate<=3 : 0.997273, loss: 0.001709, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19324 s, avg_samples: 56.0, ips: 289.79298 samples/s, eta: 1 day, 0:18:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:54:06] ppocr INFO: epoch: [335/500], global_step: 93800, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.935514, exp_rate<=1 : 0.973469, exp_rate<=2 : 0.993405, exp_rate<=3 : 0.997281, loss: 0.001673, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20257 s, avg_samples: 56.0, ips: 276.44587 samples/s, eta: 1 day, 0:18:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:54:25] ppocr INFO: epoch: [335/500], global_step: 93900, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.935609, exp_rate<=1 : 0.973747, exp_rate<=2 : 0.993491, exp_rate<=3 : 0.997366, loss: 0.001653, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18671 s, avg_samples: 56.0, ips: 299.92385 samples/s, eta: 1 day, 0:18:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:54:45] ppocr INFO: epoch: [335/500], global_step: 94000, lr: 0.000100, edit distance: 0.001453, exp_rate: 0.934024, exp_rate<=1 : 0.972734, exp_rate<=2 : 0.993303, exp_rate<=3 : 0.997295, loss: 0.001435, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19763 s, avg_samples: 56.0, ips: 283.35260 samples/s, eta: 1 day, 0:17:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:55:03] ppocr INFO: epoch: [335/500], global_step: 94100, lr: 0.000100, edit distance: 0.001448, exp_rate: 0.933709, exp_rate<=1 : 0.972656, exp_rate<=2 : 0.993170, exp_rate<=3 : 0.997147, loss: 0.002155, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18669 s, avg_samples: 56.0, ips: 299.96254 samples/s, eta: 1 day, 0:17:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:55:23] ppocr INFO: epoch: [335/500], global_step: 94200, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.933520, exp_rate<=1 : 0.972602, exp_rate<=2 : 0.993327, exp_rate<=3 : 0.997186, loss: 0.001732, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19608 s, avg_samples: 56.0, ips: 285.60060 samples/s, eta: 1 day, 0:17:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:55:42] ppocr INFO: epoch: [335/500], global_step: 94300, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.934012, exp_rate<=1 : 0.972717, exp_rate<=2 : 0.993350, exp_rate<=3 : 0.997299, loss: 0.001008, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18654 s, avg_samples: 56.0, ips: 300.20127 samples/s, eta: 1 day, 0:16:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:56:02] ppocr INFO: epoch: [335/500], global_step: 94400, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.934291, exp_rate<=1 : 0.972989, exp_rate<=2 : 0.993525, exp_rate<=3 : 0.997449, loss: 0.001059, avg_reader_cost: 0.00017 s, avg_batch_cost: 0.20321 s, avg_samples: 56.0, ips: 275.58315 samples/s, eta: 1 day, 0:16:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:56:20] ppocr INFO: epoch: [335/500], global_step: 94500, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.934531, exp_rate<=1 : 0.973187, exp_rate<=2 : 0.993424, exp_rate<=3 : 0.997437, loss: 0.001401, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18209 s, avg_samples: 56.0, ips: 307.54229 samples/s, eta: 1 day, 0:16:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:56:39] ppocr INFO: epoch: [335/500], global_step: 94600, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.934975, exp_rate<=1 : 0.973442, exp_rate<=2 : 0.993609, exp_rate<=3 : 0.997490, loss: 0.001224, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18816 s, avg_samples: 56.0, ips: 297.62689 samples/s, eta: 1 day, 0:15:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:56:58] ppocr INFO: epoch: [335/500], global_step: 94700, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.935799, exp_rate<=1 : 0.973799, exp_rate<=2 : 0.993806, exp_rate<=3 : 0.997585, loss: 0.001542, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19015 s, avg_samples: 56.0, ips: 294.50450 samples/s, eta: 1 day, 0:15:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:57:17] ppocr INFO: epoch: [335/500], global_step: 94800, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.935722, exp_rate<=1 : 0.973891, exp_rate<=2 : 0.993792, exp_rate<=3 : 0.997558, loss: 0.002145, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19129 s, avg_samples: 56.0, ips: 292.75685 samples/s, eta: 1 day, 0:15:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:57:36] ppocr INFO: epoch: [335/500], global_step: 94900, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.935794, exp_rate<=1 : 0.973934, exp_rate<=2 : 0.993848, exp_rate<=3 : 0.997645, loss: 0.002090, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18679 s, avg_samples: 56.0, ips: 299.80119 samples/s, eta: 1 day, 0:14:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:57:56] ppocr INFO: epoch: [335/500], global_step: 95000, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.935481, exp_rate<=1 : 0.973938, exp_rate<=2 : 0.993904, exp_rate<=3 : 0.997650, loss: 0.001723, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19681 s, avg_samples: 56.0, ips: 284.54359 samples/s, eta: 1 day, 0:14:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:58:15] ppocr INFO: epoch: [335/500], global_step: 95100, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.935355, exp_rate<=1 : 0.973906, exp_rate<=2 : 0.993788, exp_rate<=3 : 0.997616, loss: 0.001606, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19255 s, avg_samples: 56.0, ips: 290.83689 samples/s, eta: 1 day, 0:14:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:58:33] ppocr INFO: epoch: [335/500], global_step: 95200, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.935194, exp_rate<=1 : 0.973921, exp_rate<=2 : 0.993742, exp_rate<=3 : 0.997619, loss: 0.002727, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18382 s, avg_samples: 56.0, ips: 304.64922 samples/s, eta: 1 day, 0:13:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:58:52] ppocr INFO: epoch: [335/500], global_step: 95300, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.935045, exp_rate<=1 : 0.973844, exp_rate<=2 : 0.993777, exp_rate<=3 : 0.997682, loss: 0.001374, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18404 s, avg_samples: 56.0, ips: 304.27469 samples/s, eta: 1 day, 0:13:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:59:11] ppocr INFO: epoch: [335/500], global_step: 95400, lr: 0.000100, edit distance: 0.001413, exp_rate: 0.934651, exp_rate<=1 : 0.973841, exp_rate<=2 : 0.993815, exp_rate<=3 : 0.997701, loss: 0.001568, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19512 s, avg_samples: 56.0, ips: 286.99604 samples/s, eta: 1 day, 0:13:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:59:32] ppocr INFO: epoch: [335/500], global_step: 95500, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.934368, exp_rate<=1 : 0.973725, exp_rate<=2 : 0.993721, exp_rate<=3 : 0.997662, loss: 0.001916, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20395 s, avg_samples: 56.0, ips: 274.57616 samples/s, eta: 1 day, 0:12:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 07:59:52] ppocr INFO: epoch: [335/500], global_step: 95600, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.934077, exp_rate<=1 : 0.973637, exp_rate<=2 : 0.993623, exp_rate<=3 : 0.997627, loss: 0.001149, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19842 s, avg_samples: 56.0, ips: 282.22620 samples/s, eta: 1 day, 0:12:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:00:12] ppocr INFO: epoch: [335/500], global_step: 95700, lr: 0.000100, edit distance: 0.001407, exp_rate: 0.934087, exp_rate<=1 : 0.973744, exp_rate<=2 : 0.993612, exp_rate<=3 : 0.997628, loss: 0.001471, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19917 s, avg_samples: 56.0, ips: 281.17378 samples/s, eta: 1 day, 0:12:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:00:30] ppocr INFO: epoch: [335/500], global_step: 95800, lr: 0.000100, edit distance: 0.001413, exp_rate: 0.934379, exp_rate<=1 : 0.973791, exp_rate<=2 : 0.993620, exp_rate<=3 : 0.997651, loss: 0.001531, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18091 s, avg_samples: 56.0, ips: 309.53974 samples/s, eta: 1 day, 0:11:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:00:49] ppocr INFO: epoch: [335/500], global_step: 95900, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.933876, exp_rate<=1 : 0.973575, exp_rate<=2 : 0.993631, exp_rate<=3 : 0.997681, loss: 0.001998, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19528 s, avg_samples: 56.0, ips: 286.76228 samples/s, eta: 1 day, 0:11:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:00:52] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 08:00:53] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_335 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 08:01:14] ppocr INFO: epoch: [336/500], global_step: 96000, lr: 0.000100, edit distance: 0.001496, exp_rate: 0.928471, exp_rate<=1 : 0.970140, exp_rate<=2 : 0.993556, exp_rate<=3 : 0.997206, loss: 0.001164, avg_reader_cost: 0.03811 s, avg_batch_cost: 0.24244 s, avg_samples: 56.0, ips: 230.98251 samples/s, eta: 1 day, 0:11:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:01:32] ppocr INFO: epoch: [336/500], global_step: 96100, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.936233, exp_rate<=1 : 0.974247, exp_rate<=2 : 0.994202, exp_rate<=3 : 0.997469, loss: 0.001141, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18840 s, avg_samples: 56.0, ips: 297.24673 samples/s, eta: 1 day, 0:11:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:01:52] ppocr INFO: epoch: [336/500], global_step: 96200, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.937422, exp_rate<=1 : 0.975296, exp_rate<=2 : 0.994141, exp_rate<=3 : 0.997453, loss: 0.000943, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19621 s, avg_samples: 56.0, ips: 285.40218 samples/s, eta: 1 day, 0:11:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:02:11] ppocr INFO: epoch: [336/500], global_step: 96300, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.937693, exp_rate<=1 : 0.974787, exp_rate<=2 : 0.994048, exp_rate<=3 : 0.997691, loss: 0.001732, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18505 s, avg_samples: 56.0, ips: 302.62170 samples/s, eta: 1 day, 0:10:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:02:29] ppocr INFO: epoch: [336/500], global_step: 96400, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.936532, exp_rate<=1 : 0.974993, exp_rate<=2 : 0.994406, exp_rate<=3 : 0.997945, loss: 0.001977, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18797 s, avg_samples: 56.0, ips: 297.91787 samples/s, eta: 1 day, 0:10:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:02:48] ppocr INFO: epoch: [336/500], global_step: 96500, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.935732, exp_rate<=1 : 0.974488, exp_rate<=2 : 0.994193, exp_rate<=3 : 0.998007, loss: 0.001793, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18681 s, avg_samples: 56.0, ips: 299.76671 samples/s, eta: 1 day, 0:09:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:03:07] ppocr INFO: epoch: [336/500], global_step: 96600, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.935114, exp_rate<=1 : 0.974254, exp_rate<=2 : 0.994258, exp_rate<=3 : 0.997984, loss: 0.001985, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18688 s, avg_samples: 56.0, ips: 299.66414 samples/s, eta: 1 day, 0:09:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:03:27] ppocr INFO: epoch: [336/500], global_step: 96700, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.934630, exp_rate<=1 : 0.973920, exp_rate<=2 : 0.993696, exp_rate<=3 : 0.997681, loss: 0.001288, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20399 s, avg_samples: 56.0, ips: 274.51936 samples/s, eta: 1 day, 0:09:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:03:46] ppocr INFO: epoch: [336/500], global_step: 96800, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.933978, exp_rate<=1 : 0.973778, exp_rate<=2 : 0.993562, exp_rate<=3 : 0.997596, loss: 0.001678, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18660 s, avg_samples: 56.0, ips: 300.10091 samples/s, eta: 1 day, 0:09:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:04:05] ppocr INFO: epoch: [336/500], global_step: 96900, lr: 0.000100, edit distance: 0.001479, exp_rate: 0.932326, exp_rate<=1 : 0.972998, exp_rate<=2 : 0.993259, exp_rate<=3 : 0.997494, loss: 0.001769, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18748 s, avg_samples: 56.0, ips: 298.69207 samples/s, eta: 1 day, 0:08:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:04:24] ppocr INFO: epoch: [336/500], global_step: 97000, lr: 0.000100, edit distance: 0.001517, exp_rate: 0.930598, exp_rate<=1 : 0.972287, exp_rate<=2 : 0.993122, exp_rate<=3 : 0.997534, loss: 0.001909, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19421 s, avg_samples: 56.0, ips: 288.34124 samples/s, eta: 1 day, 0:08:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:04:43] ppocr INFO: epoch: [336/500], global_step: 97100, lr: 0.000100, edit distance: 0.001540, exp_rate: 0.930772, exp_rate<=1 : 0.972272, exp_rate<=2 : 0.993232, exp_rate<=3 : 0.997620, loss: 0.001765, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18652 s, avg_samples: 56.0, ips: 300.23123 samples/s, eta: 1 day, 0:07:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:05:02] ppocr INFO: epoch: [336/500], global_step: 97200, lr: 0.000100, edit distance: 0.001593, exp_rate: 0.929140, exp_rate<=1 : 0.971722, exp_rate<=2 : 0.993059, exp_rate<=3 : 0.997546, loss: 0.002190, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18892 s, avg_samples: 56.0, ips: 296.41825 samples/s, eta: 1 day, 0:07:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:05:22] ppocr INFO: epoch: [336/500], global_step: 97300, lr: 0.000100, edit distance: 0.001602, exp_rate: 0.928436, exp_rate<=1 : 0.971455, exp_rate<=2 : 0.992941, exp_rate<=3 : 0.997437, loss: 0.001578, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20052 s, avg_samples: 56.0, ips: 279.27234 samples/s, eta: 1 day, 0:07:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:05:42] ppocr INFO: epoch: [336/500], global_step: 97400, lr: 0.000100, edit distance: 0.001615, exp_rate: 0.928078, exp_rate<=1 : 0.971234, exp_rate<=2 : 0.992964, exp_rate<=3 : 0.997402, loss: 0.001776, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19821 s, avg_samples: 56.0, ips: 282.53301 samples/s, eta: 1 day, 0:07:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:06:01] ppocr INFO: epoch: [336/500], global_step: 97500, lr: 0.000100, edit distance: 0.001604, exp_rate: 0.927867, exp_rate<=1 : 0.971199, exp_rate<=2 : 0.992958, exp_rate<=3 : 0.997419, loss: 0.001892, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19675 s, avg_samples: 56.0, ips: 284.63079 samples/s, eta: 1 day, 0:06:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:06:22] ppocr INFO: epoch: [336/500], global_step: 97600, lr: 0.000100, edit distance: 0.001610, exp_rate: 0.926974, exp_rate<=1 : 0.970819, exp_rate<=2 : 0.992874, exp_rate<=3 : 0.997403, loss: 0.001772, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20536 s, avg_samples: 56.0, ips: 272.69838 samples/s, eta: 1 day, 0:06:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:06:41] ppocr INFO: epoch: [336/500], global_step: 97700, lr: 0.000100, edit distance: 0.001594, exp_rate: 0.927534, exp_rate<=1 : 0.971105, exp_rate<=2 : 0.992970, exp_rate<=3 : 0.997482, loss: 0.002216, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18729 s, avg_samples: 56.0, ips: 298.99925 samples/s, eta: 1 day, 0:06:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:07:01] ppocr INFO: epoch: [336/500], global_step: 97800, lr: 0.000100, edit distance: 0.001588, exp_rate: 0.927550, exp_rate<=1 : 0.971140, exp_rate<=2 : 0.993011, exp_rate<=3 : 0.997542, loss: 0.001388, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20456 s, avg_samples: 56.0, ips: 273.76009 samples/s, eta: 1 day, 0:05:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:07:19] ppocr INFO: epoch: [336/500], global_step: 97900, lr: 0.000100, edit distance: 0.001586, exp_rate: 0.927976, exp_rate<=1 : 0.971422, exp_rate<=2 : 0.993061, exp_rate<=3 : 0.997560, loss: 0.002265, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18330 s, avg_samples: 56.0, ips: 305.50741 samples/s, eta: 1 day, 0:05:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:07:38] ppocr INFO: epoch: [336/500], global_step: 98000, lr: 0.000100, edit distance: 0.001596, exp_rate: 0.927990, exp_rate<=1 : 0.971403, exp_rate<=2 : 0.993055, exp_rate<=3 : 0.997574, loss: 0.001040, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18948 s, avg_samples: 56.0, ips: 295.54977 samples/s, eta: 1 day, 0:05:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:07:57] ppocr INFO: epoch: [336/500], global_step: 98100, lr: 0.000100, edit distance: 0.001609, exp_rate: 0.927946, exp_rate<=1 : 0.971514, exp_rate<=2 : 0.993051, exp_rate<=3 : 0.997551, loss: 0.001733, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18922 s, avg_samples: 56.0, ips: 295.95682 samples/s, eta: 1 day, 0:04:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:08:17] ppocr INFO: epoch: [336/500], global_step: 98200, lr: 0.000100, edit distance: 0.001597, exp_rate: 0.928288, exp_rate<=1 : 0.971701, exp_rate<=2 : 0.993122, exp_rate<=3 : 0.997591, loss: 0.001820, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19277 s, avg_samples: 56.0, ips: 290.49450 samples/s, eta: 1 day, 0:04:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:08:36] ppocr INFO: epoch: [336/500], global_step: 98300, lr: 0.000100, edit distance: 0.001612, exp_rate: 0.927784, exp_rate<=1 : 0.971530, exp_rate<=2 : 0.993044, exp_rate<=3 : 0.997544, loss: 0.002282, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19548 s, avg_samples: 56.0, ips: 286.46929 samples/s, eta: 1 day, 0:04:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:08:55] ppocr INFO: epoch: [336/500], global_step: 98400, lr: 0.000100, edit distance: 0.001606, exp_rate: 0.927940, exp_rate<=1 : 0.971734, exp_rate<=2 : 0.993143, exp_rate<=3 : 0.997589, loss: 0.001933, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18424 s, avg_samples: 56.0, ips: 303.94874 samples/s, eta: 1 day, 0:03:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:09:13] ppocr INFO: epoch: [336/500], global_step: 98500, lr: 0.000100, edit distance: 0.001596, exp_rate: 0.928285, exp_rate<=1 : 0.971798, exp_rate<=2 : 0.993188, exp_rate<=3 : 0.997606, loss: 0.001653, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18347 s, avg_samples: 56.0, ips: 305.22746 samples/s, eta: 1 day, 0:03:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:09:31] ppocr INFO: epoch: [336/500], global_step: 98600, lr: 0.000100, edit distance: 0.001597, exp_rate: 0.928545, exp_rate<=1 : 0.971875, exp_rate<=2 : 0.993206, exp_rate<=3 : 0.997624, loss: 0.001739, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18326 s, avg_samples: 56.0, ips: 305.58265 samples/s, eta: 1 day, 0:03:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:09:39] ppocr INFO: epoch: [336/500], global_step: 98640, lr: 0.000100, edit distance: 0.001607, exp_rate: 0.928431, exp_rate<=1 : 0.971832, exp_rate<=2 : 0.993200, exp_rate<=3 : 0.997639, loss: 0.001851, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07738 s, avg_samples: 22.4, ips: 289.48653 samples/s, eta: 1 day, 0:02:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:09:41] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 08:09:54] ppocr INFO: epoch: [337/500], global_step: 98700, lr: 0.000100, edit distance: 0.001071, exp_rate: 0.945570, exp_rate<=1 : 0.977316, exp_rate<=2 : 0.993104, exp_rate<=3 : 0.997299, loss: 0.001246, avg_reader_cost: 0.02677 s, avg_batch_cost: 0.14857 s, avg_samples: 33.6, ips: 226.14924 samples/s, eta: 1 day, 0:03:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:10:14] ppocr INFO: epoch: [337/500], global_step: 98800, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.944548, exp_rate<=1 : 0.978371, exp_rate<=2 : 0.994680, exp_rate<=3 : 0.998279, loss: 0.001520, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19683 s, avg_samples: 56.0, ips: 284.51043 samples/s, eta: 1 day, 0:02:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:10:33] ppocr INFO: epoch: [337/500], global_step: 98900, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.944665, exp_rate<=1 : 0.978571, exp_rate<=2 : 0.994582, exp_rate<=3 : 0.998289, loss: 0.000958, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19425 s, avg_samples: 56.0, ips: 288.28375 samples/s, eta: 1 day, 0:02:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:10:53] ppocr INFO: epoch: [337/500], global_step: 99000, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.943071, exp_rate<=1 : 0.978067, exp_rate<=2 : 0.994618, exp_rate<=3 : 0.998370, loss: 0.001409, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20090 s, avg_samples: 56.0, ips: 278.74096 samples/s, eta: 1 day, 0:02:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:11:12] ppocr INFO: epoch: [337/500], global_step: 99100, lr: 0.000100, edit distance: 0.001132, exp_rate: 0.944552, exp_rate<=1 : 0.978388, exp_rate<=2 : 0.994728, exp_rate<=3 : 0.998530, loss: 0.001198, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18635 s, avg_samples: 56.0, ips: 300.51050 samples/s, eta: 1 day, 0:01:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:11:32] ppocr INFO: epoch: [337/500], global_step: 99200, lr: 0.000100, edit distance: 0.001157, exp_rate: 0.944309, exp_rate<=1 : 0.978513, exp_rate<=2 : 0.994792, exp_rate<=3 : 0.998559, loss: 0.001945, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20304 s, avg_samples: 56.0, ips: 275.81256 samples/s, eta: 1 day, 0:01:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:11:52] ppocr INFO: epoch: [337/500], global_step: 99300, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.942311, exp_rate<=1 : 0.977957, exp_rate<=2 : 0.994647, exp_rate<=3 : 0.998467, loss: 0.000934, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19675 s, avg_samples: 56.0, ips: 284.61919 samples/s, eta: 1 day, 0:01:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:12:11] ppocr INFO: epoch: [337/500], global_step: 99400, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.943773, exp_rate<=1 : 0.978746, exp_rate<=2 : 0.994835, exp_rate<=3 : 0.998505, loss: 0.001847, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18607 s, avg_samples: 56.0, ips: 300.95541 samples/s, eta: 1 day, 0:00:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:12:30] ppocr INFO: epoch: [337/500], global_step: 99500, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.943059, exp_rate<=1 : 0.978565, exp_rate<=2 : 0.994973, exp_rate<=3 : 0.998480, loss: 0.001608, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19896 s, avg_samples: 56.0, ips: 281.46854 samples/s, eta: 1 day, 0:00:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:12:49] ppocr INFO: epoch: [337/500], global_step: 99600, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.941891, exp_rate<=1 : 0.978110, exp_rate<=2 : 0.994894, exp_rate<=3 : 0.998372, loss: 0.001989, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18687 s, avg_samples: 56.0, ips: 299.67117 samples/s, eta: 1 day, 0:00:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:13:07] ppocr INFO: epoch: [337/500], global_step: 99700, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.940966, exp_rate<=1 : 0.977776, exp_rate<=2 : 0.994863, exp_rate<=3 : 0.998332, loss: 0.001790, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18164 s, avg_samples: 56.0, ips: 308.29477 samples/s, eta: 23:59:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:13:26] ppocr INFO: epoch: [337/500], global_step: 99800, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.940137, exp_rate<=1 : 0.977332, exp_rate<=2 : 0.994849, exp_rate<=3 : 0.998312, loss: 0.002481, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19089 s, avg_samples: 56.0, ips: 293.35868 samples/s, eta: 23:59:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:13:46] ppocr INFO: epoch: [337/500], global_step: 99900, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.939238, exp_rate<=1 : 0.976804, exp_rate<=2 : 0.994785, exp_rate<=3 : 0.998266, loss: 0.001690, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19543 s, avg_samples: 56.0, ips: 286.55007 samples/s, eta: 23:59:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:14:08] ppocr INFO: epoch: [337/500], global_step: 100000, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.937500, exp_rate<=1 : 0.975999, exp_rate<=2 : 0.994557, exp_rate<=3 : 0.998210, loss: 0.001625, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21600 s, avg_samples: 56.0, ips: 259.25852 samples/s, eta: 23:59:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:14:27] ppocr INFO: epoch: [337/500], global_step: 100100, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.937150, exp_rate<=1 : 0.975924, exp_rate<=2 : 0.994530, exp_rate<=3 : 0.998215, loss: 0.001829, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19750 s, avg_samples: 56.0, ips: 283.54668 samples/s, eta: 23:58:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:14:47] ppocr INFO: epoch: [337/500], global_step: 100200, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.936167, exp_rate<=1 : 0.975647, exp_rate<=2 : 0.994423, exp_rate<=3 : 0.998104, loss: 0.001976, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19741 s, avg_samples: 56.0, ips: 283.67775 samples/s, eta: 23:58:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:15:05] ppocr INFO: epoch: [337/500], global_step: 100300, lr: 0.000100, edit distance: 0.001393, exp_rate: 0.935959, exp_rate<=1 : 0.975644, exp_rate<=2 : 0.994582, exp_rate<=3 : 0.998174, loss: 0.001951, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18224 s, avg_samples: 56.0, ips: 307.28990 samples/s, eta: 23:58:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:15:26] ppocr INFO: epoch: [337/500], global_step: 100400, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.934775, exp_rate<=1 : 0.975177, exp_rate<=2 : 0.994438, exp_rate<=3 : 0.998096, loss: 0.001604, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20077 s, avg_samples: 56.0, ips: 278.92205 samples/s, eta: 23:57:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:15:44] ppocr INFO: epoch: [337/500], global_step: 100500, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.934851, exp_rate<=1 : 0.975154, exp_rate<=2 : 0.994404, exp_rate<=3 : 0.998075, loss: 0.001530, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17972 s, avg_samples: 56.0, ips: 311.59169 samples/s, eta: 23:57:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:16:02] ppocr INFO: epoch: [337/500], global_step: 100600, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.934950, exp_rate<=1 : 0.975134, exp_rate<=2 : 0.994359, exp_rate<=3 : 0.998075, loss: 0.002222, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18721 s, avg_samples: 56.0, ips: 299.12365 samples/s, eta: 23:57:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:16:21] ppocr INFO: epoch: [337/500], global_step: 100700, lr: 0.000100, edit distance: 0.001450, exp_rate: 0.934682, exp_rate<=1 : 0.974875, exp_rate<=2 : 0.994310, exp_rate<=3 : 0.998075, loss: 0.003357, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18495 s, avg_samples: 56.0, ips: 302.77907 samples/s, eta: 23:56:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:16:40] ppocr INFO: epoch: [337/500], global_step: 100800, lr: 0.000100, edit distance: 0.001464, exp_rate: 0.934116, exp_rate<=1 : 0.974704, exp_rate<=2 : 0.994178, exp_rate<=3 : 0.998033, loss: 0.001425, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19177 s, avg_samples: 56.0, ips: 292.01106 samples/s, eta: 23:56:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:16:59] ppocr INFO: epoch: [337/500], global_step: 100900, lr: 0.000100, edit distance: 0.001460, exp_rate: 0.933995, exp_rate<=1 : 0.974607, exp_rate<=2 : 0.994140, exp_rate<=3 : 0.997992, loss: 0.002962, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19391 s, avg_samples: 56.0, ips: 288.79960 samples/s, eta: 23:56:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:17:18] ppocr INFO: epoch: [337/500], global_step: 101000, lr: 0.000100, edit distance: 0.001462, exp_rate: 0.934001, exp_rate<=1 : 0.974653, exp_rate<=2 : 0.994156, exp_rate<=3 : 0.997979, loss: 0.001336, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18437 s, avg_samples: 56.0, ips: 303.74018 samples/s, eta: 23:55:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:17:36] ppocr INFO: epoch: [337/500], global_step: 101100, lr: 0.000100, edit distance: 0.001466, exp_rate: 0.933860, exp_rate<=1 : 0.974621, exp_rate<=2 : 0.994145, exp_rate<=3 : 0.997989, loss: 0.002026, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18011 s, avg_samples: 56.0, ips: 310.91248 samples/s, eta: 23:55:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:17:55] ppocr INFO: epoch: [337/500], global_step: 101200, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.933940, exp_rate<=1 : 0.974676, exp_rate<=2 : 0.994203, exp_rate<=3 : 0.998000, loss: 0.002075, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19339 s, avg_samples: 56.0, ips: 289.56323 samples/s, eta: 23:54:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:18:13] ppocr INFO: epoch: [337/500], global_step: 101300, lr: 0.000100, edit distance: 0.001480, exp_rate: 0.933613, exp_rate<=1 : 0.974641, exp_rate<=2 : 0.994190, exp_rate<=3 : 0.998013, loss: 0.001881, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18192 s, avg_samples: 56.0, ips: 307.83565 samples/s, eta: 23:54:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:18:29] ppocr INFO: epoch: [337/500], global_step: 101380, lr: 0.000100, edit distance: 0.001482, exp_rate: 0.933369, exp_rate<=1 : 0.974527, exp_rate<=2 : 0.994221, exp_rate<=3 : 0.998030, loss: 0.001892, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15339 s, avg_samples: 44.8, ips: 292.07145 samples/s, eta: 23:54:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:18:31] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 08:18:34] ppocr INFO: epoch: [338/500], global_step: 101400, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.964811, exp_rate<=1 : 0.989938, exp_rate<=2 : 0.996136, exp_rate<=3 : 1.000000, loss: 0.000768, avg_reader_cost: 0.02701 s, avg_batch_cost: 0.05592 s, avg_samples: 11.2, ips: 200.30046 samples/s, eta: 23:54:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:18:53] ppocr INFO: epoch: [338/500], global_step: 101500, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.945750, exp_rate<=1 : 0.980991, exp_rate<=2 : 0.995475, exp_rate<=3 : 0.998061, loss: 0.001863, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18698 s, avg_samples: 56.0, ips: 299.49603 samples/s, eta: 23:53:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:19:12] ppocr INFO: epoch: [338/500], global_step: 101600, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.943570, exp_rate<=1 : 0.979160, exp_rate<=2 : 0.994647, exp_rate<=3 : 0.997722, loss: 0.001797, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18911 s, avg_samples: 56.0, ips: 296.12284 samples/s, eta: 23:53:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:19:32] ppocr INFO: epoch: [338/500], global_step: 101700, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.936670, exp_rate<=1 : 0.975446, exp_rate<=2 : 0.993143, exp_rate<=3 : 0.997191, loss: 0.001152, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20050 s, avg_samples: 56.0, ips: 279.30863 samples/s, eta: 23:53:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:19:51] ppocr INFO: epoch: [338/500], global_step: 101800, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.937391, exp_rate<=1 : 0.976219, exp_rate<=2 : 0.993661, exp_rate<=3 : 0.997564, loss: 0.002010, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18477 s, avg_samples: 56.0, ips: 303.07231 samples/s, eta: 23:52:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:20:10] ppocr INFO: epoch: [338/500], global_step: 101900, lr: 0.000100, edit distance: 0.001354, exp_rate: 0.937717, exp_rate<=1 : 0.976102, exp_rate<=2 : 0.993494, exp_rate<=3 : 0.997555, loss: 0.001384, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19179 s, avg_samples: 56.0, ips: 291.98200 samples/s, eta: 23:52:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:20:28] ppocr INFO: epoch: [338/500], global_step: 102000, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.938202, exp_rate<=1 : 0.976230, exp_rate<=2 : 0.993628, exp_rate<=3 : 0.997660, loss: 0.001659, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18285 s, avg_samples: 56.0, ips: 306.25415 samples/s, eta: 23:52:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:20:48] ppocr INFO: epoch: [338/500], global_step: 102100, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.938181, exp_rate<=1 : 0.976052, exp_rate<=2 : 0.993909, exp_rate<=3 : 0.997788, loss: 0.001480, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19379 s, avg_samples: 56.0, ips: 288.97814 samples/s, eta: 23:51:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:21:07] ppocr INFO: epoch: [338/500], global_step: 102200, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.937789, exp_rate<=1 : 0.975366, exp_rate<=2 : 0.993785, exp_rate<=3 : 0.997819, loss: 0.001729, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19297 s, avg_samples: 56.0, ips: 290.20609 samples/s, eta: 23:51:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:21:25] ppocr INFO: epoch: [338/500], global_step: 102300, lr: 0.000100, edit distance: 0.001449, exp_rate: 0.938085, exp_rate<=1 : 0.975767, exp_rate<=2 : 0.993877, exp_rate<=3 : 0.997841, loss: 0.001559, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17725 s, avg_samples: 56.0, ips: 315.94470 samples/s, eta: 23:51:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:21:44] ppocr INFO: epoch: [338/500], global_step: 102400, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.937709, exp_rate<=1 : 0.975784, exp_rate<=2 : 0.993991, exp_rate<=3 : 0.997920, loss: 0.001124, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19294 s, avg_samples: 56.0, ips: 290.24389 samples/s, eta: 23:50:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:22:04] ppocr INFO: epoch: [338/500], global_step: 102500, lr: 0.000100, edit distance: 0.001471, exp_rate: 0.935873, exp_rate<=1 : 0.974947, exp_rate<=2 : 0.993631, exp_rate<=3 : 0.997780, loss: 0.001339, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19639 s, avg_samples: 56.0, ips: 285.14405 samples/s, eta: 23:50:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:22:24] ppocr INFO: epoch: [338/500], global_step: 102600, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.934787, exp_rate<=1 : 0.974572, exp_rate<=2 : 0.993562, exp_rate<=3 : 0.997686, loss: 0.001615, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20144 s, avg_samples: 56.0, ips: 277.99987 samples/s, eta: 23:50:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:22:42] ppocr INFO: epoch: [338/500], global_step: 102700, lr: 0.000100, edit distance: 0.001473, exp_rate: 0.935238, exp_rate<=1 : 0.974688, exp_rate<=2 : 0.993710, exp_rate<=3 : 0.997758, loss: 0.001191, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18449 s, avg_samples: 56.0, ips: 303.54073 samples/s, eta: 23:49:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:23:02] ppocr INFO: epoch: [338/500], global_step: 102800, lr: 0.000100, edit distance: 0.001483, exp_rate: 0.935208, exp_rate<=1 : 0.974571, exp_rate<=2 : 0.993673, exp_rate<=3 : 0.997713, loss: 0.002267, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19499 s, avg_samples: 56.0, ips: 287.18985 samples/s, eta: 23:49:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:23:21] ppocr INFO: epoch: [338/500], global_step: 102900, lr: 0.000100, edit distance: 0.001506, exp_rate: 0.934699, exp_rate<=1 : 0.974337, exp_rate<=2 : 0.993560, exp_rate<=3 : 0.997678, loss: 0.002187, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19502 s, avg_samples: 56.0, ips: 287.14464 samples/s, eta: 23:49:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:23:39] ppocr INFO: epoch: [338/500], global_step: 103000, lr: 0.000100, edit distance: 0.001506, exp_rate: 0.934653, exp_rate<=1 : 0.974376, exp_rate<=2 : 0.993580, exp_rate<=3 : 0.997652, loss: 0.001265, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18099 s, avg_samples: 56.0, ips: 309.40387 samples/s, eta: 23:48:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:23:58] ppocr INFO: epoch: [338/500], global_step: 103100, lr: 0.000100, edit distance: 0.001514, exp_rate: 0.934306, exp_rate<=1 : 0.974172, exp_rate<=2 : 0.993400, exp_rate<=3 : 0.997553, loss: 0.002125, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18921 s, avg_samples: 56.0, ips: 295.96438 samples/s, eta: 23:48:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:24:18] ppocr INFO: epoch: [338/500], global_step: 103200, lr: 0.000100, edit distance: 0.001511, exp_rate: 0.933864, exp_rate<=1 : 0.973890, exp_rate<=2 : 0.993350, exp_rate<=3 : 0.997601, loss: 0.001857, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19961 s, avg_samples: 56.0, ips: 280.54178 samples/s, eta: 23:48:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:24:38] ppocr INFO: epoch: [338/500], global_step: 103300, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.933739, exp_rate<=1 : 0.973733, exp_rate<=2 : 0.993342, exp_rate<=3 : 0.997559, loss: 0.001834, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19493 s, avg_samples: 56.0, ips: 287.27893 samples/s, eta: 23:48:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:24:57] ppocr INFO: epoch: [338/500], global_step: 103400, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.933331, exp_rate<=1 : 0.973591, exp_rate<=2 : 0.993324, exp_rate<=3 : 0.997594, loss: 0.001897, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18872 s, avg_samples: 56.0, ips: 296.73907 samples/s, eta: 23:47:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:25:16] ppocr INFO: epoch: [338/500], global_step: 103500, lr: 0.000100, edit distance: 0.001510, exp_rate: 0.933445, exp_rate<=1 : 0.973527, exp_rate<=2 : 0.993361, exp_rate<=3 : 0.997605, loss: 0.001546, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19188 s, avg_samples: 56.0, ips: 291.85433 samples/s, eta: 23:47:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:25:36] ppocr INFO: epoch: [338/500], global_step: 103600, lr: 0.000100, edit distance: 0.001513, exp_rate: 0.933256, exp_rate<=1 : 0.973509, exp_rate<=2 : 0.993323, exp_rate<=3 : 0.997536, loss: 0.001851, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20465 s, avg_samples: 56.0, ips: 273.63699 samples/s, eta: 23:47:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:25:55] ppocr INFO: epoch: [338/500], global_step: 103700, lr: 0.000100, edit distance: 0.001525, exp_rate: 0.933282, exp_rate<=1 : 0.973503, exp_rate<=2 : 0.993355, exp_rate<=3 : 0.997547, loss: 0.003288, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18868 s, avg_samples: 56.0, ips: 296.79508 samples/s, eta: 23:46:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:26:15] ppocr INFO: epoch: [338/500], global_step: 103800, lr: 0.000100, edit distance: 0.001559, exp_rate: 0.931953, exp_rate<=1 : 0.972908, exp_rate<=2 : 0.993138, exp_rate<=3 : 0.997428, loss: 0.001465, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19860 s, avg_samples: 56.0, ips: 281.97417 samples/s, eta: 23:46:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:26:34] ppocr INFO: epoch: [338/500], global_step: 103900, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.932181, exp_rate<=1 : 0.973001, exp_rate<=2 : 0.993172, exp_rate<=3 : 0.997444, loss: 0.002765, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18541 s, avg_samples: 56.0, ips: 302.02668 samples/s, eta: 23:46:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:26:54] ppocr INFO: epoch: [338/500], global_step: 104000, lr: 0.000100, edit distance: 0.001561, exp_rate: 0.931711, exp_rate<=1 : 0.972802, exp_rate<=2 : 0.993062, exp_rate<=3 : 0.997363, loss: 0.002090, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20052 s, avg_samples: 56.0, ips: 279.26957 samples/s, eta: 23:45:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:27:13] ppocr INFO: epoch: [338/500], global_step: 104100, lr: 0.000100, edit distance: 0.001557, exp_rate: 0.931786, exp_rate<=1 : 0.972857, exp_rate<=2 : 0.993134, exp_rate<=3 : 0.997389, loss: 0.001662, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18725 s, avg_samples: 56.0, ips: 299.06758 samples/s, eta: 23:45:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:27:16] ppocr INFO: epoch: [338/500], global_step: 104120, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.931741, exp_rate<=1 : 0.972824, exp_rate<=2 : 0.993150, exp_rate<=3 : 0.997404, loss: 0.002227, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03207 s, avg_samples: 11.2, ips: 349.21670 samples/s, eta: 23:45:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:27:18] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 08:27:36] ppocr INFO: epoch: [339/500], global_step: 104200, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.931853, exp_rate<=1 : 0.971850, exp_rate<=2 : 0.994395, exp_rate<=3 : 0.998480, loss: 0.001551, avg_reader_cost: 0.03114 s, avg_batch_cost: 0.19995 s, avg_samples: 44.8, ips: 224.05542 samples/s, eta: 23:45:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:27:56] ppocr INFO: epoch: [339/500], global_step: 104300, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.938891, exp_rate<=1 : 0.977018, exp_rate<=2 : 0.994876, exp_rate<=3 : 0.998848, loss: 0.001145, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19803 s, avg_samples: 56.0, ips: 282.78718 samples/s, eta: 23:45:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:28:15] ppocr INFO: epoch: [339/500], global_step: 104400, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.938626, exp_rate<=1 : 0.977012, exp_rate<=2 : 0.994726, exp_rate<=3 : 0.998548, loss: 0.001260, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19325 s, avg_samples: 56.0, ips: 289.77594 samples/s, eta: 23:44:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:28:34] ppocr INFO: epoch: [339/500], global_step: 104500, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.939419, exp_rate<=1 : 0.977397, exp_rate<=2 : 0.994874, exp_rate<=3 : 0.998339, loss: 0.000922, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19012 s, avg_samples: 56.0, ips: 294.55403 samples/s, eta: 23:44:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:28:55] ppocr INFO: epoch: [339/500], global_step: 104600, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.938151, exp_rate<=1 : 0.976512, exp_rate<=2 : 0.994765, exp_rate<=3 : 0.998373, loss: 0.001578, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20471 s, avg_samples: 56.0, ips: 273.56103 samples/s, eta: 23:44:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:29:13] ppocr INFO: epoch: [339/500], global_step: 104700, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.939744, exp_rate<=1 : 0.977499, exp_rate<=2 : 0.995160, exp_rate<=3 : 0.998387, loss: 0.000912, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18131 s, avg_samples: 56.0, ips: 308.85671 samples/s, eta: 23:43:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:29:32] ppocr INFO: epoch: [339/500], global_step: 104800, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.939107, exp_rate<=1 : 0.977635, exp_rate<=2 : 0.995107, exp_rate<=3 : 0.998274, loss: 0.001699, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19407 s, avg_samples: 56.0, ips: 288.54852 samples/s, eta: 23:43:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:29:51] ppocr INFO: epoch: [339/500], global_step: 104900, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.938299, exp_rate<=1 : 0.977383, exp_rate<=2 : 0.994941, exp_rate<=3 : 0.998156, loss: 0.001600, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19248 s, avg_samples: 56.0, ips: 290.94120 samples/s, eta: 23:43:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:30:10] ppocr INFO: epoch: [339/500], global_step: 105000, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.937125, exp_rate<=1 : 0.976891, exp_rate<=2 : 0.994819, exp_rate<=3 : 0.998121, loss: 0.001710, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19064 s, avg_samples: 56.0, ips: 293.74494 samples/s, eta: 23:42:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:30:29] ppocr INFO: epoch: [339/500], global_step: 105100, lr: 0.000100, edit distance: 0.001413, exp_rate: 0.937135, exp_rate<=1 : 0.976749, exp_rate<=2 : 0.994701, exp_rate<=3 : 0.998052, loss: 0.002403, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18797 s, avg_samples: 56.0, ips: 297.91593 samples/s, eta: 23:42:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:30:48] ppocr INFO: epoch: [339/500], global_step: 105200, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.936573, exp_rate<=1 : 0.976246, exp_rate<=2 : 0.994589, exp_rate<=3 : 0.998037, loss: 0.001210, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18996 s, avg_samples: 56.0, ips: 294.79231 samples/s, eta: 23:42:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:31:06] ppocr INFO: epoch: [339/500], global_step: 105300, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.936743, exp_rate<=1 : 0.976183, exp_rate<=2 : 0.994641, exp_rate<=3 : 0.998049, loss: 0.001900, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18068 s, avg_samples: 56.0, ips: 309.94439 samples/s, eta: 23:41:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:31:26] ppocr INFO: epoch: [339/500], global_step: 105400, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.935637, exp_rate<=1 : 0.975882, exp_rate<=2 : 0.994636, exp_rate<=3 : 0.998067, loss: 0.001904, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19882 s, avg_samples: 56.0, ips: 281.65647 samples/s, eta: 23:41:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:31:44] ppocr INFO: epoch: [339/500], global_step: 105500, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.935794, exp_rate<=1 : 0.975959, exp_rate<=2 : 0.994586, exp_rate<=3 : 0.998049, loss: 0.001707, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17736 s, avg_samples: 56.0, ips: 315.73968 samples/s, eta: 23:41:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:32:04] ppocr INFO: epoch: [339/500], global_step: 105600, lr: 0.000100, edit distance: 0.001470, exp_rate: 0.934572, exp_rate<=1 : 0.975263, exp_rate<=2 : 0.994308, exp_rate<=3 : 0.997943, loss: 0.001971, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19624 s, avg_samples: 56.0, ips: 285.36086 samples/s, eta: 23:40:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:32:22] ppocr INFO: epoch: [339/500], global_step: 105700, lr: 0.000100, edit distance: 0.001467, exp_rate: 0.934531, exp_rate<=1 : 0.975392, exp_rate<=2 : 0.994389, exp_rate<=3 : 0.997994, loss: 0.001492, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18577 s, avg_samples: 56.0, ips: 301.44324 samples/s, eta: 23:40:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:32:40] ppocr INFO: epoch: [339/500], global_step: 105800, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.934965, exp_rate<=1 : 0.975500, exp_rate<=2 : 0.994467, exp_rate<=3 : 0.998027, loss: 0.001316, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17221 s, avg_samples: 56.0, ips: 325.17702 samples/s, eta: 23:39:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:32:58] ppocr INFO: epoch: [339/500], global_step: 105900, lr: 0.000100, edit distance: 0.001484, exp_rate: 0.934670, exp_rate<=1 : 0.975327, exp_rate<=2 : 0.994516, exp_rate<=3 : 0.998059, loss: 0.001956, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18490 s, avg_samples: 56.0, ips: 302.86465 samples/s, eta: 23:39:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:33:18] ppocr INFO: epoch: [339/500], global_step: 106000, lr: 0.000100, edit distance: 0.001510, exp_rate: 0.933505, exp_rate<=1 : 0.974752, exp_rate<=2 : 0.994493, exp_rate<=3 : 0.998054, loss: 0.002090, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19915 s, avg_samples: 56.0, ips: 281.19685 samples/s, eta: 23:39:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:33:38] ppocr INFO: epoch: [339/500], global_step: 106100, lr: 0.000100, edit distance: 0.001512, exp_rate: 0.932642, exp_rate<=1 : 0.974298, exp_rate<=2 : 0.994346, exp_rate<=3 : 0.998005, loss: 0.001943, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19908 s, avg_samples: 56.0, ips: 281.29055 samples/s, eta: 23:39:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:33:56] ppocr INFO: epoch: [339/500], global_step: 106200, lr: 0.000100, edit distance: 0.001519, exp_rate: 0.932561, exp_rate<=1 : 0.974001, exp_rate<=2 : 0.994198, exp_rate<=3 : 0.997971, loss: 0.002020, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18420 s, avg_samples: 56.0, ips: 304.01856 samples/s, eta: 23:38:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:34:15] ppocr INFO: epoch: [339/500], global_step: 106300, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.932391, exp_rate<=1 : 0.973905, exp_rate<=2 : 0.994096, exp_rate<=3 : 0.997937, loss: 0.001562, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18827 s, avg_samples: 56.0, ips: 297.44350 samples/s, eta: 23:38:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:34:34] ppocr INFO: epoch: [339/500], global_step: 106400, lr: 0.000100, edit distance: 0.001523, exp_rate: 0.932170, exp_rate<=1 : 0.973790, exp_rate<=2 : 0.994136, exp_rate<=3 : 0.997935, loss: 0.002228, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18789 s, avg_samples: 56.0, ips: 298.04746 samples/s, eta: 23:37:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:34:54] ppocr INFO: epoch: [339/500], global_step: 106500, lr: 0.000100, edit distance: 0.001542, exp_rate: 0.931732, exp_rate<=1 : 0.973547, exp_rate<=2 : 0.994025, exp_rate<=3 : 0.997900, loss: 0.001804, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19630 s, avg_samples: 56.0, ips: 285.27136 samples/s, eta: 23:37:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:35:14] ppocr INFO: epoch: [339/500], global_step: 106600, lr: 0.000100, edit distance: 0.001544, exp_rate: 0.931354, exp_rate<=1 : 0.973522, exp_rate<=2 : 0.994028, exp_rate<=3 : 0.997901, loss: 0.002088, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20791 s, avg_samples: 56.0, ips: 269.34992 samples/s, eta: 23:37:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:35:34] ppocr INFO: epoch: [339/500], global_step: 106700, lr: 0.000100, edit distance: 0.001541, exp_rate: 0.930983, exp_rate<=1 : 0.973244, exp_rate<=2 : 0.993884, exp_rate<=3 : 0.997820, loss: 0.001266, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19766 s, avg_samples: 56.0, ips: 283.31617 samples/s, eta: 23:37:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:35:53] ppocr INFO: epoch: [339/500], global_step: 106800, lr: 0.000100, edit distance: 0.001535, exp_rate: 0.931199, exp_rate<=1 : 0.973366, exp_rate<=2 : 0.993928, exp_rate<=3 : 0.997848, loss: 0.001498, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18823 s, avg_samples: 56.0, ips: 297.51031 samples/s, eta: 23:36:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:36:05] ppocr INFO: epoch: [339/500], global_step: 106860, lr: 0.000100, edit distance: 0.001533, exp_rate: 0.931177, exp_rate<=1 : 0.973327, exp_rate<=2 : 0.993882, exp_rate<=3 : 0.997812, loss: 0.001263, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.11509 s, avg_samples: 33.6, ips: 291.93874 samples/s, eta: 23:36:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:36:07] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 08:36:14] ppocr INFO: epoch: [340/500], global_step: 106900, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.937379, exp_rate<=1 : 0.979463, exp_rate<=2 : 0.996598, exp_rate<=3 : 1.000000, loss: 0.001916, avg_reader_cost: 0.02520 s, avg_batch_cost: 0.09757 s, avg_samples: 22.4, ips: 229.58568 samples/s, eta: 23:36:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:36:34] ppocr INFO: epoch: [340/500], global_step: 107000, lr: 0.000100, edit distance: 0.001451, exp_rate: 0.934838, exp_rate<=1 : 0.976826, exp_rate<=2 : 0.994474, exp_rate<=3 : 0.998370, loss: 0.001831, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.20072 s, avg_samples: 56.0, ips: 278.99607 samples/s, eta: 23:36:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:36:54] ppocr INFO: epoch: [340/500], global_step: 107100, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.937054, exp_rate<=1 : 0.977018, exp_rate<=2 : 0.993969, exp_rate<=3 : 0.998141, loss: 0.001548, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19571 s, avg_samples: 56.0, ips: 286.13776 samples/s, eta: 23:36:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:37:13] ppocr INFO: epoch: [340/500], global_step: 107200, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.936815, exp_rate<=1 : 0.977128, exp_rate<=2 : 0.994483, exp_rate<=3 : 0.998090, loss: 0.001267, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18898 s, avg_samples: 56.0, ips: 296.32257 samples/s, eta: 23:35:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:37:32] ppocr INFO: epoch: [340/500], global_step: 107300, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.937122, exp_rate<=1 : 0.976910, exp_rate<=2 : 0.994563, exp_rate<=3 : 0.998091, loss: 0.001885, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18542 s, avg_samples: 56.0, ips: 302.02136 samples/s, eta: 23:35:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:37:51] ppocr INFO: epoch: [340/500], global_step: 107400, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.937220, exp_rate<=1 : 0.977005, exp_rate<=2 : 0.994608, exp_rate<=3 : 0.998128, loss: 0.001468, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19021 s, avg_samples: 56.0, ips: 294.41916 samples/s, eta: 23:34:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:38:09] ppocr INFO: epoch: [340/500], global_step: 107500, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.937880, exp_rate<=1 : 0.976705, exp_rate<=2 : 0.994664, exp_rate<=3 : 0.998145, loss: 0.001805, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17960 s, avg_samples: 56.0, ips: 311.81146 samples/s, eta: 23:34:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:38:28] ppocr INFO: epoch: [340/500], global_step: 107600, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.937921, exp_rate<=1 : 0.976509, exp_rate<=2 : 0.994700, exp_rate<=3 : 0.998054, loss: 0.001673, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19232 s, avg_samples: 56.0, ips: 291.18477 samples/s, eta: 23:34:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:38:47] ppocr INFO: epoch: [340/500], global_step: 107700, lr: 0.000100, edit distance: 0.001416, exp_rate: 0.937592, exp_rate<=1 : 0.976241, exp_rate<=2 : 0.994826, exp_rate<=3 : 0.998099, loss: 0.001284, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18665 s, avg_samples: 56.0, ips: 300.02609 samples/s, eta: 23:33:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:39:06] ppocr INFO: epoch: [340/500], global_step: 107800, lr: 0.000100, edit distance: 0.001481, exp_rate: 0.935451, exp_rate<=1 : 0.975250, exp_rate<=2 : 0.994347, exp_rate<=3 : 0.997958, loss: 0.002314, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19404 s, avg_samples: 56.0, ips: 288.59767 samples/s, eta: 23:33:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:39:25] ppocr INFO: epoch: [340/500], global_step: 107900, lr: 0.000100, edit distance: 0.001520, exp_rate: 0.934745, exp_rate<=1 : 0.975141, exp_rate<=2 : 0.994162, exp_rate<=3 : 0.997864, loss: 0.001379, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18656 s, avg_samples: 56.0, ips: 300.17302 samples/s, eta: 23:33:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:39:43] ppocr INFO: epoch: [340/500], global_step: 108000, lr: 0.000100, edit distance: 0.001528, exp_rate: 0.934352, exp_rate<=1 : 0.974839, exp_rate<=2 : 0.994058, exp_rate<=3 : 0.997840, loss: 0.002472, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18624 s, avg_samples: 56.0, ips: 300.68405 samples/s, eta: 23:32:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:40:03] ppocr INFO: epoch: [340/500], global_step: 108100, lr: 0.000100, edit distance: 0.001512, exp_rate: 0.934657, exp_rate<=1 : 0.974964, exp_rate<=2 : 0.994176, exp_rate<=3 : 0.997847, loss: 0.001637, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19939 s, avg_samples: 56.0, ips: 280.85209 samples/s, eta: 23:32:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:40:23] ppocr INFO: epoch: [340/500], global_step: 108200, lr: 0.000100, edit distance: 0.001497, exp_rate: 0.934356, exp_rate<=1 : 0.974870, exp_rate<=2 : 0.994184, exp_rate<=3 : 0.997883, loss: 0.001109, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19862 s, avg_samples: 56.0, ips: 281.94163 samples/s, eta: 23:32:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:40:42] ppocr INFO: epoch: [340/500], global_step: 108300, lr: 0.000100, edit distance: 0.001493, exp_rate: 0.934447, exp_rate<=1 : 0.975031, exp_rate<=2 : 0.994295, exp_rate<=3 : 0.997928, loss: 0.001844, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19355 s, avg_samples: 56.0, ips: 289.32943 samples/s, eta: 23:31:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:41:03] ppocr INFO: epoch: [340/500], global_step: 108400, lr: 0.000100, edit distance: 0.001483, exp_rate: 0.934001, exp_rate<=1 : 0.974877, exp_rate<=2 : 0.994243, exp_rate<=3 : 0.997911, loss: 0.001164, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20576 s, avg_samples: 56.0, ips: 272.15559 samples/s, eta: 23:31:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:41:24] ppocr INFO: epoch: [340/500], global_step: 108500, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.933944, exp_rate<=1 : 0.974694, exp_rate<=2 : 0.994183, exp_rate<=3 : 0.997896, loss: 0.001753, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20380 s, avg_samples: 56.0, ips: 274.77447 samples/s, eta: 23:31:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:41:42] ppocr INFO: epoch: [340/500], global_step: 108600, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.934469, exp_rate<=1 : 0.974691, exp_rate<=2 : 0.994163, exp_rate<=3 : 0.997874, loss: 0.001450, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18075 s, avg_samples: 56.0, ips: 309.81772 samples/s, eta: 23:31:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:42:02] ppocr INFO: epoch: [340/500], global_step: 108700, lr: 0.000100, edit distance: 0.001492, exp_rate: 0.933982, exp_rate<=1 : 0.974429, exp_rate<=2 : 0.994205, exp_rate<=3 : 0.997883, loss: 0.001416, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.19994 s, avg_samples: 56.0, ips: 280.08542 samples/s, eta: 23:30:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:42:22] ppocr INFO: epoch: [340/500], global_step: 108800, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.933555, exp_rate<=1 : 0.974124, exp_rate<=2 : 0.994102, exp_rate<=3 : 0.997846, loss: 0.001801, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19904 s, avg_samples: 56.0, ips: 281.35387 samples/s, eta: 23:30:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:42:41] ppocr INFO: epoch: [340/500], global_step: 108900, lr: 0.000100, edit distance: 0.001482, exp_rate: 0.933549, exp_rate<=1 : 0.974141, exp_rate<=2 : 0.994156, exp_rate<=3 : 0.997878, loss: 0.002350, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19092 s, avg_samples: 56.0, ips: 293.31033 samples/s, eta: 23:30:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:43:00] ppocr INFO: epoch: [340/500], global_step: 109000, lr: 0.000100, edit distance: 0.001477, exp_rate: 0.933483, exp_rate<=1 : 0.974147, exp_rate<=2 : 0.994150, exp_rate<=3 : 0.997896, loss: 0.001665, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18959 s, avg_samples: 56.0, ips: 295.37284 samples/s, eta: 23:29:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:43:19] ppocr INFO: epoch: [340/500], global_step: 109100, lr: 0.000100, edit distance: 0.001494, exp_rate: 0.932790, exp_rate<=1 : 0.973668, exp_rate<=2 : 0.994021, exp_rate<=3 : 0.997855, loss: 0.001959, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19123 s, avg_samples: 56.0, ips: 292.84105 samples/s, eta: 23:29:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:43:36] ppocr INFO: epoch: [340/500], global_step: 109200, lr: 0.000100, edit distance: 0.001488, exp_rate: 0.933076, exp_rate<=1 : 0.973887, exp_rate<=2 : 0.994101, exp_rate<=3 : 0.997885, loss: 0.001820, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17642 s, avg_samples: 56.0, ips: 317.43331 samples/s, eta: 23:29:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:43:56] ppocr INFO: epoch: [340/500], global_step: 109300, lr: 0.000100, edit distance: 0.001502, exp_rate: 0.932236, exp_rate<=1 : 0.973413, exp_rate<=2 : 0.993983, exp_rate<=3 : 0.997853, loss: 0.001959, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19246 s, avg_samples: 56.0, ips: 290.96872 samples/s, eta: 23:28:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:44:16] ppocr INFO: epoch: [340/500], global_step: 109400, lr: 0.000100, edit distance: 0.001508, exp_rate: 0.931829, exp_rate<=1 : 0.973234, exp_rate<=2 : 0.993954, exp_rate<=3 : 0.997852, loss: 0.002185, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20267 s, avg_samples: 56.0, ips: 276.30925 samples/s, eta: 23:28:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:44:36] ppocr INFO: epoch: [340/500], global_step: 109500, lr: 0.000100, edit distance: 0.001503, exp_rate: 0.931828, exp_rate<=1 : 0.973228, exp_rate<=2 : 0.993942, exp_rate<=3 : 0.997849, loss: 0.001706, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19811 s, avg_samples: 56.0, ips: 282.67561 samples/s, eta: 23:28:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:44:55] ppocr INFO: epoch: [340/500], global_step: 109600, lr: 0.000100, edit distance: 0.001503, exp_rate: 0.931593, exp_rate<=1 : 0.973041, exp_rate<=2 : 0.993838, exp_rate<=3 : 0.997842, loss: 0.000983, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19191 s, avg_samples: 56.0, ips: 291.79977 samples/s, eta: 23:27:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:44:57] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 08:44:58] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_340 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 08:45:19] ppocr INFO: epoch: [341/500], global_step: 109700, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.935912, exp_rate<=1 : 0.974026, exp_rate<=2 : 0.993541, exp_rate<=3 : 0.998943, loss: 0.001451, avg_reader_cost: 0.03401 s, avg_batch_cost: 0.23418 s, avg_samples: 56.0, ips: 239.12815 samples/s, eta: 23:27:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:45:37] ppocr INFO: epoch: [341/500], global_step: 109800, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.942950, exp_rate<=1 : 0.978930, exp_rate<=2 : 0.995284, exp_rate<=3 : 0.998819, loss: 0.001306, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17982 s, avg_samples: 56.0, ips: 311.41931 samples/s, eta: 23:27:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:45:56] ppocr INFO: epoch: [341/500], global_step: 109900, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.942216, exp_rate<=1 : 0.977596, exp_rate<=2 : 0.995048, exp_rate<=3 : 0.998629, loss: 0.001320, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19139 s, avg_samples: 56.0, ips: 292.60364 samples/s, eta: 23:27:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:46:15] ppocr INFO: epoch: [341/500], global_step: 110000, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.941218, exp_rate<=1 : 0.977189, exp_rate<=2 : 0.995042, exp_rate<=3 : 0.998609, loss: 0.001488, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19677 s, avg_samples: 56.0, ips: 284.59446 samples/s, eta: 23:26:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:46:34] ppocr INFO: epoch: [341/500], global_step: 110100, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.941852, exp_rate<=1 : 0.977494, exp_rate<=2 : 0.995337, exp_rate<=3 : 0.998762, loss: 0.001903, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18184 s, avg_samples: 56.0, ips: 307.96923 samples/s, eta: 23:26:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:46:52] ppocr INFO: epoch: [341/500], global_step: 110200, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.941578, exp_rate<=1 : 0.977551, exp_rate<=2 : 0.995329, exp_rate<=3 : 0.998700, loss: 0.001724, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18383 s, avg_samples: 56.0, ips: 304.62780 samples/s, eta: 23:26:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:47:12] ppocr INFO: epoch: [341/500], global_step: 110300, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.939882, exp_rate<=1 : 0.977301, exp_rate<=2 : 0.995204, exp_rate<=3 : 0.998629, loss: 0.001677, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19762 s, avg_samples: 56.0, ips: 283.36899 samples/s, eta: 23:25:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:47:32] ppocr INFO: epoch: [341/500], global_step: 110400, lr: 0.000100, edit distance: 0.001398, exp_rate: 0.938470, exp_rate<=1 : 0.976452, exp_rate<=2 : 0.994893, exp_rate<=3 : 0.998490, loss: 0.001550, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19732 s, avg_samples: 56.0, ips: 283.79946 samples/s, eta: 23:25:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:47:51] ppocr INFO: epoch: [341/500], global_step: 110500, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.937590, exp_rate<=1 : 0.975910, exp_rate<=2 : 0.994750, exp_rate<=3 : 0.998394, loss: 0.001810, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19142 s, avg_samples: 56.0, ips: 292.54889 samples/s, eta: 23:25:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:48:09] ppocr INFO: epoch: [341/500], global_step: 110600, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.937630, exp_rate<=1 : 0.975981, exp_rate<=2 : 0.994913, exp_rate<=3 : 0.998392, loss: 0.001787, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18673 s, avg_samples: 56.0, ips: 299.90369 samples/s, eta: 23:24:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:48:28] ppocr INFO: epoch: [341/500], global_step: 110700, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.937189, exp_rate<=1 : 0.975923, exp_rate<=2 : 0.994797, exp_rate<=3 : 0.998315, loss: 0.001786, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18579 s, avg_samples: 56.0, ips: 301.41852 samples/s, eta: 23:24:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:48:47] ppocr INFO: epoch: [341/500], global_step: 110800, lr: 0.000100, edit distance: 0.001590, exp_rate: 0.931465, exp_rate<=1 : 0.972868, exp_rate<=2 : 0.992801, exp_rate<=3 : 0.996860, loss: 0.003401, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18658 s, avg_samples: 56.0, ips: 300.13283 samples/s, eta: 23:24:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:49:05] ppocr INFO: epoch: [341/500], global_step: 110900, lr: 0.000100, edit distance: 0.001633, exp_rate: 0.929098, exp_rate<=1 : 0.971665, exp_rate<=2 : 0.992396, exp_rate<=3 : 0.996712, loss: 0.002307, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18720 s, avg_samples: 56.0, ips: 299.14733 samples/s, eta: 23:23:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:49:26] ppocr INFO: epoch: [341/500], global_step: 111000, lr: 0.000100, edit distance: 0.001656, exp_rate: 0.927612, exp_rate<=1 : 0.971038, exp_rate<=2 : 0.992007, exp_rate<=3 : 0.996430, loss: 0.001685, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20197 s, avg_samples: 56.0, ips: 277.27205 samples/s, eta: 23:23:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:49:44] ppocr INFO: epoch: [341/500], global_step: 111100, lr: 0.000100, edit distance: 0.001653, exp_rate: 0.927502, exp_rate<=1 : 0.971040, exp_rate<=2 : 0.992119, exp_rate<=3 : 0.996521, loss: 0.001905, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18810 s, avg_samples: 56.0, ips: 297.71754 samples/s, eta: 23:23:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:50:04] ppocr INFO: epoch: [341/500], global_step: 111200, lr: 0.000100, edit distance: 0.001648, exp_rate: 0.927698, exp_rate<=1 : 0.971032, exp_rate<=2 : 0.992200, exp_rate<=3 : 0.996619, loss: 0.001979, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19272 s, avg_samples: 56.0, ips: 290.58207 samples/s, eta: 23:22:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:50:22] ppocr INFO: epoch: [341/500], global_step: 111300, lr: 0.000100, edit distance: 0.001640, exp_rate: 0.928017, exp_rate<=1 : 0.971221, exp_rate<=2 : 0.992321, exp_rate<=3 : 0.996704, loss: 0.001725, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18565 s, avg_samples: 56.0, ips: 301.64445 samples/s, eta: 23:22:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:50:42] ppocr INFO: epoch: [341/500], global_step: 111400, lr: 0.000100, edit distance: 0.001625, exp_rate: 0.928311, exp_rate<=1 : 0.971412, exp_rate<=2 : 0.992342, exp_rate<=3 : 0.996727, loss: 0.001657, avg_reader_cost: 0.00017 s, avg_batch_cost: 0.19764 s, avg_samples: 56.0, ips: 283.34692 samples/s, eta: 23:22:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:51:02] ppocr INFO: epoch: [341/500], global_step: 111500, lr: 0.000100, edit distance: 0.001617, exp_rate: 0.928278, exp_rate<=1 : 0.971331, exp_rate<=2 : 0.992240, exp_rate<=3 : 0.996684, loss: 0.001577, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19689 s, avg_samples: 56.0, ips: 284.42140 samples/s, eta: 23:21:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:51:21] ppocr INFO: epoch: [341/500], global_step: 111600, lr: 0.000100, edit distance: 0.001602, exp_rate: 0.928867, exp_rate<=1 : 0.971638, exp_rate<=2 : 0.992351, exp_rate<=3 : 0.996742, loss: 0.002124, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19005 s, avg_samples: 56.0, ips: 294.65863 samples/s, eta: 23:21:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:51:39] ppocr INFO: epoch: [341/500], global_step: 111700, lr: 0.000100, edit distance: 0.001582, exp_rate: 0.929442, exp_rate<=1 : 0.971949, exp_rate<=2 : 0.992518, exp_rate<=3 : 0.996874, loss: 0.001408, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17713 s, avg_samples: 56.0, ips: 316.15511 samples/s, eta: 23:21:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:51:57] ppocr INFO: epoch: [341/500], global_step: 111800, lr: 0.000100, edit distance: 0.001583, exp_rate: 0.929427, exp_rate<=1 : 0.971983, exp_rate<=2 : 0.992508, exp_rate<=3 : 0.996870, loss: 0.001454, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18526 s, avg_samples: 56.0, ips: 302.27247 samples/s, eta: 23:20:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:52:17] ppocr INFO: epoch: [341/500], global_step: 111900, lr: 0.000100, edit distance: 0.001575, exp_rate: 0.929480, exp_rate<=1 : 0.972004, exp_rate<=2 : 0.992570, exp_rate<=3 : 0.996944, loss: 0.001340, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19504 s, avg_samples: 56.0, ips: 287.11856 samples/s, eta: 23:20:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:52:35] ppocr INFO: epoch: [341/500], global_step: 112000, lr: 0.000100, edit distance: 0.001566, exp_rate: 0.929632, exp_rate<=1 : 0.972167, exp_rate<=2 : 0.992574, exp_rate<=3 : 0.996932, loss: 0.002080, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.24943 samples/s, eta: 23:20:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:52:56] ppocr INFO: epoch: [341/500], global_step: 112100, lr: 0.000100, edit distance: 0.001573, exp_rate: 0.929263, exp_rate<=1 : 0.972011, exp_rate<=2 : 0.992537, exp_rate<=3 : 0.996941, loss: 0.001876, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20907 s, avg_samples: 56.0, ips: 267.85912 samples/s, eta: 23:19:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:53:16] ppocr INFO: epoch: [341/500], global_step: 112200, lr: 0.000100, edit distance: 0.001574, exp_rate: 0.929199, exp_rate<=1 : 0.971902, exp_rate<=2 : 0.992509, exp_rate<=3 : 0.996929, loss: 0.001856, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.36684 samples/s, eta: 23:19:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:53:35] ppocr INFO: epoch: [341/500], global_step: 112300, lr: 0.000100, edit distance: 0.001573, exp_rate: 0.929080, exp_rate<=1 : 0.971805, exp_rate<=2 : 0.992488, exp_rate<=3 : 0.996918, loss: 0.001295, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19122 s, avg_samples: 56.0, ips: 292.86320 samples/s, eta: 23:19:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:53:43] ppocr INFO: epoch: [341/500], global_step: 112340, lr: 0.000100, edit distance: 0.001568, exp_rate: 0.929136, exp_rate<=1 : 0.971874, exp_rate<=2 : 0.992529, exp_rate<=3 : 0.996940, loss: 0.002233, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07547 s, avg_samples: 22.4, ips: 296.78905 samples/s, eta: 23:19:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:53:45] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 08:53:57] ppocr INFO: epoch: [342/500], global_step: 112400, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.931743, exp_rate<=1 : 0.970503, exp_rate<=2 : 0.991865, exp_rate<=3 : 0.996641, loss: 0.001211, avg_reader_cost: 0.02683 s, avg_batch_cost: 0.14918 s, avg_samples: 33.6, ips: 225.23484 samples/s, eta: 23:19:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:54:15] ppocr INFO: epoch: [342/500], global_step: 112500, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.944648, exp_rate<=1 : 0.978346, exp_rate<=2 : 0.994779, exp_rate<=3 : 0.997983, loss: 0.000792, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17179 s, avg_samples: 56.0, ips: 325.98327 samples/s, eta: 23:18:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:54:35] ppocr INFO: epoch: [342/500], global_step: 112600, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.945550, exp_rate<=1 : 0.979103, exp_rate<=2 : 0.995420, exp_rate<=3 : 0.998232, loss: 0.001707, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19832 s, avg_samples: 56.0, ips: 282.36572 samples/s, eta: 23:18:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:54:53] ppocr INFO: epoch: [342/500], global_step: 112700, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.944378, exp_rate<=1 : 0.978571, exp_rate<=2 : 0.995236, exp_rate<=3 : 0.998347, loss: 0.001702, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18541 s, avg_samples: 56.0, ips: 302.02610 samples/s, eta: 23:17:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:55:14] ppocr INFO: epoch: [342/500], global_step: 112800, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.941515, exp_rate<=1 : 0.976853, exp_rate<=2 : 0.994433, exp_rate<=3 : 0.998036, loss: 0.001223, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20600 s, avg_samples: 56.0, ips: 271.83917 samples/s, eta: 23:17:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:55:34] ppocr INFO: epoch: [342/500], global_step: 112900, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.941371, exp_rate<=1 : 0.976951, exp_rate<=2 : 0.994102, exp_rate<=3 : 0.997948, loss: 0.001753, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19969 s, avg_samples: 56.0, ips: 280.43846 samples/s, eta: 23:17:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:55:53] ppocr INFO: epoch: [342/500], global_step: 113000, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.940760, exp_rate<=1 : 0.976556, exp_rate<=2 : 0.993604, exp_rate<=3 : 0.997584, loss: 0.001179, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19053 s, avg_samples: 56.0, ips: 293.91914 samples/s, eta: 23:17:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:56:12] ppocr INFO: epoch: [342/500], global_step: 113100, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.940654, exp_rate<=1 : 0.976618, exp_rate<=2 : 0.993714, exp_rate<=3 : 0.997597, loss: 0.001585, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18973 s, avg_samples: 56.0, ips: 295.16123 samples/s, eta: 23:16:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:56:32] ppocr INFO: epoch: [342/500], global_step: 113200, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.939403, exp_rate<=1 : 0.976096, exp_rate<=2 : 0.993726, exp_rate<=3 : 0.997567, loss: 0.002274, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20239 s, avg_samples: 56.0, ips: 276.69149 samples/s, eta: 23:16:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:56:53] ppocr INFO: epoch: [342/500], global_step: 113300, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.937852, exp_rate<=1 : 0.975563, exp_rate<=2 : 0.993520, exp_rate<=3 : 0.997577, loss: 0.001964, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20596 s, avg_samples: 56.0, ips: 271.89290 samples/s, eta: 23:16:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:57:12] ppocr INFO: epoch: [342/500], global_step: 113400, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.937844, exp_rate<=1 : 0.975768, exp_rate<=2 : 0.993593, exp_rate<=3 : 0.997620, loss: 0.001210, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19475 s, avg_samples: 56.0, ips: 287.55492 samples/s, eta: 23:15:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:57:31] ppocr INFO: epoch: [342/500], global_step: 113500, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.936972, exp_rate<=1 : 0.975475, exp_rate<=2 : 0.993632, exp_rate<=3 : 0.997658, loss: 0.001760, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18911 s, avg_samples: 56.0, ips: 296.11870 samples/s, eta: 23:15:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:57:51] ppocr INFO: epoch: [342/500], global_step: 113600, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.936899, exp_rate<=1 : 0.975469, exp_rate<=2 : 0.993610, exp_rate<=3 : 0.997685, loss: 0.002029, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20141 s, avg_samples: 56.0, ips: 278.04221 samples/s, eta: 23:15:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:58:09] ppocr INFO: epoch: [342/500], global_step: 113700, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.937761, exp_rate<=1 : 0.975820, exp_rate<=2 : 0.993739, exp_rate<=3 : 0.997724, loss: 0.001387, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18050 s, avg_samples: 56.0, ips: 310.25191 samples/s, eta: 23:14:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:58:28] ppocr INFO: epoch: [342/500], global_step: 113800, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.938357, exp_rate<=1 : 0.976117, exp_rate<=2 : 0.993820, exp_rate<=3 : 0.997821, loss: 0.001776, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18529 s, avg_samples: 56.0, ips: 302.22726 samples/s, eta: 23:14:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:58:45] ppocr INFO: epoch: [342/500], global_step: 113900, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.938228, exp_rate<=1 : 0.976158, exp_rate<=2 : 0.993938, exp_rate<=3 : 0.997892, loss: 0.001588, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17438 s, avg_samples: 56.0, ips: 321.14214 samples/s, eta: 23:14:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:59:04] ppocr INFO: epoch: [342/500], global_step: 114000, lr: 0.000100, edit distance: 0.001354, exp_rate: 0.937808, exp_rate<=1 : 0.975986, exp_rate<=2 : 0.993959, exp_rate<=3 : 0.997924, loss: 0.001204, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19071 s, avg_samples: 56.0, ips: 293.63369 samples/s, eta: 23:13:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:59:23] ppocr INFO: epoch: [342/500], global_step: 114100, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.937439, exp_rate<=1 : 0.975836, exp_rate<=2 : 0.993995, exp_rate<=3 : 0.997869, loss: 0.001409, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18999 s, avg_samples: 56.0, ips: 294.75844 samples/s, eta: 23:13:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 08:59:41] ppocr INFO: epoch: [342/500], global_step: 114200, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.937418, exp_rate<=1 : 0.975785, exp_rate<=2 : 0.993961, exp_rate<=3 : 0.997891, loss: 0.002337, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17840 s, avg_samples: 56.0, ips: 313.89624 samples/s, eta: 23:13:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:00:00] ppocr INFO: epoch: [342/500], global_step: 114300, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.936566, exp_rate<=1 : 0.975430, exp_rate<=2 : 0.993792, exp_rate<=3 : 0.997779, loss: 0.002012, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19022 s, avg_samples: 56.0, ips: 294.39335 samples/s, eta: 23:12:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:00:20] ppocr INFO: epoch: [342/500], global_step: 114400, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.935873, exp_rate<=1 : 0.975285, exp_rate<=2 : 0.993691, exp_rate<=3 : 0.997736, loss: 0.001427, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19966 s, avg_samples: 56.0, ips: 280.47496 samples/s, eta: 23:12:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:00:38] ppocr INFO: epoch: [342/500], global_step: 114500, lr: 0.000100, edit distance: 0.001398, exp_rate: 0.935919, exp_rate<=1 : 0.975316, exp_rate<=2 : 0.993763, exp_rate<=3 : 0.997762, loss: 0.001710, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17882 s, avg_samples: 56.0, ips: 313.15825 samples/s, eta: 23:12:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:00:57] ppocr INFO: epoch: [342/500], global_step: 114600, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.935845, exp_rate<=1 : 0.975270, exp_rate<=2 : 0.993754, exp_rate<=3 : 0.997780, loss: 0.001749, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19046 s, avg_samples: 56.0, ips: 294.02705 samples/s, eta: 23:11:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:01:17] ppocr INFO: epoch: [342/500], global_step: 114700, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.935391, exp_rate<=1 : 0.975034, exp_rate<=2 : 0.993724, exp_rate<=3 : 0.997755, loss: 0.001923, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19832 s, avg_samples: 56.0, ips: 282.37463 samples/s, eta: 23:11:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:01:36] ppocr INFO: epoch: [342/500], global_step: 114800, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.935130, exp_rate<=1 : 0.974942, exp_rate<=2 : 0.993733, exp_rate<=3 : 0.997761, loss: 0.002023, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18555 s, avg_samples: 56.0, ips: 301.80629 samples/s, eta: 23:11:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:01:56] ppocr INFO: epoch: [342/500], global_step: 114900, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.934723, exp_rate<=1 : 0.974800, exp_rate<=2 : 0.993651, exp_rate<=3 : 0.997699, loss: 0.001543, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19961 s, avg_samples: 56.0, ips: 280.55017 samples/s, eta: 23:10:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:02:15] ppocr INFO: epoch: [342/500], global_step: 115000, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.934489, exp_rate<=1 : 0.974773, exp_rate<=2 : 0.993701, exp_rate<=3 : 0.997724, loss: 0.001531, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19610 s, avg_samples: 56.0, ips: 285.57436 samples/s, eta: 23:10:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:02:30] ppocr INFO: epoch: [342/500], global_step: 115080, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.934273, exp_rate<=1 : 0.974635, exp_rate<=2 : 0.993655, exp_rate<=3 : 0.997708, loss: 0.001347, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14950 s, avg_samples: 44.8, ips: 299.66795 samples/s, eta: 23:10:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:02:32] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 09:02:37] ppocr INFO: epoch: [343/500], global_step: 115100, lr: 0.000100, edit distance: 0.001065, exp_rate: 0.946429, exp_rate<=1 : 0.976190, exp_rate<=2 : 0.993464, exp_rate<=3 : 0.997385, loss: 0.001446, avg_reader_cost: 0.02637 s, avg_batch_cost: 0.06786 s, avg_samples: 11.2, ips: 165.04479 samples/s, eta: 23:10:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:02:56] ppocr INFO: epoch: [343/500], global_step: 115200, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.941845, exp_rate<=1 : 0.974468, exp_rate<=2 : 0.994788, exp_rate<=3 : 0.998167, loss: 0.000929, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19013 s, avg_samples: 56.0, ips: 294.53753 samples/s, eta: 23:09:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:03:15] ppocr INFO: epoch: [343/500], global_step: 115300, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.944343, exp_rate<=1 : 0.977006, exp_rate<=2 : 0.995450, exp_rate<=3 : 0.998526, loss: 0.001249, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19177 s, avg_samples: 56.0, ips: 292.01721 samples/s, eta: 23:09:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:03:36] ppocr INFO: epoch: [343/500], global_step: 115400, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.940875, exp_rate<=1 : 0.975277, exp_rate<=2 : 0.993916, exp_rate<=3 : 0.998291, loss: 0.001000, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21102 s, avg_samples: 56.0, ips: 265.37143 samples/s, eta: 23:09:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:03:54] ppocr INFO: epoch: [343/500], global_step: 115500, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.944043, exp_rate<=1 : 0.978097, exp_rate<=2 : 0.994821, exp_rate<=3 : 0.998505, loss: 0.001455, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17193 s, avg_samples: 56.0, ips: 325.71322 samples/s, eta: 23:08:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:04:12] ppocr INFO: epoch: [343/500], global_step: 115600, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.942579, exp_rate<=1 : 0.977344, exp_rate<=2 : 0.994391, exp_rate<=3 : 0.998200, loss: 0.001348, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18426 s, avg_samples: 56.0, ips: 303.91223 samples/s, eta: 23:08:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:04:32] ppocr INFO: epoch: [343/500], global_step: 115700, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.940486, exp_rate<=1 : 0.976600, exp_rate<=2 : 0.994062, exp_rate<=3 : 0.998011, loss: 0.001431, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19505 s, avg_samples: 56.0, ips: 287.10432 samples/s, eta: 23:08:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:04:51] ppocr INFO: epoch: [343/500], global_step: 115800, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.939581, exp_rate<=1 : 0.976469, exp_rate<=2 : 0.994010, exp_rate<=3 : 0.997947, loss: 0.001311, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18969 s, avg_samples: 56.0, ips: 295.22494 samples/s, eta: 23:07:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:05:09] ppocr INFO: epoch: [343/500], global_step: 115900, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.939330, exp_rate<=1 : 0.976500, exp_rate<=2 : 0.994129, exp_rate<=3 : 0.998037, loss: 0.002001, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18648 s, avg_samples: 56.0, ips: 300.29344 samples/s, eta: 23:07:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:05:29] ppocr INFO: epoch: [343/500], global_step: 116000, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.938788, exp_rate<=1 : 0.976187, exp_rate<=2 : 0.994097, exp_rate<=3 : 0.998034, loss: 0.002040, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19850 s, avg_samples: 56.0, ips: 282.11749 samples/s, eta: 23:07:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:05:49] ppocr INFO: epoch: [343/500], global_step: 116100, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.937876, exp_rate<=1 : 0.975878, exp_rate<=2 : 0.993942, exp_rate<=3 : 0.998028, loss: 0.001626, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19912 s, avg_samples: 56.0, ips: 281.23596 samples/s, eta: 23:07:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:06:09] ppocr INFO: epoch: [343/500], global_step: 116200, lr: 0.000100, edit distance: 0.001387, exp_rate: 0.936934, exp_rate<=1 : 0.975510, exp_rate<=2 : 0.993801, exp_rate<=3 : 0.997952, loss: 0.001197, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19749 s, avg_samples: 56.0, ips: 283.55795 samples/s, eta: 23:06:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:06:28] ppocr INFO: epoch: [343/500], global_step: 116300, lr: 0.000100, edit distance: 0.001414, exp_rate: 0.936401, exp_rate<=1 : 0.975450, exp_rate<=2 : 0.993760, exp_rate<=3 : 0.997925, loss: 0.001587, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19152 s, avg_samples: 56.0, ips: 292.39347 samples/s, eta: 23:06:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:06:47] ppocr INFO: epoch: [343/500], global_step: 116400, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.935818, exp_rate<=1 : 0.975068, exp_rate<=2 : 0.993699, exp_rate<=3 : 0.997920, loss: 0.001873, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18659 s, avg_samples: 56.0, ips: 300.12939 samples/s, eta: 23:06:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:07:06] ppocr INFO: epoch: [343/500], global_step: 116500, lr: 0.000100, edit distance: 0.001450, exp_rate: 0.934953, exp_rate<=1 : 0.974752, exp_rate<=2 : 0.993570, exp_rate<=3 : 0.997823, loss: 0.001323, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19498 s, avg_samples: 56.0, ips: 287.21348 samples/s, eta: 23:05:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:07:25] ppocr INFO: epoch: [343/500], global_step: 116600, lr: 0.000100, edit distance: 0.001457, exp_rate: 0.934412, exp_rate<=1 : 0.974482, exp_rate<=2 : 0.993430, exp_rate<=3 : 0.997839, loss: 0.001634, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18531 s, avg_samples: 56.0, ips: 302.19699 samples/s, eta: 23:05:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:07:45] ppocr INFO: epoch: [343/500], global_step: 116700, lr: 0.000100, edit distance: 0.001481, exp_rate: 0.933170, exp_rate<=1 : 0.973820, exp_rate<=2 : 0.993293, exp_rate<=3 : 0.997739, loss: 0.001835, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20262 s, avg_samples: 56.0, ips: 276.38370 samples/s, eta: 23:05:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:08:04] ppocr INFO: epoch: [343/500], global_step: 116800, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.932448, exp_rate<=1 : 0.973509, exp_rate<=2 : 0.993218, exp_rate<=3 : 0.997765, loss: 0.001589, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19184 s, avg_samples: 56.0, ips: 291.91402 samples/s, eta: 23:04:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:08:24] ppocr INFO: epoch: [343/500], global_step: 116900, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.931735, exp_rate<=1 : 0.973012, exp_rate<=2 : 0.993208, exp_rate<=3 : 0.997743, loss: 0.002017, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19650 s, avg_samples: 56.0, ips: 284.98711 samples/s, eta: 23:04:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:08:44] ppocr INFO: epoch: [343/500], global_step: 117000, lr: 0.000100, edit distance: 0.001524, exp_rate: 0.931043, exp_rate<=1 : 0.972715, exp_rate<=2 : 0.993161, exp_rate<=3 : 0.997718, loss: 0.001657, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19682 s, avg_samples: 56.0, ips: 284.52390 samples/s, eta: 23:04:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:09:03] ppocr INFO: epoch: [343/500], global_step: 117100, lr: 0.000100, edit distance: 0.001526, exp_rate: 0.930950, exp_rate<=1 : 0.972668, exp_rate<=2 : 0.993204, exp_rate<=3 : 0.997711, loss: 0.001433, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19323 s, avg_samples: 56.0, ips: 289.81574 samples/s, eta: 23:03:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:09:21] ppocr INFO: epoch: [343/500], global_step: 117200, lr: 0.000100, edit distance: 0.001526, exp_rate: 0.931077, exp_rate<=1 : 0.972724, exp_rate<=2 : 0.993225, exp_rate<=3 : 0.997749, loss: 0.001395, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18231 s, avg_samples: 56.0, ips: 307.17732 samples/s, eta: 23:03:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:09:39] ppocr INFO: epoch: [343/500], global_step: 117300, lr: 0.000100, edit distance: 0.001526, exp_rate: 0.930979, exp_rate<=1 : 0.972533, exp_rate<=2 : 0.993174, exp_rate<=3 : 0.997738, loss: 0.001319, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17973 s, avg_samples: 56.0, ips: 311.57907 samples/s, eta: 23:03:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:09:58] ppocr INFO: epoch: [343/500], global_step: 117400, lr: 0.000100, edit distance: 0.001533, exp_rate: 0.930846, exp_rate<=1 : 0.972350, exp_rate<=2 : 0.993211, exp_rate<=3 : 0.997729, loss: 0.002147, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19051 s, avg_samples: 56.0, ips: 293.95269 samples/s, eta: 23:02:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:10:19] ppocr INFO: epoch: [343/500], global_step: 117500, lr: 0.000100, edit distance: 0.001535, exp_rate: 0.930285, exp_rate<=1 : 0.972025, exp_rate<=2 : 0.993153, exp_rate<=3 : 0.997743, loss: 0.002011, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20508 s, avg_samples: 56.0, ips: 273.06182 samples/s, eta: 23:02:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:10:37] ppocr INFO: epoch: [343/500], global_step: 117600, lr: 0.000100, edit distance: 0.001545, exp_rate: 0.930194, exp_rate<=1 : 0.972016, exp_rate<=2 : 0.993132, exp_rate<=3 : 0.997767, loss: 0.001711, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18332 s, avg_samples: 56.0, ips: 305.48506 samples/s, eta: 23:02:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:10:57] ppocr INFO: epoch: [343/500], global_step: 117700, lr: 0.000100, edit distance: 0.001554, exp_rate: 0.929848, exp_rate<=1 : 0.971884, exp_rate<=2 : 0.993155, exp_rate<=3 : 0.997780, loss: 0.001590, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19951 s, avg_samples: 56.0, ips: 280.68602 samples/s, eta: 23:01:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:11:16] ppocr INFO: epoch: [343/500], global_step: 117800, lr: 0.000100, edit distance: 0.001548, exp_rate: 0.930071, exp_rate<=1 : 0.971916, exp_rate<=2 : 0.993160, exp_rate<=3 : 0.997770, loss: 0.001864, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19154 s, avg_samples: 56.0, ips: 292.35979 samples/s, eta: 23:01:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:11:20] ppocr INFO: epoch: [343/500], global_step: 117820, lr: 0.000100, edit distance: 0.001548, exp_rate: 0.930048, exp_rate<=1 : 0.971914, exp_rate<=2 : 0.993133, exp_rate<=3 : 0.997772, loss: 0.002100, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03761 s, avg_samples: 11.2, ips: 297.82486 samples/s, eta: 23:01:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:11:22] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 09:11:38] ppocr INFO: epoch: [344/500], global_step: 117900, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.938188, exp_rate<=1 : 0.971356, exp_rate<=2 : 0.994985, exp_rate<=3 : 0.998827, loss: 0.001777, avg_reader_cost: 0.02655 s, avg_batch_cost: 0.17806 s, avg_samples: 44.8, ips: 251.59693 samples/s, eta: 23:01:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:11:57] ppocr INFO: epoch: [344/500], global_step: 118000, lr: 0.000100, edit distance: 0.001449, exp_rate: 0.938444, exp_rate<=1 : 0.974422, exp_rate<=2 : 0.994574, exp_rate<=3 : 0.998429, loss: 0.001269, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19298 s, avg_samples: 56.0, ips: 290.19088 samples/s, eta: 23:01:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:12:17] ppocr INFO: epoch: [344/500], global_step: 118100, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.937450, exp_rate<=1 : 0.974167, exp_rate<=2 : 0.994578, exp_rate<=3 : 0.998374, loss: 0.001305, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19506 s, avg_samples: 56.0, ips: 287.09274 samples/s, eta: 23:00:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:12:36] ppocr INFO: epoch: [344/500], global_step: 118200, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.938586, exp_rate<=1 : 0.975131, exp_rate<=2 : 0.994585, exp_rate<=3 : 0.998281, loss: 0.001983, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18982 s, avg_samples: 56.0, ips: 295.01159 samples/s, eta: 23:00:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:12:55] ppocr INFO: epoch: [344/500], global_step: 118300, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.938939, exp_rate<=1 : 0.975534, exp_rate<=2 : 0.994573, exp_rate<=3 : 0.998292, loss: 0.001460, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18905 s, avg_samples: 56.0, ips: 296.21953 samples/s, eta: 23:00:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:13:14] ppocr INFO: epoch: [344/500], global_step: 118400, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.940382, exp_rate<=1 : 0.976341, exp_rate<=2 : 0.994797, exp_rate<=3 : 0.998399, loss: 0.002104, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19022 s, avg_samples: 56.0, ips: 294.39896 samples/s, eta: 22:59:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:13:33] ppocr INFO: epoch: [344/500], global_step: 118500, lr: 0.000100, edit distance: 0.001387, exp_rate: 0.939544, exp_rate<=1 : 0.975968, exp_rate<=2 : 0.994603, exp_rate<=3 : 0.998250, loss: 0.001406, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19233 s, avg_samples: 56.0, ips: 291.16542 samples/s, eta: 22:59:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:13:50] ppocr INFO: epoch: [344/500], global_step: 118600, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.940318, exp_rate<=1 : 0.976328, exp_rate<=2 : 0.994573, exp_rate<=3 : 0.998280, loss: 0.001791, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16481 s, avg_samples: 56.0, ips: 339.78224 samples/s, eta: 22:58:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:14:10] ppocr INFO: epoch: [344/500], global_step: 118700, lr: 0.000100, edit distance: 0.001448, exp_rate: 0.938634, exp_rate<=1 : 0.975806, exp_rate<=2 : 0.994342, exp_rate<=3 : 0.998078, loss: 0.001439, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20449 s, avg_samples: 56.0, ips: 273.85112 samples/s, eta: 22:58:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:14:29] ppocr INFO: epoch: [344/500], global_step: 118800, lr: 0.000100, edit distance: 0.001462, exp_rate: 0.937964, exp_rate<=1 : 0.975253, exp_rate<=2 : 0.993943, exp_rate<=3 : 0.997774, loss: 0.001881, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19228 s, avg_samples: 56.0, ips: 291.24605 samples/s, eta: 22:58:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:14:49] ppocr INFO: epoch: [344/500], global_step: 118900, lr: 0.000100, edit distance: 0.001452, exp_rate: 0.937646, exp_rate<=1 : 0.974945, exp_rate<=2 : 0.993978, exp_rate<=3 : 0.997798, loss: 0.001542, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19653 s, avg_samples: 56.0, ips: 284.94974 samples/s, eta: 22:58:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:15:09] ppocr INFO: epoch: [344/500], global_step: 119000, lr: 0.000100, edit distance: 0.001449, exp_rate: 0.937340, exp_rate<=1 : 0.974562, exp_rate<=2 : 0.993806, exp_rate<=3 : 0.997742, loss: 0.001345, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19553 s, avg_samples: 56.0, ips: 286.40256 samples/s, eta: 22:57:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:15:29] ppocr INFO: epoch: [344/500], global_step: 119100, lr: 0.000100, edit distance: 0.001452, exp_rate: 0.936527, exp_rate<=1 : 0.974267, exp_rate<=2 : 0.993822, exp_rate<=3 : 0.997772, loss: 0.001201, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20378 s, avg_samples: 56.0, ips: 274.80487 samples/s, eta: 22:57:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:15:47] ppocr INFO: epoch: [344/500], global_step: 119200, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.937177, exp_rate<=1 : 0.974679, exp_rate<=2 : 0.993950, exp_rate<=3 : 0.997868, loss: 0.001539, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17564 s, avg_samples: 56.0, ips: 318.82623 samples/s, eta: 22:57:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:16:06] ppocr INFO: epoch: [344/500], global_step: 119300, lr: 0.000100, edit distance: 0.001447, exp_rate: 0.937096, exp_rate<=1 : 0.974629, exp_rate<=2 : 0.993942, exp_rate<=3 : 0.997852, loss: 0.001942, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19145 s, avg_samples: 56.0, ips: 292.49882 samples/s, eta: 22:56:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:16:26] ppocr INFO: epoch: [344/500], global_step: 119400, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.936825, exp_rate<=1 : 0.974403, exp_rate<=2 : 0.993854, exp_rate<=3 : 0.997851, loss: 0.001501, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20558 s, avg_samples: 56.0, ips: 272.40609 samples/s, eta: 22:56:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:16:45] ppocr INFO: epoch: [344/500], global_step: 119500, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.936722, exp_rate<=1 : 0.974464, exp_rate<=2 : 0.993852, exp_rate<=3 : 0.997903, loss: 0.001336, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18627 s, avg_samples: 56.0, ips: 300.64197 samples/s, eta: 22:56:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:17:04] ppocr INFO: epoch: [344/500], global_step: 119600, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.936223, exp_rate<=1 : 0.974269, exp_rate<=2 : 0.993618, exp_rate<=3 : 0.997874, loss: 0.001423, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19569 s, avg_samples: 56.0, ips: 286.17203 samples/s, eta: 22:55:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:17:23] ppocr INFO: epoch: [344/500], global_step: 119700, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.935902, exp_rate<=1 : 0.974120, exp_rate<=2 : 0.993598, exp_rate<=3 : 0.997850, loss: 0.001875, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18966 s, avg_samples: 56.0, ips: 295.26605 samples/s, eta: 22:55:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:17:43] ppocr INFO: epoch: [344/500], global_step: 119800, lr: 0.000100, edit distance: 0.001460, exp_rate: 0.935903, exp_rate<=1 : 0.974144, exp_rate<=2 : 0.993622, exp_rate<=3 : 0.997873, loss: 0.002019, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19132 s, avg_samples: 56.0, ips: 292.69800 samples/s, eta: 22:55:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:18:02] ppocr INFO: epoch: [344/500], global_step: 119900, lr: 0.000100, edit distance: 0.001464, exp_rate: 0.935351, exp_rate<=1 : 0.974039, exp_rate<=2 : 0.993657, exp_rate<=3 : 0.997890, loss: 0.001605, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19571 s, avg_samples: 56.0, ips: 286.14401 samples/s, eta: 22:54:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:18:20] ppocr INFO: epoch: [344/500], global_step: 120000, lr: 0.000100, edit distance: 0.001463, exp_rate: 0.935110, exp_rate<=1 : 0.973932, exp_rate<=2 : 0.993698, exp_rate<=3 : 0.997910, loss: 0.001537, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17854 s, avg_samples: 56.0, ips: 313.66270 samples/s, eta: 22:54:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:18:40] ppocr INFO: epoch: [344/500], global_step: 120100, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.934647, exp_rate<=1 : 0.973706, exp_rate<=2 : 0.993655, exp_rate<=3 : 0.997872, loss: 0.002162, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19598 s, avg_samples: 56.0, ips: 285.73894 samples/s, eta: 22:54:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:18:59] ppocr INFO: epoch: [344/500], global_step: 120200, lr: 0.000100, edit distance: 0.001461, exp_rate: 0.934341, exp_rate<=1 : 0.973641, exp_rate<=2 : 0.993702, exp_rate<=3 : 0.997876, loss: 0.001684, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19555 s, avg_samples: 56.0, ips: 286.36499 samples/s, eta: 22:53:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:19:17] ppocr INFO: epoch: [344/500], global_step: 120300, lr: 0.000100, edit distance: 0.001461, exp_rate: 0.934414, exp_rate<=1 : 0.973769, exp_rate<=2 : 0.993734, exp_rate<=3 : 0.997896, loss: 0.001833, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17585 s, avg_samples: 56.0, ips: 318.45100 samples/s, eta: 22:53:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:19:35] ppocr INFO: epoch: [344/500], global_step: 120400, lr: 0.000100, edit distance: 0.001473, exp_rate: 0.934095, exp_rate<=1 : 0.973710, exp_rate<=2 : 0.993674, exp_rate<=3 : 0.997860, loss: 0.001764, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17954 s, avg_samples: 56.0, ips: 311.90323 samples/s, eta: 22:53:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:19:55] ppocr INFO: epoch: [344/500], global_step: 120500, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.933814, exp_rate<=1 : 0.973673, exp_rate<=2 : 0.993700, exp_rate<=3 : 0.997872, loss: 0.001650, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19710 s, avg_samples: 56.0, ips: 284.12364 samples/s, eta: 22:52:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:20:07] ppocr INFO: epoch: [344/500], global_step: 120560, lr: 0.000100, edit distance: 0.001471, exp_rate: 0.933571, exp_rate<=1 : 0.973635, exp_rate<=2 : 0.993753, exp_rate<=3 : 0.997898, loss: 0.001746, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.12395 s, avg_samples: 33.6, ips: 271.08127 samples/s, eta: 22:52:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:20:09] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 09:20:18] ppocr INFO: epoch: [345/500], global_step: 120600, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.931170, exp_rate<=1 : 0.968211, exp_rate<=2 : 0.987245, exp_rate<=3 : 0.994447, loss: 0.001795, avg_reader_cost: 0.02664 s, avg_batch_cost: 0.10924 s, avg_samples: 22.4, ips: 205.05383 samples/s, eta: 22:52:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:20:38] ppocr INFO: epoch: [345/500], global_step: 120700, lr: 0.000100, edit distance: 0.001386, exp_rate: 0.934358, exp_rate<=1 : 0.971858, exp_rate<=2 : 0.991306, exp_rate<=3 : 0.996656, loss: 0.001713, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19898 s, avg_samples: 56.0, ips: 281.42963 samples/s, eta: 22:52:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:20:59] ppocr INFO: epoch: [345/500], global_step: 120800, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.935533, exp_rate<=1 : 0.973119, exp_rate<=2 : 0.992489, exp_rate<=3 : 0.996746, loss: 0.001413, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20617 s, avg_samples: 56.0, ips: 271.62261 samples/s, eta: 22:52:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:21:16] ppocr INFO: epoch: [345/500], global_step: 120900, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.938624, exp_rate<=1 : 0.975476, exp_rate<=2 : 0.993290, exp_rate<=3 : 0.997216, loss: 0.001139, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17137 s, avg_samples: 56.0, ips: 326.77841 samples/s, eta: 22:51:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:21:33] ppocr INFO: epoch: [345/500], global_step: 121000, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.941005, exp_rate<=1 : 0.976579, exp_rate<=2 : 0.993897, exp_rate<=3 : 0.997493, loss: 0.002280, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16859 s, avg_samples: 56.0, ips: 332.17341 samples/s, eta: 22:51:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:21:53] ppocr INFO: epoch: [345/500], global_step: 121100, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.937693, exp_rate<=1 : 0.975024, exp_rate<=2 : 0.993066, exp_rate<=3 : 0.997179, loss: 0.001727, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20230 s, avg_samples: 56.0, ips: 276.82249 samples/s, eta: 22:50:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:22:11] ppocr INFO: epoch: [345/500], global_step: 121200, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.938925, exp_rate<=1 : 0.975438, exp_rate<=2 : 0.993123, exp_rate<=3 : 0.997229, loss: 0.002126, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18638 s, avg_samples: 56.0, ips: 300.46672 samples/s, eta: 22:50:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:22:32] ppocr INFO: epoch: [345/500], global_step: 121300, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.937415, exp_rate<=1 : 0.974905, exp_rate<=2 : 0.992985, exp_rate<=3 : 0.997131, loss: 0.002312, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20378 s, avg_samples: 56.0, ips: 274.81003 samples/s, eta: 22:50:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:22:50] ppocr INFO: epoch: [345/500], global_step: 121400, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.937731, exp_rate<=1 : 0.975277, exp_rate<=2 : 0.993270, exp_rate<=3 : 0.997248, loss: 0.001686, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17734 s, avg_samples: 56.0, ips: 315.77228 samples/s, eta: 22:49:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:23:09] ppocr INFO: epoch: [345/500], global_step: 121500, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.937385, exp_rate<=1 : 0.975245, exp_rate<=2 : 0.993382, exp_rate<=3 : 0.997313, loss: 0.001668, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19251 s, avg_samples: 56.0, ips: 290.89579 samples/s, eta: 22:49:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:23:29] ppocr INFO: epoch: [345/500], global_step: 121600, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.936387, exp_rate<=1 : 0.974902, exp_rate<=2 : 0.993597, exp_rate<=3 : 0.997418, loss: 0.001909, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20150 s, avg_samples: 56.0, ips: 277.91959 samples/s, eta: 22:49:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:23:48] ppocr INFO: epoch: [345/500], global_step: 121700, lr: 0.000100, edit distance: 0.001351, exp_rate: 0.936661, exp_rate<=1 : 0.975124, exp_rate<=2 : 0.993751, exp_rate<=3 : 0.997464, loss: 0.001928, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18679 s, avg_samples: 56.0, ips: 299.80450 samples/s, eta: 22:48:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:24:06] ppocr INFO: epoch: [345/500], global_step: 121800, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.936760, exp_rate<=1 : 0.975147, exp_rate<=2 : 0.993862, exp_rate<=3 : 0.997597, loss: 0.001184, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18515 s, avg_samples: 56.0, ips: 302.45448 samples/s, eta: 22:48:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:24:25] ppocr INFO: epoch: [345/500], global_step: 121900, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.936255, exp_rate<=1 : 0.974918, exp_rate<=2 : 0.993855, exp_rate<=3 : 0.997642, loss: 0.001604, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18923 s, avg_samples: 56.0, ips: 295.93207 samples/s, eta: 22:48:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:24:45] ppocr INFO: epoch: [345/500], global_step: 122000, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.936271, exp_rate<=1 : 0.974831, exp_rate<=2 : 0.993841, exp_rate<=3 : 0.997634, loss: 0.001760, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19672 s, avg_samples: 56.0, ips: 284.66321 samples/s, eta: 22:47:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:25:03] ppocr INFO: epoch: [345/500], global_step: 122100, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.936044, exp_rate<=1 : 0.974679, exp_rate<=2 : 0.993793, exp_rate<=3 : 0.997608, loss: 0.001692, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17913 s, avg_samples: 56.0, ips: 312.62835 samples/s, eta: 22:47:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:25:21] ppocr INFO: epoch: [345/500], global_step: 122200, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.936322, exp_rate<=1 : 0.974800, exp_rate<=2 : 0.993836, exp_rate<=3 : 0.997593, loss: 0.000952, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18402 s, avg_samples: 56.0, ips: 304.30919 samples/s, eta: 22:47:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:25:41] ppocr INFO: epoch: [345/500], global_step: 122300, lr: 0.000100, edit distance: 0.001360, exp_rate: 0.935957, exp_rate<=1 : 0.974631, exp_rate<=2 : 0.993872, exp_rate<=3 : 0.997594, loss: 0.002198, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19415 s, avg_samples: 56.0, ips: 288.42986 samples/s, eta: 22:46:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:26:00] ppocr INFO: epoch: [345/500], global_step: 122400, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.935782, exp_rate<=1 : 0.974500, exp_rate<=2 : 0.993879, exp_rate<=3 : 0.997576, loss: 0.001194, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19248 s, avg_samples: 56.0, ips: 290.94309 samples/s, eta: 22:46:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:26:20] ppocr INFO: epoch: [345/500], global_step: 122500, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.935600, exp_rate<=1 : 0.974506, exp_rate<=2 : 0.993969, exp_rate<=3 : 0.997613, loss: 0.001895, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19591 s, avg_samples: 56.0, ips: 285.84239 samples/s, eta: 22:46:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:26:39] ppocr INFO: epoch: [345/500], global_step: 122600, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.935340, exp_rate<=1 : 0.974470, exp_rate<=2 : 0.993945, exp_rate<=3 : 0.997598, loss: 0.001690, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19797 s, avg_samples: 56.0, ips: 282.86554 samples/s, eta: 22:45:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:26:59] ppocr INFO: epoch: [345/500], global_step: 122700, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.934849, exp_rate<=1 : 0.974324, exp_rate<=2 : 0.993985, exp_rate<=3 : 0.997634, loss: 0.001147, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19758 s, avg_samples: 56.0, ips: 283.42575 samples/s, eta: 22:45:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:27:17] ppocr INFO: epoch: [345/500], global_step: 122800, lr: 0.000100, edit distance: 0.001384, exp_rate: 0.934988, exp_rate<=1 : 0.974413, exp_rate<=2 : 0.993970, exp_rate<=3 : 0.997643, loss: 0.001254, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18005 s, avg_samples: 56.0, ips: 311.01857 samples/s, eta: 22:45:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:27:36] ppocr INFO: epoch: [345/500], global_step: 122900, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.935265, exp_rate<=1 : 0.974653, exp_rate<=2 : 0.993988, exp_rate<=3 : 0.997660, loss: 0.001736, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18901 s, avg_samples: 56.0, ips: 296.28069 samples/s, eta: 22:44:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:27:56] ppocr INFO: epoch: [345/500], global_step: 123000, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.934910, exp_rate<=1 : 0.974395, exp_rate<=2 : 0.993885, exp_rate<=3 : 0.997627, loss: 0.002588, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19536 s, avg_samples: 56.0, ips: 286.65023 samples/s, eta: 22:44:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:28:15] ppocr INFO: epoch: [345/500], global_step: 123100, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.934753, exp_rate<=1 : 0.974311, exp_rate<=2 : 0.993903, exp_rate<=3 : 0.997650, loss: 0.000912, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19335 s, avg_samples: 56.0, ips: 289.62628 samples/s, eta: 22:44:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:28:36] ppocr INFO: epoch: [345/500], global_step: 123200, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.934640, exp_rate<=1 : 0.974280, exp_rate<=2 : 0.993866, exp_rate<=3 : 0.997624, loss: 0.002246, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20560 s, avg_samples: 56.0, ips: 272.37878 samples/s, eta: 22:44:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:28:53] ppocr INFO: epoch: [345/500], global_step: 123300, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.934453, exp_rate<=1 : 0.974182, exp_rate<=2 : 0.993840, exp_rate<=3 : 0.997640, loss: 0.001568, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17364 s, avg_samples: 56.0, ips: 322.49717 samples/s, eta: 22:43:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:28:55] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 09:28:56] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_345 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 09:29:16] ppocr INFO: epoch: [346/500], global_step: 123400, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.939532, exp_rate<=1 : 0.975586, exp_rate<=2 : 0.993515, exp_rate<=3 : 0.996976, loss: 0.001391, avg_reader_cost: 0.03356 s, avg_batch_cost: 0.22513 s, avg_samples: 56.0, ips: 248.74586 samples/s, eta: 22:43:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:29:34] ppocr INFO: epoch: [346/500], global_step: 123500, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.941468, exp_rate<=1 : 0.976749, exp_rate<=2 : 0.994428, exp_rate<=3 : 0.997880, loss: 0.001827, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18221 s, avg_samples: 56.0, ips: 307.33327 samples/s, eta: 22:43:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:29:51] ppocr INFO: epoch: [346/500], global_step: 123600, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.939304, exp_rate<=1 : 0.976699, exp_rate<=2 : 0.994609, exp_rate<=3 : 0.997988, loss: 0.001885, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17495 s, avg_samples: 56.0, ips: 320.08773 samples/s, eta: 22:42:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:30:09] ppocr INFO: epoch: [346/500], global_step: 123700, lr: 0.000100, edit distance: 0.001372, exp_rate: 0.937838, exp_rate<=1 : 0.975790, exp_rate<=2 : 0.994556, exp_rate<=3 : 0.997843, loss: 0.002154, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18080 s, avg_samples: 56.0, ips: 309.73235 samples/s, eta: 22:42:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:30:30] ppocr INFO: epoch: [346/500], global_step: 123800, lr: 0.000100, edit distance: 0.001435, exp_rate: 0.935819, exp_rate<=1 : 0.974077, exp_rate<=2 : 0.993903, exp_rate<=3 : 0.997712, loss: 0.001405, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20166 s, avg_samples: 56.0, ips: 277.70117 samples/s, eta: 22:42:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:30:50] ppocr INFO: epoch: [346/500], global_step: 123900, lr: 0.000100, edit distance: 0.001495, exp_rate: 0.932953, exp_rate<=1 : 0.972972, exp_rate<=2 : 0.993424, exp_rate<=3 : 0.997430, loss: 0.001868, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20434 s, avg_samples: 56.0, ips: 274.05837 samples/s, eta: 22:41:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:31:09] ppocr INFO: epoch: [346/500], global_step: 124000, lr: 0.000100, edit distance: 0.001508, exp_rate: 0.932584, exp_rate<=1 : 0.972727, exp_rate<=2 : 0.993114, exp_rate<=3 : 0.997257, loss: 0.001696, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18853 s, avg_samples: 56.0, ips: 297.02962 samples/s, eta: 22:41:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:31:28] ppocr INFO: epoch: [346/500], global_step: 124100, lr: 0.000100, edit distance: 0.001489, exp_rate: 0.932256, exp_rate<=1 : 0.972579, exp_rate<=2 : 0.992929, exp_rate<=3 : 0.997205, loss: 0.001336, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19273 s, avg_samples: 56.0, ips: 290.55838 samples/s, eta: 22:41:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:31:49] ppocr INFO: epoch: [346/500], global_step: 124200, lr: 0.000100, edit distance: 0.001494, exp_rate: 0.932089, exp_rate<=1 : 0.972739, exp_rate<=2 : 0.992981, exp_rate<=3 : 0.997092, loss: 0.001321, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20262 s, avg_samples: 56.0, ips: 276.37385 samples/s, eta: 22:40:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:32:08] ppocr INFO: epoch: [346/500], global_step: 124300, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.932988, exp_rate<=1 : 0.973106, exp_rate<=2 : 0.993221, exp_rate<=3 : 0.997245, loss: 0.001626, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18995 s, avg_samples: 56.0, ips: 294.81692 samples/s, eta: 22:40:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:32:26] ppocr INFO: epoch: [346/500], global_step: 124400, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.932496, exp_rate<=1 : 0.972865, exp_rate<=2 : 0.993013, exp_rate<=3 : 0.997314, loss: 0.001260, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18916 s, avg_samples: 56.0, ips: 296.04630 samples/s, eta: 22:40:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:32:47] ppocr INFO: epoch: [346/500], global_step: 124500, lr: 0.000100, edit distance: 0.001441, exp_rate: 0.932743, exp_rate<=1 : 0.972962, exp_rate<=2 : 0.993130, exp_rate<=3 : 0.997381, loss: 0.001250, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20193 s, avg_samples: 56.0, ips: 277.32309 samples/s, eta: 22:39:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:33:07] ppocr INFO: epoch: [346/500], global_step: 124600, lr: 0.000100, edit distance: 0.001431, exp_rate: 0.933058, exp_rate<=1 : 0.973131, exp_rate<=2 : 0.993178, exp_rate<=3 : 0.997482, loss: 0.001367, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19913 s, avg_samples: 56.0, ips: 281.23029 samples/s, eta: 22:39:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:33:25] ppocr INFO: epoch: [346/500], global_step: 124700, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.933775, exp_rate<=1 : 0.973323, exp_rate<=2 : 0.993352, exp_rate<=3 : 0.997544, loss: 0.001489, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18135 s, avg_samples: 56.0, ips: 308.80067 samples/s, eta: 22:39:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:33:43] ppocr INFO: epoch: [346/500], global_step: 124800, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.934184, exp_rate<=1 : 0.973546, exp_rate<=2 : 0.993427, exp_rate<=3 : 0.997604, loss: 0.001729, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17908 s, avg_samples: 56.0, ips: 312.71674 samples/s, eta: 22:38:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:34:01] ppocr INFO: epoch: [346/500], global_step: 124900, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.934831, exp_rate<=1 : 0.973827, exp_rate<=2 : 0.993539, exp_rate<=3 : 0.997687, loss: 0.002080, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18080 s, avg_samples: 56.0, ips: 309.74085 samples/s, eta: 22:38:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:34:21] ppocr INFO: epoch: [346/500], global_step: 125000, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.933821, exp_rate<=1 : 0.973543, exp_rate<=2 : 0.993511, exp_rate<=3 : 0.997698, loss: 0.002014, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20174 s, avg_samples: 56.0, ips: 277.58394 samples/s, eta: 22:38:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:34:41] ppocr INFO: epoch: [346/500], global_step: 125100, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.933202, exp_rate<=1 : 0.973337, exp_rate<=2 : 0.993464, exp_rate<=3 : 0.997730, loss: 0.001820, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19705 s, avg_samples: 56.0, ips: 284.19636 samples/s, eta: 22:37:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:34:59] ppocr INFO: epoch: [346/500], global_step: 125200, lr: 0.000100, edit distance: 0.001407, exp_rate: 0.933348, exp_rate<=1 : 0.973547, exp_rate<=2 : 0.993528, exp_rate<=3 : 0.997728, loss: 0.002045, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18454 s, avg_samples: 56.0, ips: 303.45639 samples/s, eta: 22:37:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:35:19] ppocr INFO: epoch: [346/500], global_step: 125300, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.933189, exp_rate<=1 : 0.973290, exp_rate<=2 : 0.993523, exp_rate<=3 : 0.997712, loss: 0.001904, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19592 s, avg_samples: 56.0, ips: 285.82499 samples/s, eta: 22:37:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:35:37] ppocr INFO: epoch: [346/500], global_step: 125400, lr: 0.000100, edit distance: 0.001416, exp_rate: 0.932980, exp_rate<=1 : 0.973208, exp_rate<=2 : 0.993562, exp_rate<=3 : 0.997762, loss: 0.002047, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18140 s, avg_samples: 56.0, ips: 308.70411 samples/s, eta: 22:36:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:35:57] ppocr INFO: epoch: [346/500], global_step: 125500, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.932666, exp_rate<=1 : 0.973121, exp_rate<=2 : 0.993583, exp_rate<=3 : 0.997783, loss: 0.001829, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19995 s, avg_samples: 56.0, ips: 280.07272 samples/s, eta: 22:36:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:36:15] ppocr INFO: epoch: [346/500], global_step: 125600, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.933029, exp_rate<=1 : 0.973448, exp_rate<=2 : 0.993666, exp_rate<=3 : 0.997768, loss: 0.001738, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18217 s, avg_samples: 56.0, ips: 307.40849 samples/s, eta: 22:36:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:36:35] ppocr INFO: epoch: [346/500], global_step: 125700, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.932985, exp_rate<=1 : 0.973473, exp_rate<=2 : 0.993662, exp_rate<=3 : 0.997791, loss: 0.001574, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19283 s, avg_samples: 56.0, ips: 290.41066 samples/s, eta: 22:35:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:36:55] ppocr INFO: epoch: [346/500], global_step: 125800, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.932280, exp_rate<=1 : 0.973133, exp_rate<=2 : 0.993493, exp_rate<=3 : 0.997682, loss: 0.001851, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20527 s, avg_samples: 56.0, ips: 272.81229 samples/s, eta: 22:35:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:37:14] ppocr INFO: epoch: [346/500], global_step: 125900, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.931980, exp_rate<=1 : 0.973038, exp_rate<=2 : 0.993507, exp_rate<=3 : 0.997663, loss: 0.001655, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18849 s, avg_samples: 56.0, ips: 297.09414 samples/s, eta: 22:35:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:37:34] ppocr INFO: epoch: [346/500], global_step: 126000, lr: 0.000100, edit distance: 0.001449, exp_rate: 0.931686, exp_rate<=1 : 0.972941, exp_rate<=2 : 0.993529, exp_rate<=3 : 0.997664, loss: 0.002575, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19527 s, avg_samples: 56.0, ips: 286.78451 samples/s, eta: 22:34:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:37:41] ppocr INFO: epoch: [346/500], global_step: 126040, lr: 0.000100, edit distance: 0.001448, exp_rate: 0.931528, exp_rate<=1 : 0.972895, exp_rate<=2 : 0.993545, exp_rate<=3 : 0.997685, loss: 0.001929, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.07741 s, avg_samples: 22.4, ips: 289.36579 samples/s, eta: 22:34:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:37:43] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 09:37:56] ppocr INFO: epoch: [347/500], global_step: 126100, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.938834, exp_rate<=1 : 0.974557, exp_rate<=2 : 0.994099, exp_rate<=3 : 0.996567, loss: 0.000828, avg_reader_cost: 0.02622 s, avg_batch_cost: 0.14367 s, avg_samples: 33.6, ips: 233.87487 samples/s, eta: 22:34:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:38:16] ppocr INFO: epoch: [347/500], global_step: 126200, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.935278, exp_rate<=1 : 0.973400, exp_rate<=2 : 0.992252, exp_rate<=3 : 0.995888, loss: 0.001692, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19965 s, avg_samples: 56.0, ips: 280.49705 samples/s, eta: 22:34:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:38:36] ppocr INFO: epoch: [347/500], global_step: 126300, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.932891, exp_rate<=1 : 0.973036, exp_rate<=2 : 0.992372, exp_rate<=3 : 0.996364, loss: 0.001192, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20497 s, avg_samples: 56.0, ips: 273.20791 samples/s, eta: 22:34:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:38:55] ppocr INFO: epoch: [347/500], global_step: 126400, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.935465, exp_rate<=1 : 0.975402, exp_rate<=2 : 0.993551, exp_rate<=3 : 0.996989, loss: 0.002228, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19199 s, avg_samples: 56.0, ips: 291.68071 samples/s, eta: 22:33:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:39:16] ppocr INFO: epoch: [347/500], global_step: 126500, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.934860, exp_rate<=1 : 0.975221, exp_rate<=2 : 0.993380, exp_rate<=3 : 0.997067, loss: 0.001097, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20183 s, avg_samples: 56.0, ips: 277.46678 samples/s, eta: 22:33:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:39:36] ppocr INFO: epoch: [347/500], global_step: 126600, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.934311, exp_rate<=1 : 0.975122, exp_rate<=2 : 0.993308, exp_rate<=3 : 0.997135, loss: 0.001632, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20181 s, avg_samples: 56.0, ips: 277.48438 samples/s, eta: 22:33:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:39:55] ppocr INFO: epoch: [347/500], global_step: 126700, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.935460, exp_rate<=1 : 0.975607, exp_rate<=2 : 0.993601, exp_rate<=3 : 0.997474, loss: 0.002090, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19013 s, avg_samples: 56.0, ips: 294.52885 samples/s, eta: 22:33:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:40:14] ppocr INFO: epoch: [347/500], global_step: 126800, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.935796, exp_rate<=1 : 0.975734, exp_rate<=2 : 0.993703, exp_rate<=3 : 0.997564, loss: 0.001444, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18746 s, avg_samples: 56.0, ips: 298.73216 samples/s, eta: 22:32:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:40:33] ppocr INFO: epoch: [347/500], global_step: 126900, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.935966, exp_rate<=1 : 0.975656, exp_rate<=2 : 0.993700, exp_rate<=3 : 0.997531, loss: 0.001486, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19106 s, avg_samples: 56.0, ips: 293.10117 samples/s, eta: 22:32:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:40:51] ppocr INFO: epoch: [347/500], global_step: 127000, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.936612, exp_rate<=1 : 0.975498, exp_rate<=2 : 0.993865, exp_rate<=3 : 0.997689, loss: 0.001245, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18385 s, avg_samples: 56.0, ips: 304.60096 samples/s, eta: 22:32:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:41:11] ppocr INFO: epoch: [347/500], global_step: 127100, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.935920, exp_rate<=1 : 0.975132, exp_rate<=2 : 0.993963, exp_rate<=3 : 0.997732, loss: 0.001835, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19853 s, avg_samples: 56.0, ips: 282.07738 samples/s, eta: 22:31:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:41:31] ppocr INFO: epoch: [347/500], global_step: 127200, lr: 0.000100, edit distance: 0.001377, exp_rate: 0.935017, exp_rate<=1 : 0.974635, exp_rate<=2 : 0.993849, exp_rate<=3 : 0.997691, loss: 0.001965, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19610 s, avg_samples: 56.0, ips: 285.56452 samples/s, eta: 22:31:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:41:51] ppocr INFO: epoch: [347/500], global_step: 127300, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.934502, exp_rate<=1 : 0.974476, exp_rate<=2 : 0.993962, exp_rate<=3 : 0.997729, loss: 0.001220, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20447 s, avg_samples: 56.0, ips: 273.88517 samples/s, eta: 22:31:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:42:11] ppocr INFO: epoch: [347/500], global_step: 127400, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.934708, exp_rate<=1 : 0.974628, exp_rate<=2 : 0.993986, exp_rate<=3 : 0.997749, loss: 0.001856, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19400 s, avg_samples: 56.0, ips: 288.66272 samples/s, eta: 22:30:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:42:29] ppocr INFO: epoch: [347/500], global_step: 127500, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.934798, exp_rate<=1 : 0.974481, exp_rate<=2 : 0.993968, exp_rate<=3 : 0.997786, loss: 0.001431, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18497 s, avg_samples: 56.0, ips: 302.75237 samples/s, eta: 22:30:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:42:48] ppocr INFO: epoch: [347/500], global_step: 127600, lr: 0.000100, edit distance: 0.001377, exp_rate: 0.934717, exp_rate<=1 : 0.974331, exp_rate<=2 : 0.994013, exp_rate<=3 : 0.997823, loss: 0.001796, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19266 s, avg_samples: 56.0, ips: 290.66589 samples/s, eta: 22:30:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:43:08] ppocr INFO: epoch: [347/500], global_step: 127700, lr: 0.000100, edit distance: 0.001386, exp_rate: 0.934318, exp_rate<=1 : 0.974229, exp_rate<=2 : 0.994002, exp_rate<=3 : 0.997831, loss: 0.001626, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19135 s, avg_samples: 56.0, ips: 292.66337 samples/s, eta: 22:29:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:43:25] ppocr INFO: epoch: [347/500], global_step: 127800, lr: 0.000100, edit distance: 0.001400, exp_rate: 0.934493, exp_rate<=1 : 0.974444, exp_rate<=2 : 0.994082, exp_rate<=3 : 0.997881, loss: 0.001806, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17608 s, avg_samples: 56.0, ips: 318.04368 samples/s, eta: 22:29:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:43:44] ppocr INFO: epoch: [347/500], global_step: 127900, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.933684, exp_rate<=1 : 0.974165, exp_rate<=2 : 0.994019, exp_rate<=3 : 0.997889, loss: 0.001810, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19158 s, avg_samples: 56.0, ips: 292.30057 samples/s, eta: 22:29:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:44:03] ppocr INFO: epoch: [347/500], global_step: 128000, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.934012, exp_rate<=1 : 0.974340, exp_rate<=2 : 0.994086, exp_rate<=3 : 0.997931, loss: 0.001378, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18303 s, avg_samples: 56.0, ips: 305.95818 samples/s, eta: 22:28:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:44:23] ppocr INFO: epoch: [347/500], global_step: 128100, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.933256, exp_rate<=1 : 0.973766, exp_rate<=2 : 0.993793, exp_rate<=3 : 0.997748, loss: 0.001403, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20362 s, avg_samples: 56.0, ips: 275.02795 samples/s, eta: 22:28:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:44:42] ppocr INFO: epoch: [347/500], global_step: 128200, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.933031, exp_rate<=1 : 0.973523, exp_rate<=2 : 0.993573, exp_rate<=3 : 0.997607, loss: 0.001597, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18665 s, avg_samples: 56.0, ips: 300.02779 samples/s, eta: 22:28:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:44:59] ppocr INFO: epoch: [347/500], global_step: 128300, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.933135, exp_rate<=1 : 0.973638, exp_rate<=2 : 0.993622, exp_rate<=3 : 0.997593, loss: 0.001847, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17764 s, avg_samples: 56.0, ips: 315.24699 samples/s, eta: 22:27:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:45:17] ppocr INFO: epoch: [347/500], global_step: 128400, lr: 0.000100, edit distance: 0.001447, exp_rate: 0.933283, exp_rate<=1 : 0.973617, exp_rate<=2 : 0.993586, exp_rate<=3 : 0.997535, loss: 0.002237, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17829 s, avg_samples: 56.0, ips: 314.09533 samples/s, eta: 22:27:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:45:36] ppocr INFO: epoch: [347/500], global_step: 128500, lr: 0.000100, edit distance: 0.001451, exp_rate: 0.932943, exp_rate<=1 : 0.973586, exp_rate<=2 : 0.993532, exp_rate<=3 : 0.997538, loss: 0.001799, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18916 s, avg_samples: 56.0, ips: 296.04236 samples/s, eta: 22:26:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:45:56] ppocr INFO: epoch: [347/500], global_step: 128600, lr: 0.000100, edit distance: 0.001457, exp_rate: 0.932749, exp_rate<=1 : 0.973494, exp_rate<=2 : 0.993429, exp_rate<=3 : 0.997487, loss: 0.001367, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19291 s, avg_samples: 56.0, ips: 290.28786 samples/s, eta: 22:26:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:46:15] ppocr INFO: epoch: [347/500], global_step: 128700, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.932518, exp_rate<=1 : 0.973450, exp_rate<=2 : 0.993463, exp_rate<=3 : 0.997517, loss: 0.001747, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19759 s, avg_samples: 56.0, ips: 283.40972 samples/s, eta: 22:26:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:46:30] ppocr INFO: epoch: [347/500], global_step: 128780, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.932471, exp_rate<=1 : 0.973425, exp_rate<=2 : 0.993482, exp_rate<=3 : 0.997546, loss: 0.001917, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.14757 s, avg_samples: 44.8, ips: 303.59307 samples/s, eta: 22:26:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:46:32] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 09:46:36] ppocr INFO: epoch: [348/500], global_step: 128800, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.949838, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.994463, exp_rate<=3 : 1.000000, loss: 0.001375, avg_reader_cost: 0.02679 s, avg_batch_cost: 0.06021 s, avg_samples: 11.2, ips: 186.01653 samples/s, eta: 22:26:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:46:55] ppocr INFO: epoch: [348/500], global_step: 128900, lr: 0.000100, edit distance: 0.001447, exp_rate: 0.939051, exp_rate<=1 : 0.977488, exp_rate<=2 : 0.993600, exp_rate<=3 : 0.998752, loss: 0.001499, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18998 s, avg_samples: 56.0, ips: 294.77374 samples/s, eta: 22:25:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:47:14] ppocr INFO: epoch: [348/500], global_step: 129000, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.939508, exp_rate<=1 : 0.976651, exp_rate<=2 : 0.992985, exp_rate<=3 : 0.998292, loss: 0.001128, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19041 s, avg_samples: 56.0, ips: 294.10215 samples/s, eta: 22:25:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:47:33] ppocr INFO: epoch: [348/500], global_step: 129100, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.941318, exp_rate<=1 : 0.977585, exp_rate<=2 : 0.993641, exp_rate<=3 : 0.998390, loss: 0.001878, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18455 s, avg_samples: 56.0, ips: 303.44411 samples/s, eta: 22:25:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:47:52] ppocr INFO: epoch: [348/500], global_step: 129200, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.938816, exp_rate<=1 : 0.976220, exp_rate<=2 : 0.993422, exp_rate<=3 : 0.998406, loss: 0.001650, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19483 s, avg_samples: 56.0, ips: 287.43111 samples/s, eta: 22:24:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:48:11] ppocr INFO: epoch: [348/500], global_step: 129300, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.939247, exp_rate<=1 : 0.976317, exp_rate<=2 : 0.993591, exp_rate<=3 : 0.998272, loss: 0.001346, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18451 s, avg_samples: 56.0, ips: 303.50772 samples/s, eta: 22:24:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:48:30] ppocr INFO: epoch: [348/500], global_step: 129400, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.935694, exp_rate<=1 : 0.974354, exp_rate<=2 : 0.992583, exp_rate<=3 : 0.997543, loss: 0.001684, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19507 s, avg_samples: 56.0, ips: 287.07558 samples/s, eta: 22:24:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:48:51] ppocr INFO: epoch: [348/500], global_step: 129500, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.935324, exp_rate<=1 : 0.974180, exp_rate<=2 : 0.992715, exp_rate<=3 : 0.997562, loss: 0.001218, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20304 s, avg_samples: 56.0, ips: 275.80305 samples/s, eta: 22:23:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:49:10] ppocr INFO: epoch: [348/500], global_step: 129600, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.935575, exp_rate<=1 : 0.974504, exp_rate<=2 : 0.993016, exp_rate<=3 : 0.997639, loss: 0.001338, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.18980 s, avg_samples: 56.0, ips: 295.05311 samples/s, eta: 22:23:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:49:29] ppocr INFO: epoch: [348/500], global_step: 129700, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.935495, exp_rate<=1 : 0.974385, exp_rate<=2 : 0.992928, exp_rate<=3 : 0.997628, loss: 0.001201, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19780 s, avg_samples: 56.0, ips: 283.11478 samples/s, eta: 22:23:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:49:49] ppocr INFO: epoch: [348/500], global_step: 129800, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.935690, exp_rate<=1 : 0.974463, exp_rate<=2 : 0.993089, exp_rate<=3 : 0.997652, loss: 0.001759, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19597 s, avg_samples: 56.0, ips: 285.76073 samples/s, eta: 22:22:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:50:09] ppocr INFO: epoch: [348/500], global_step: 129900, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.934877, exp_rate<=1 : 0.973718, exp_rate<=2 : 0.992907, exp_rate<=3 : 0.997581, loss: 0.001408, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20033 s, avg_samples: 56.0, ips: 279.53401 samples/s, eta: 22:22:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:50:28] ppocr INFO: epoch: [348/500], global_step: 130000, lr: 0.000100, edit distance: 0.001348, exp_rate: 0.935483, exp_rate<=1 : 0.973852, exp_rate<=2 : 0.993022, exp_rate<=3 : 0.997595, loss: 0.001269, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18994 s, avg_samples: 56.0, ips: 294.82434 samples/s, eta: 22:22:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:50:48] ppocr INFO: epoch: [348/500], global_step: 130100, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.935410, exp_rate<=1 : 0.974030, exp_rate<=2 : 0.993173, exp_rate<=3 : 0.997695, loss: 0.001880, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19541 s, avg_samples: 56.0, ips: 286.58357 samples/s, eta: 22:21:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:51:07] ppocr INFO: epoch: [348/500], global_step: 130200, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.935330, exp_rate<=1 : 0.974094, exp_rate<=2 : 0.993294, exp_rate<=3 : 0.997668, loss: 0.001628, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19375 s, avg_samples: 56.0, ips: 289.02613 samples/s, eta: 22:21:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:51:27] ppocr INFO: epoch: [348/500], global_step: 130300, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.934713, exp_rate<=1 : 0.974045, exp_rate<=2 : 0.993322, exp_rate<=3 : 0.997702, loss: 0.001601, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19986 s, avg_samples: 56.0, ips: 280.19867 samples/s, eta: 22:21:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:51:46] ppocr INFO: epoch: [348/500], global_step: 130400, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.934694, exp_rate<=1 : 0.973951, exp_rate<=2 : 0.993299, exp_rate<=3 : 0.997645, loss: 0.001502, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18938 s, avg_samples: 56.0, ips: 295.69654 samples/s, eta: 22:21:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:52:04] ppocr INFO: epoch: [348/500], global_step: 130500, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.934707, exp_rate<=1 : 0.973907, exp_rate<=2 : 0.993268, exp_rate<=3 : 0.997680, loss: 0.001956, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18473 s, avg_samples: 56.0, ips: 303.14026 samples/s, eta: 22:20:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:52:23] ppocr INFO: epoch: [348/500], global_step: 130600, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.934632, exp_rate<=1 : 0.974054, exp_rate<=2 : 0.993423, exp_rate<=3 : 0.997791, loss: 0.001773, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18104 s, avg_samples: 56.0, ips: 309.33209 samples/s, eta: 22:20:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:52:41] ppocr INFO: epoch: [348/500], global_step: 130700, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.934780, exp_rate<=1 : 0.974202, exp_rate<=2 : 0.993444, exp_rate<=3 : 0.997763, loss: 0.001921, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18821 s, avg_samples: 56.0, ips: 297.54139 samples/s, eta: 22:19:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:53:01] ppocr INFO: epoch: [348/500], global_step: 130800, lr: 0.000100, edit distance: 0.001384, exp_rate: 0.934811, exp_rate<=1 : 0.974159, exp_rate<=2 : 0.993380, exp_rate<=3 : 0.997724, loss: 0.002008, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19356 s, avg_samples: 56.0, ips: 289.31532 samples/s, eta: 22:19:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:53:20] ppocr INFO: epoch: [348/500], global_step: 130900, lr: 0.000100, edit distance: 0.001390, exp_rate: 0.934455, exp_rate<=1 : 0.974134, exp_rate<=2 : 0.993360, exp_rate<=3 : 0.997724, loss: 0.001767, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18839 s, avg_samples: 56.0, ips: 297.26013 samples/s, eta: 22:19:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:53:38] ppocr INFO: epoch: [348/500], global_step: 131000, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.934167, exp_rate<=1 : 0.973926, exp_rate<=2 : 0.993338, exp_rate<=3 : 0.997767, loss: 0.001807, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18653 s, avg_samples: 56.0, ips: 300.22517 samples/s, eta: 22:18:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:53:57] ppocr INFO: epoch: [348/500], global_step: 131100, lr: 0.000100, edit distance: 0.001431, exp_rate: 0.933701, exp_rate<=1 : 0.973806, exp_rate<=2 : 0.993321, exp_rate<=3 : 0.997762, loss: 0.003236, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18661 s, avg_samples: 56.0, ips: 300.08740 samples/s, eta: 22:18:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:54:16] ppocr INFO: epoch: [348/500], global_step: 131200, lr: 0.000100, edit distance: 0.001447, exp_rate: 0.933073, exp_rate<=1 : 0.973451, exp_rate<=2 : 0.993279, exp_rate<=3 : 0.997730, loss: 0.001405, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18662 s, avg_samples: 56.0, ips: 300.07853 samples/s, eta: 22:18:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:54:35] ppocr INFO: epoch: [348/500], global_step: 131300, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.933035, exp_rate<=1 : 0.973476, exp_rate<=2 : 0.993307, exp_rate<=3 : 0.997767, loss: 0.001118, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19381 s, avg_samples: 56.0, ips: 288.94017 samples/s, eta: 22:17:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:54:54] ppocr INFO: epoch: [348/500], global_step: 131400, lr: 0.000100, edit distance: 0.001458, exp_rate: 0.932855, exp_rate<=1 : 0.973392, exp_rate<=2 : 0.993339, exp_rate<=3 : 0.997804, loss: 0.002653, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19097 s, avg_samples: 56.0, ips: 293.23951 samples/s, eta: 22:17:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:55:14] ppocr INFO: epoch: [348/500], global_step: 131500, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.932189, exp_rate<=1 : 0.973102, exp_rate<=2 : 0.993286, exp_rate<=3 : 0.997806, loss: 0.001154, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20099 s, avg_samples: 56.0, ips: 278.62297 samples/s, eta: 22:17:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:55:19] ppocr INFO: epoch: [348/500], global_step: 131520, lr: 0.000100, edit distance: 0.001470, exp_rate: 0.932215, exp_rate<=1 : 0.973121, exp_rate<=2 : 0.993295, exp_rate<=3 : 0.997801, loss: 0.002230, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.04314 s, avg_samples: 11.2, ips: 259.63202 samples/s, eta: 22:17:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:55:21] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 09:55:37] ppocr INFO: epoch: [349/500], global_step: 131600, lr: 0.000100, edit distance: 0.001537, exp_rate: 0.942073, exp_rate<=1 : 0.978174, exp_rate<=2 : 0.993304, exp_rate<=3 : 0.996242, loss: 0.001392, avg_reader_cost: 0.02577 s, avg_batch_cost: 0.18201 s, avg_samples: 44.8, ips: 246.14400 samples/s, eta: 22:17:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:55:56] ppocr INFO: epoch: [349/500], global_step: 131700, lr: 0.000100, edit distance: 0.001277, exp_rate: 0.948523, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.995668, exp_rate<=3 : 0.998140, loss: 0.001098, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18838 s, avg_samples: 56.0, ips: 297.27807 samples/s, eta: 22:16:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:56:15] ppocr INFO: epoch: [349/500], global_step: 131800, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.944104, exp_rate<=1 : 0.978994, exp_rate<=2 : 0.994945, exp_rate<=3 : 0.997780, loss: 0.001855, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19533 s, avg_samples: 56.0, ips: 286.68883 samples/s, eta: 22:16:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:56:35] ppocr INFO: epoch: [349/500], global_step: 131900, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.943926, exp_rate<=1 : 0.978931, exp_rate<=2 : 0.994550, exp_rate<=3 : 0.997786, loss: 0.001447, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19474 s, avg_samples: 56.0, ips: 287.56427 samples/s, eta: 22:16:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:56:55] ppocr INFO: epoch: [349/500], global_step: 132000, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.941859, exp_rate<=1 : 0.977346, exp_rate<=2 : 0.994253, exp_rate<=3 : 0.997673, loss: 0.001738, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20130 s, avg_samples: 56.0, ips: 278.18778 samples/s, eta: 22:15:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:57:13] ppocr INFO: epoch: [349/500], global_step: 132100, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.942542, exp_rate<=1 : 0.977753, exp_rate<=2 : 0.994407, exp_rate<=3 : 0.997852, loss: 0.001718, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18193 s, avg_samples: 56.0, ips: 307.80978 samples/s, eta: 22:15:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:57:32] ppocr INFO: epoch: [349/500], global_step: 132200, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.942234, exp_rate<=1 : 0.977802, exp_rate<=2 : 0.994247, exp_rate<=3 : 0.997869, loss: 0.001049, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18557 s, avg_samples: 56.0, ips: 301.77389 samples/s, eta: 22:15:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:57:49] ppocr INFO: epoch: [349/500], global_step: 132300, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.942182, exp_rate<=1 : 0.977650, exp_rate<=2 : 0.994410, exp_rate<=3 : 0.997923, loss: 0.001603, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17589 s, avg_samples: 56.0, ips: 318.37454 samples/s, eta: 22:14:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:58:10] ppocr INFO: epoch: [349/500], global_step: 132400, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.941474, exp_rate<=1 : 0.977179, exp_rate<=2 : 0.994365, exp_rate<=3 : 0.997833, loss: 0.001317, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20438 s, avg_samples: 56.0, ips: 274.00246 samples/s, eta: 22:14:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:58:29] ppocr INFO: epoch: [349/500], global_step: 132500, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.940962, exp_rate<=1 : 0.976891, exp_rate<=2 : 0.994308, exp_rate<=3 : 0.997884, loss: 0.001527, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18980 s, avg_samples: 56.0, ips: 295.05437 samples/s, eta: 22:14:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:58:49] ppocr INFO: epoch: [349/500], global_step: 132600, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.940315, exp_rate<=1 : 0.976882, exp_rate<=2 : 0.994325, exp_rate<=3 : 0.997829, loss: 0.001352, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19727 s, avg_samples: 56.0, ips: 283.88196 samples/s, eta: 22:13:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:59:08] ppocr INFO: epoch: [349/500], global_step: 132700, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.940158, exp_rate<=1 : 0.976747, exp_rate<=2 : 0.994390, exp_rate<=3 : 0.997904, loss: 0.001240, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19787 s, avg_samples: 56.0, ips: 283.01387 samples/s, eta: 22:13:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:59:27] ppocr INFO: epoch: [349/500], global_step: 132800, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.940512, exp_rate<=1 : 0.976855, exp_rate<=2 : 0.994391, exp_rate<=3 : 0.997936, loss: 0.001824, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18516 s, avg_samples: 56.0, ips: 302.44307 samples/s, eta: 22:13:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 09:59:46] ppocr INFO: epoch: [349/500], global_step: 132900, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.940197, exp_rate<=1 : 0.976606, exp_rate<=2 : 0.994263, exp_rate<=3 : 0.997943, loss: 0.001366, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19265 s, avg_samples: 56.0, ips: 290.68362 samples/s, eta: 22:12:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:00:05] ppocr INFO: epoch: [349/500], global_step: 133000, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.939829, exp_rate<=1 : 0.976478, exp_rate<=2 : 0.994226, exp_rate<=3 : 0.997890, loss: 0.000812, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19264 s, avg_samples: 56.0, ips: 290.69965 samples/s, eta: 22:12:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:00:24] ppocr INFO: epoch: [349/500], global_step: 133100, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.940210, exp_rate<=1 : 0.976599, exp_rate<=2 : 0.994313, exp_rate<=3 : 0.997972, loss: 0.001295, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18772 s, avg_samples: 56.0, ips: 298.31245 samples/s, eta: 22:12:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:00:43] ppocr INFO: epoch: [349/500], global_step: 133200, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.939977, exp_rate<=1 : 0.976575, exp_rate<=2 : 0.994303, exp_rate<=3 : 0.998008, loss: 0.002096, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19015 s, avg_samples: 56.0, ips: 294.51000 samples/s, eta: 22:11:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:01:03] ppocr INFO: epoch: [349/500], global_step: 133300, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.939403, exp_rate<=1 : 0.976152, exp_rate<=2 : 0.994150, exp_rate<=3 : 0.997921, loss: 0.001408, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19908 s, avg_samples: 56.0, ips: 281.29335 samples/s, eta: 22:11:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:01:23] ppocr INFO: epoch: [349/500], global_step: 133400, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.939424, exp_rate<=1 : 0.976265, exp_rate<=2 : 0.994191, exp_rate<=3 : 0.997904, loss: 0.002036, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19236 s, avg_samples: 56.0, ips: 291.11968 samples/s, eta: 22:11:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:01:42] ppocr INFO: epoch: [349/500], global_step: 133500, lr: 0.000100, edit distance: 0.001387, exp_rate: 0.938724, exp_rate<=1 : 0.976109, exp_rate<=2 : 0.994202, exp_rate<=3 : 0.997856, loss: 0.001970, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19299 s, avg_samples: 56.0, ips: 290.17419 samples/s, eta: 22:10:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:02:02] ppocr INFO: epoch: [349/500], global_step: 133600, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.937648, exp_rate<=1 : 0.975599, exp_rate<=2 : 0.994012, exp_rate<=3 : 0.997809, loss: 0.001951, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20587 s, avg_samples: 56.0, ips: 272.01745 samples/s, eta: 22:10:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:02:23] ppocr INFO: epoch: [349/500], global_step: 133700, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.936720, exp_rate<=1 : 0.975137, exp_rate<=2 : 0.993819, exp_rate<=3 : 0.997756, loss: 0.002069, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20768 s, avg_samples: 56.0, ips: 269.64602 samples/s, eta: 22:10:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:02:41] ppocr INFO: epoch: [349/500], global_step: 133800, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.936806, exp_rate<=1 : 0.975269, exp_rate<=2 : 0.993896, exp_rate<=3 : 0.997798, loss: 0.001994, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17647 s, avg_samples: 56.0, ips: 317.33751 samples/s, eta: 22:10:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:02:59] ppocr INFO: epoch: [349/500], global_step: 133900, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.936656, exp_rate<=1 : 0.975220, exp_rate<=2 : 0.993933, exp_rate<=3 : 0.997832, loss: 0.001751, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18084 s, avg_samples: 56.0, ips: 309.66705 samples/s, eta: 22:09:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:03:17] ppocr INFO: epoch: [349/500], global_step: 134000, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.936549, exp_rate<=1 : 0.975135, exp_rate<=2 : 0.993958, exp_rate<=3 : 0.997827, loss: 0.001936, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18220 s, avg_samples: 56.0, ips: 307.34629 samples/s, eta: 22:09:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:03:36] ppocr INFO: epoch: [349/500], global_step: 134100, lr: 0.000100, edit distance: 0.001450, exp_rate: 0.936345, exp_rate<=1 : 0.975018, exp_rate<=2 : 0.993993, exp_rate<=3 : 0.997835, loss: 0.002296, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18712 s, avg_samples: 56.0, ips: 299.27189 samples/s, eta: 22:08:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:03:55] ppocr INFO: epoch: [349/500], global_step: 134200, lr: 0.000100, edit distance: 0.001466, exp_rate: 0.935541, exp_rate<=1 : 0.974654, exp_rate<=2 : 0.993891, exp_rate<=3 : 0.997845, loss: 0.001528, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19221 s, avg_samples: 56.0, ips: 291.34205 samples/s, eta: 22:08:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:04:07] ppocr INFO: epoch: [349/500], global_step: 134260, lr: 0.000100, edit distance: 0.001474, exp_rate: 0.935066, exp_rate<=1 : 0.974403, exp_rate<=2 : 0.993794, exp_rate<=3 : 0.997790, loss: 0.001837, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11821 s, avg_samples: 33.6, ips: 284.25168 samples/s, eta: 22:08:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:04:10] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 10:04:17] ppocr INFO: epoch: [350/500], global_step: 134300, lr: 0.000100, edit distance: 0.000938, exp_rate: 0.949357, exp_rate<=1 : 0.983766, exp_rate<=2 : 0.997402, exp_rate<=3 : 0.998701, loss: 0.001452, avg_reader_cost: 0.03045 s, avg_batch_cost: 0.10189 s, avg_samples: 22.4, ips: 219.84028 samples/s, eta: 22:08:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:04:36] ppocr INFO: epoch: [350/500], global_step: 134400, lr: 0.000100, edit distance: 0.001121, exp_rate: 0.941947, exp_rate<=1 : 0.978448, exp_rate<=2 : 0.995722, exp_rate<=3 : 0.997811, loss: 0.001566, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19025 s, avg_samples: 56.0, ips: 294.35255 samples/s, eta: 22:08:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:04:55] ppocr INFO: epoch: [350/500], global_step: 134500, lr: 0.000100, edit distance: 0.001095, exp_rate: 0.946541, exp_rate<=1 : 0.980330, exp_rate<=2 : 0.995816, exp_rate<=3 : 0.998091, loss: 0.001204, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18472 s, avg_samples: 56.0, ips: 303.16798 samples/s, eta: 22:07:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:05:13] ppocr INFO: epoch: [350/500], global_step: 134600, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.945116, exp_rate<=1 : 0.978708, exp_rate<=2 : 0.995080, exp_rate<=3 : 0.997748, loss: 0.002007, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18044 s, avg_samples: 56.0, ips: 310.34738 samples/s, eta: 22:07:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:05:31] ppocr INFO: epoch: [350/500], global_step: 134700, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.943109, exp_rate<=1 : 0.978019, exp_rate<=2 : 0.994400, exp_rate<=3 : 0.997659, loss: 0.002273, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18527 s, avg_samples: 56.0, ips: 302.25491 samples/s, eta: 22:07:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:05:50] ppocr INFO: epoch: [350/500], global_step: 134800, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.943413, exp_rate<=1 : 0.978404, exp_rate<=2 : 0.994411, exp_rate<=3 : 0.997713, loss: 0.001485, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18896 s, avg_samples: 56.0, ips: 296.35663 samples/s, eta: 22:06:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:06:10] ppocr INFO: epoch: [350/500], global_step: 134900, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.942382, exp_rate<=1 : 0.977821, exp_rate<=2 : 0.994470, exp_rate<=3 : 0.997777, loss: 0.001175, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19283 s, avg_samples: 56.0, ips: 290.40869 samples/s, eta: 22:06:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:06:29] ppocr INFO: epoch: [350/500], global_step: 135000, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.940350, exp_rate<=1 : 0.976577, exp_rate<=2 : 0.994279, exp_rate<=3 : 0.997687, loss: 0.002368, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19495 s, avg_samples: 56.0, ips: 287.24818 samples/s, eta: 22:06:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:06:50] ppocr INFO: epoch: [350/500], global_step: 135100, lr: 0.000100, edit distance: 0.001446, exp_rate: 0.937806, exp_rate<=1 : 0.975658, exp_rate<=2 : 0.993909, exp_rate<=3 : 0.997579, loss: 0.001004, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21018 s, avg_samples: 56.0, ips: 266.44197 samples/s, eta: 22:05:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:07:10] ppocr INFO: epoch: [350/500], global_step: 135200, lr: 0.000100, edit distance: 0.001448, exp_rate: 0.937212, exp_rate<=1 : 0.975323, exp_rate<=2 : 0.993869, exp_rate<=3 : 0.997597, loss: 0.001929, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19496 s, avg_samples: 56.0, ips: 287.23364 samples/s, eta: 22:05:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:07:29] ppocr INFO: epoch: [350/500], global_step: 135300, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.936901, exp_rate<=1 : 0.975052, exp_rate<=2 : 0.993743, exp_rate<=3 : 0.997596, loss: 0.001671, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19448 s, avg_samples: 56.0, ips: 287.94201 samples/s, eta: 22:05:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:07:47] ppocr INFO: epoch: [350/500], global_step: 135400, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.936794, exp_rate<=1 : 0.974922, exp_rate<=2 : 0.993774, exp_rate<=3 : 0.997695, loss: 0.001322, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17939 s, avg_samples: 56.0, ips: 312.17493 samples/s, eta: 22:04:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:08:08] ppocr INFO: epoch: [350/500], global_step: 135500, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.936136, exp_rate<=1 : 0.974356, exp_rate<=2 : 0.993460, exp_rate<=3 : 0.997518, loss: 0.001309, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20400 s, avg_samples: 56.0, ips: 274.51346 samples/s, eta: 22:04:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:08:27] ppocr INFO: epoch: [350/500], global_step: 135600, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.936032, exp_rate<=1 : 0.974360, exp_rate<=2 : 0.993505, exp_rate<=3 : 0.997548, loss: 0.001513, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19874 s, avg_samples: 56.0, ips: 281.77507 samples/s, eta: 22:04:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:08:46] ppocr INFO: epoch: [350/500], global_step: 135700, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.936228, exp_rate<=1 : 0.974503, exp_rate<=2 : 0.993589, exp_rate<=3 : 0.997575, loss: 0.001841, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18137 s, avg_samples: 56.0, ips: 308.76274 samples/s, eta: 22:03:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:09:06] ppocr INFO: epoch: [350/500], global_step: 135800, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.935644, exp_rate<=1 : 0.974391, exp_rate<=2 : 0.993596, exp_rate<=3 : 0.997573, loss: 0.002541, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20226 s, avg_samples: 56.0, ips: 276.87721 samples/s, eta: 22:03:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:09:24] ppocr INFO: epoch: [350/500], global_step: 135900, lr: 0.000100, edit distance: 0.001460, exp_rate: 0.935524, exp_rate<=1 : 0.974334, exp_rate<=2 : 0.993640, exp_rate<=3 : 0.997635, loss: 0.001884, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.17791 s, avg_samples: 56.0, ips: 314.76142 samples/s, eta: 22:03:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:09:43] ppocr INFO: epoch: [350/500], global_step: 136000, lr: 0.000100, edit distance: 0.001456, exp_rate: 0.935312, exp_rate<=1 : 0.974256, exp_rate<=2 : 0.993689, exp_rate<=3 : 0.997660, loss: 0.001462, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18835 s, avg_samples: 56.0, ips: 297.31315 samples/s, eta: 22:02:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:10:03] ppocr INFO: epoch: [350/500], global_step: 136100, lr: 0.000100, edit distance: 0.001464, exp_rate: 0.934834, exp_rate<=1 : 0.974012, exp_rate<=2 : 0.993591, exp_rate<=3 : 0.997606, loss: 0.002559, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20802 s, avg_samples: 56.0, ips: 269.20130 samples/s, eta: 22:02:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:10:22] ppocr INFO: epoch: [350/500], global_step: 136200, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.934006, exp_rate<=1 : 0.973894, exp_rate<=2 : 0.993653, exp_rate<=3 : 0.997660, loss: 0.001425, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18822 s, avg_samples: 56.0, ips: 297.51725 samples/s, eta: 22:02:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:10:42] ppocr INFO: epoch: [350/500], global_step: 136300, lr: 0.000100, edit distance: 0.001509, exp_rate: 0.933810, exp_rate<=1 : 0.973874, exp_rate<=2 : 0.993656, exp_rate<=3 : 0.997666, loss: 0.001875, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19846 s, avg_samples: 56.0, ips: 282.17716 samples/s, eta: 22:01:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:11:04] ppocr INFO: epoch: [350/500], global_step: 136400, lr: 0.000100, edit distance: 0.001508, exp_rate: 0.933465, exp_rate<=1 : 0.973809, exp_rate<=2 : 0.993630, exp_rate<=3 : 0.997645, loss: 0.001519, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.21633 s, avg_samples: 56.0, ips: 258.85794 samples/s, eta: 22:01:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:11:24] ppocr INFO: epoch: [350/500], global_step: 136500, lr: 0.000100, edit distance: 0.001510, exp_rate: 0.933118, exp_rate<=1 : 0.973630, exp_rate<=2 : 0.993602, exp_rate<=3 : 0.997646, loss: 0.002114, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20238 s, avg_samples: 56.0, ips: 276.70200 samples/s, eta: 22:01:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:11:42] ppocr INFO: epoch: [350/500], global_step: 136600, lr: 0.000100, edit distance: 0.001515, exp_rate: 0.933021, exp_rate<=1 : 0.973668, exp_rate<=2 : 0.993653, exp_rate<=3 : 0.997682, loss: 0.001738, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17877 s, avg_samples: 56.0, ips: 313.24910 samples/s, eta: 22:01:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:12:00] ppocr INFO: epoch: [350/500], global_step: 136700, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.933357, exp_rate<=1 : 0.973954, exp_rate<=2 : 0.993751, exp_rate<=3 : 0.997716, loss: 0.001755, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17658 s, avg_samples: 56.0, ips: 317.13925 samples/s, eta: 22:00:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:12:19] ppocr INFO: epoch: [350/500], global_step: 136800, lr: 0.000100, edit distance: 0.001516, exp_rate: 0.933240, exp_rate<=1 : 0.973989, exp_rate<=2 : 0.993783, exp_rate<=3 : 0.997714, loss: 0.001379, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19018 s, avg_samples: 56.0, ips: 294.45259 samples/s, eta: 22:00:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:12:37] ppocr INFO: epoch: [350/500], global_step: 136900, lr: 0.000100, edit distance: 0.001511, exp_rate: 0.933375, exp_rate<=1 : 0.974029, exp_rate<=2 : 0.993824, exp_rate<=3 : 0.997717, loss: 0.001716, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18108 s, avg_samples: 56.0, ips: 309.25804 samples/s, eta: 22:00:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:12:55] ppocr INFO: epoch: [350/500], global_step: 137000, lr: 0.000100, edit distance: 0.001526, exp_rate: 0.933596, exp_rate<=1 : 0.974144, exp_rate<=2 : 0.993908, exp_rate<=3 : 0.997754, loss: 0.001833, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18054 s, avg_samples: 56.0, ips: 310.18738 samples/s, eta: 21:59:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:12:57] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 10:12:58] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_350 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 10:13:16] ppocr INFO: epoch: [351/500], global_step: 137100, lr: 0.000100, edit distance: 0.001483, exp_rate: 0.947481, exp_rate<=1 : 0.981941, exp_rate<=2 : 0.995273, exp_rate<=3 : 0.999202, loss: 0.001476, avg_reader_cost: 0.03285 s, avg_batch_cost: 0.21305 s, avg_samples: 56.0, ips: 262.84862 samples/s, eta: 21:59:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:13:36] ppocr INFO: epoch: [351/500], global_step: 137200, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.944832, exp_rate<=1 : 0.979410, exp_rate<=2 : 0.994952, exp_rate<=3 : 0.998128, loss: 0.001892, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20197 s, avg_samples: 56.0, ips: 277.27389 samples/s, eta: 21:59:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:13:54] ppocr INFO: epoch: [351/500], global_step: 137300, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.944495, exp_rate<=1 : 0.979353, exp_rate<=2 : 0.995265, exp_rate<=3 : 0.998158, loss: 0.001867, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17782 s, avg_samples: 56.0, ips: 314.92367 samples/s, eta: 21:58:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:14:14] ppocr INFO: epoch: [351/500], global_step: 137400, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.942329, exp_rate<=1 : 0.978147, exp_rate<=2 : 0.994866, exp_rate<=3 : 0.998114, loss: 0.001586, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20096 s, avg_samples: 56.0, ips: 278.66175 samples/s, eta: 21:58:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:14:34] ppocr INFO: epoch: [351/500], global_step: 137500, lr: 0.000100, edit distance: 0.001343, exp_rate: 0.940063, exp_rate<=1 : 0.976771, exp_rate<=2 : 0.994526, exp_rate<=3 : 0.997779, loss: 0.001763, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19806 s, avg_samples: 56.0, ips: 282.74807 samples/s, eta: 21:58:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:14:54] ppocr INFO: epoch: [351/500], global_step: 137600, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.939163, exp_rate<=1 : 0.976715, exp_rate<=2 : 0.994538, exp_rate<=3 : 0.997785, loss: 0.001603, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19533 s, avg_samples: 56.0, ips: 286.69262 samples/s, eta: 21:57:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:15:14] ppocr INFO: epoch: [351/500], global_step: 137700, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.938062, exp_rate<=1 : 0.976234, exp_rate<=2 : 0.994336, exp_rate<=3 : 0.997729, loss: 0.000929, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20028 s, avg_samples: 56.0, ips: 279.61279 samples/s, eta: 21:57:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:15:34] ppocr INFO: epoch: [351/500], global_step: 137800, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.938798, exp_rate<=1 : 0.976459, exp_rate<=2 : 0.994191, exp_rate<=3 : 0.997764, loss: 0.001811, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20118 s, avg_samples: 56.0, ips: 278.36437 samples/s, eta: 21:57:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:15:54] ppocr INFO: epoch: [351/500], global_step: 137900, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.938287, exp_rate<=1 : 0.976110, exp_rate<=2 : 0.994148, exp_rate<=3 : 0.997762, loss: 0.001487, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20630 s, avg_samples: 56.0, ips: 271.44767 samples/s, eta: 21:57:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:16:14] ppocr INFO: epoch: [351/500], global_step: 138000, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.937933, exp_rate<=1 : 0.975860, exp_rate<=2 : 0.994149, exp_rate<=3 : 0.997739, loss: 0.001382, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19783 s, avg_samples: 56.0, ips: 283.06907 samples/s, eta: 21:56:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:16:34] ppocr INFO: epoch: [351/500], global_step: 138100, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.937770, exp_rate<=1 : 0.975952, exp_rate<=2 : 0.994151, exp_rate<=3 : 0.997796, loss: 0.001772, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19964 s, avg_samples: 56.0, ips: 280.50570 samples/s, eta: 21:56:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:16:54] ppocr INFO: epoch: [351/500], global_step: 138200, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.937567, exp_rate<=1 : 0.975870, exp_rate<=2 : 0.994187, exp_rate<=3 : 0.997841, loss: 0.001811, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19702 s, avg_samples: 56.0, ips: 284.23703 samples/s, eta: 21:56:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:17:13] ppocr INFO: epoch: [351/500], global_step: 138300, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.936641, exp_rate<=1 : 0.975387, exp_rate<=2 : 0.993962, exp_rate<=3 : 0.997732, loss: 0.002499, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19453 s, avg_samples: 56.0, ips: 287.86893 samples/s, eta: 21:55:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:17:32] ppocr INFO: epoch: [351/500], global_step: 138400, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.936853, exp_rate<=1 : 0.975400, exp_rate<=2 : 0.993934, exp_rate<=3 : 0.997733, loss: 0.001917, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18464 s, avg_samples: 56.0, ips: 303.28690 samples/s, eta: 21:55:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:17:50] ppocr INFO: epoch: [351/500], global_step: 138500, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.936377, exp_rate<=1 : 0.975396, exp_rate<=2 : 0.993984, exp_rate<=3 : 0.997784, loss: 0.002261, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18332 s, avg_samples: 56.0, ips: 305.47663 samples/s, eta: 21:55:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:18:09] ppocr INFO: epoch: [351/500], global_step: 138600, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.936469, exp_rate<=1 : 0.975312, exp_rate<=2 : 0.993943, exp_rate<=3 : 0.997726, loss: 0.001738, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19022 s, avg_samples: 56.0, ips: 294.38971 samples/s, eta: 21:54:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:18:28] ppocr INFO: epoch: [351/500], global_step: 138700, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.936628, exp_rate<=1 : 0.975435, exp_rate<=2 : 0.994009, exp_rate<=3 : 0.997761, loss: 0.002412, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18328 s, avg_samples: 56.0, ips: 305.54034 samples/s, eta: 21:54:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:18:47] ppocr INFO: epoch: [351/500], global_step: 138800, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.935981, exp_rate<=1 : 0.975208, exp_rate<=2 : 0.993978, exp_rate<=3 : 0.997790, loss: 0.002338, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19383 s, avg_samples: 56.0, ips: 288.91577 samples/s, eta: 21:54:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:19:07] ppocr INFO: epoch: [351/500], global_step: 138900, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.935329, exp_rate<=1 : 0.974816, exp_rate<=2 : 0.993850, exp_rate<=3 : 0.997722, loss: 0.002052, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19963 s, avg_samples: 56.0, ips: 280.52201 samples/s, eta: 21:53:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:19:27] ppocr INFO: epoch: [351/500], global_step: 139000, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.934757, exp_rate<=1 : 0.974574, exp_rate<=2 : 0.993811, exp_rate<=3 : 0.997725, loss: 0.001849, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19437 s, avg_samples: 56.0, ips: 288.11183 samples/s, eta: 21:53:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:19:45] ppocr INFO: epoch: [351/500], global_step: 139100, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.934757, exp_rate<=1 : 0.974536, exp_rate<=2 : 0.993833, exp_rate<=3 : 0.997745, loss: 0.001467, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18582 s, avg_samples: 56.0, ips: 301.36940 samples/s, eta: 21:53:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:20:04] ppocr INFO: epoch: [351/500], global_step: 139200, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.934685, exp_rate<=1 : 0.974532, exp_rate<=2 : 0.993878, exp_rate<=3 : 0.997774, loss: 0.001770, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18493 s, avg_samples: 56.0, ips: 302.81791 samples/s, eta: 21:52:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:20:21] ppocr INFO: epoch: [351/500], global_step: 139300, lr: 0.000100, edit distance: 0.001407, exp_rate: 0.934676, exp_rate<=1 : 0.974573, exp_rate<=2 : 0.993942, exp_rate<=3 : 0.997800, loss: 0.001975, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17516 s, avg_samples: 56.0, ips: 319.70095 samples/s, eta: 21:52:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:20:41] ppocr INFO: epoch: [351/500], global_step: 139400, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.934398, exp_rate<=1 : 0.974491, exp_rate<=2 : 0.993953, exp_rate<=3 : 0.997835, loss: 0.001676, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19638 s, avg_samples: 56.0, ips: 285.16185 samples/s, eta: 21:52:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:21:00] ppocr INFO: epoch: [351/500], global_step: 139500, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.934053, exp_rate<=1 : 0.974350, exp_rate<=2 : 0.993956, exp_rate<=3 : 0.997862, loss: 0.001284, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18945 s, avg_samples: 56.0, ips: 295.59781 samples/s, eta: 21:51:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:21:19] ppocr INFO: epoch: [351/500], global_step: 139600, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.933694, exp_rate<=1 : 0.974256, exp_rate<=2 : 0.993918, exp_rate<=3 : 0.997866, loss: 0.001647, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18790 s, avg_samples: 56.0, ips: 298.03003 samples/s, eta: 21:51:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:21:38] ppocr INFO: epoch: [351/500], global_step: 139700, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.933497, exp_rate<=1 : 0.974145, exp_rate<=2 : 0.993889, exp_rate<=3 : 0.997869, loss: 0.001834, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19071 s, avg_samples: 56.0, ips: 293.63723 samples/s, eta: 21:51:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:21:45] ppocr INFO: epoch: [351/500], global_step: 139740, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.933496, exp_rate<=1 : 0.974112, exp_rate<=2 : 0.993866, exp_rate<=3 : 0.997863, loss: 0.001739, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.07598 s, avg_samples: 22.4, ips: 294.81288 samples/s, eta: 21:50:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:21:47] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 10:22:00] ppocr INFO: epoch: [352/500], global_step: 139800, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.941017, exp_rate<=1 : 0.977311, exp_rate<=2 : 0.993867, exp_rate<=3 : 0.998557, loss: 0.001296, avg_reader_cost: 0.02600 s, avg_batch_cost: 0.14348 s, avg_samples: 33.6, ips: 234.18302 samples/s, eta: 21:50:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:22:17] ppocr INFO: epoch: [352/500], global_step: 139900, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.948151, exp_rate<=1 : 0.981283, exp_rate<=2 : 0.995939, exp_rate<=3 : 0.998747, loss: 0.001188, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17593 s, avg_samples: 56.0, ips: 318.31188 samples/s, eta: 21:50:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:22:36] ppocr INFO: epoch: [352/500], global_step: 140000, lr: 0.000100, edit distance: 0.001020, exp_rate: 0.949027, exp_rate<=1 : 0.981207, exp_rate<=2 : 0.995860, exp_rate<=3 : 0.998432, loss: 0.001245, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19060 s, avg_samples: 56.0, ips: 293.81628 samples/s, eta: 21:50:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:22:56] ppocr INFO: epoch: [352/500], global_step: 140100, lr: 0.000100, edit distance: 0.001078, exp_rate: 0.946378, exp_rate<=1 : 0.980687, exp_rate<=2 : 0.995686, exp_rate<=3 : 0.998262, loss: 0.001719, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19566 s, avg_samples: 56.0, ips: 286.20894 samples/s, eta: 21:49:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:23:15] ppocr INFO: epoch: [352/500], global_step: 140200, lr: 0.000100, edit distance: 0.001093, exp_rate: 0.945949, exp_rate<=1 : 0.980292, exp_rate<=2 : 0.995526, exp_rate<=3 : 0.998382, loss: 0.001218, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19367 s, avg_samples: 56.0, ips: 289.15375 samples/s, eta: 21:49:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:23:34] ppocr INFO: epoch: [352/500], global_step: 140300, lr: 0.000100, edit distance: 0.001115, exp_rate: 0.945661, exp_rate<=1 : 0.980226, exp_rate<=2 : 0.995757, exp_rate<=3 : 0.998488, loss: 0.001349, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18534 s, avg_samples: 56.0, ips: 302.14782 samples/s, eta: 21:49:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:23:54] ppocr INFO: epoch: [352/500], global_step: 140400, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.945043, exp_rate<=1 : 0.979887, exp_rate<=2 : 0.995522, exp_rate<=3 : 0.998309, loss: 0.001358, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19591 s, avg_samples: 56.0, ips: 285.84982 samples/s, eta: 21:48:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:24:13] ppocr INFO: epoch: [352/500], global_step: 140500, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.944241, exp_rate<=1 : 0.979763, exp_rate<=2 : 0.995598, exp_rate<=3 : 0.998382, loss: 0.001640, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19605 s, avg_samples: 56.0, ips: 285.64814 samples/s, eta: 21:48:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:24:32] ppocr INFO: epoch: [352/500], global_step: 140600, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.943698, exp_rate<=1 : 0.979465, exp_rate<=2 : 0.995693, exp_rate<=3 : 0.998418, loss: 0.001731, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18790 s, avg_samples: 56.0, ips: 298.03249 samples/s, eta: 21:48:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:24:51] ppocr INFO: epoch: [352/500], global_step: 140700, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.941656, exp_rate<=1 : 0.978357, exp_rate<=2 : 0.995531, exp_rate<=3 : 0.998392, loss: 0.001370, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18932 s, avg_samples: 56.0, ips: 295.79711 samples/s, eta: 21:47:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:25:10] ppocr INFO: epoch: [352/500], global_step: 140800, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.941076, exp_rate<=1 : 0.978224, exp_rate<=2 : 0.995483, exp_rate<=3 : 0.998342, loss: 0.002335, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18678 s, avg_samples: 56.0, ips: 299.82192 samples/s, eta: 21:47:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:25:30] ppocr INFO: epoch: [352/500], global_step: 140900, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.939315, exp_rate<=1 : 0.977435, exp_rate<=2 : 0.995270, exp_rate<=3 : 0.998217, loss: 0.001943, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20065 s, avg_samples: 56.0, ips: 279.09258 samples/s, eta: 21:47:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:25:48] ppocr INFO: epoch: [352/500], global_step: 141000, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.938548, exp_rate<=1 : 0.977006, exp_rate<=2 : 0.995092, exp_rate<=3 : 0.998188, loss: 0.001357, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18349 s, avg_samples: 56.0, ips: 305.19059 samples/s, eta: 21:46:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:26:06] ppocr INFO: epoch: [352/500], global_step: 141100, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.938734, exp_rate<=1 : 0.977437, exp_rate<=2 : 0.995249, exp_rate<=3 : 0.998261, loss: 0.002173, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18035 s, avg_samples: 56.0, ips: 310.51197 samples/s, eta: 21:46:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:26:26] ppocr INFO: epoch: [352/500], global_step: 141200, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.938043, exp_rate<=1 : 0.977274, exp_rate<=2 : 0.995193, exp_rate<=3 : 0.998248, loss: 0.002082, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19916 s, avg_samples: 56.0, ips: 281.18777 samples/s, eta: 21:46:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:26:44] ppocr INFO: epoch: [352/500], global_step: 141300, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.936954, exp_rate<=1 : 0.976675, exp_rate<=2 : 0.995125, exp_rate<=3 : 0.998251, loss: 0.001753, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18178 s, avg_samples: 56.0, ips: 308.06513 samples/s, eta: 21:45:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:27:04] ppocr INFO: epoch: [352/500], global_step: 141400, lr: 0.000100, edit distance: 0.001453, exp_rate: 0.935923, exp_rate<=1 : 0.976097, exp_rate<=2 : 0.995019, exp_rate<=3 : 0.998238, loss: 0.002033, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19316 s, avg_samples: 56.0, ips: 289.91336 samples/s, eta: 21:45:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:27:25] ppocr INFO: epoch: [352/500], global_step: 141500, lr: 0.000100, edit distance: 0.001475, exp_rate: 0.934485, exp_rate<=1 : 0.975246, exp_rate<=2 : 0.994744, exp_rate<=3 : 0.998164, loss: 0.001646, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21351 s, avg_samples: 56.0, ips: 262.28799 samples/s, eta: 21:45:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:27:45] ppocr INFO: epoch: [352/500], global_step: 141600, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.933764, exp_rate<=1 : 0.974899, exp_rate<=2 : 0.994621, exp_rate<=3 : 0.998109, loss: 0.001605, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19513 s, avg_samples: 56.0, ips: 286.99172 samples/s, eta: 21:44:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:28:04] ppocr INFO: epoch: [352/500], global_step: 141700, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.933556, exp_rate<=1 : 0.974865, exp_rate<=2 : 0.994624, exp_rate<=3 : 0.998115, loss: 0.001767, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19218 s, avg_samples: 56.0, ips: 291.39850 samples/s, eta: 21:44:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:28:22] ppocr INFO: epoch: [352/500], global_step: 141800, lr: 0.000100, edit distance: 0.001490, exp_rate: 0.933274, exp_rate<=1 : 0.974656, exp_rate<=2 : 0.994469, exp_rate<=3 : 0.998042, loss: 0.001516, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18363 s, avg_samples: 56.0, ips: 304.96256 samples/s, eta: 21:44:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:28:40] ppocr INFO: epoch: [352/500], global_step: 141900, lr: 0.000100, edit distance: 0.001485, exp_rate: 0.933430, exp_rate<=1 : 0.974727, exp_rate<=2 : 0.994469, exp_rate<=3 : 0.998016, loss: 0.001837, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18243 s, avg_samples: 56.0, ips: 306.97339 samples/s, eta: 21:43:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:29:00] ppocr INFO: epoch: [352/500], global_step: 142000, lr: 0.000100, edit distance: 0.001483, exp_rate: 0.933267, exp_rate<=1 : 0.974679, exp_rate<=2 : 0.994442, exp_rate<=3 : 0.998008, loss: 0.001766, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19846 s, avg_samples: 56.0, ips: 282.17627 samples/s, eta: 21:43:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:29:19] ppocr INFO: epoch: [352/500], global_step: 142100, lr: 0.000100, edit distance: 0.001495, exp_rate: 0.933239, exp_rate<=1 : 0.974625, exp_rate<=2 : 0.994377, exp_rate<=3 : 0.997969, loss: 0.002138, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18469 s, avg_samples: 56.0, ips: 303.21081 samples/s, eta: 21:43:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:29:39] ppocr INFO: epoch: [352/500], global_step: 142200, lr: 0.000100, edit distance: 0.001499, exp_rate: 0.933086, exp_rate<=1 : 0.974598, exp_rate<=2 : 0.994250, exp_rate<=3 : 0.997908, loss: 0.001721, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20044 s, avg_samples: 56.0, ips: 279.38285 samples/s, eta: 21:42:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:29:56] ppocr INFO: epoch: [352/500], global_step: 142300, lr: 0.000100, edit distance: 0.001494, exp_rate: 0.933525, exp_rate<=1 : 0.974771, exp_rate<=2 : 0.994300, exp_rate<=3 : 0.997943, loss: 0.001435, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17128 s, avg_samples: 56.0, ips: 326.94692 samples/s, eta: 21:42:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:30:16] ppocr INFO: epoch: [352/500], global_step: 142400, lr: 0.000100, edit distance: 0.001497, exp_rate: 0.933277, exp_rate<=1 : 0.974661, exp_rate<=2 : 0.994306, exp_rate<=3 : 0.997941, loss: 0.002130, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19533 s, avg_samples: 56.0, ips: 286.70053 samples/s, eta: 21:42:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:30:32] ppocr INFO: epoch: [352/500], global_step: 142480, lr: 0.000100, edit distance: 0.001498, exp_rate: 0.933155, exp_rate<=1 : 0.974558, exp_rate<=2 : 0.994328, exp_rate<=3 : 0.997973, loss: 0.001599, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.15969 s, avg_samples: 44.8, ips: 280.54216 samples/s, eta: 21:42:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:30:33] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 10:30:37] ppocr INFO: epoch: [353/500], global_step: 142500, lr: 0.000100, edit distance: 0.000923, exp_rate: 0.966901, exp_rate<=1 : 0.987753, exp_rate<=2 : 0.998847, exp_rate<=3 : 1.000000, loss: 0.001186, avg_reader_cost: 0.02413 s, avg_batch_cost: 0.05781 s, avg_samples: 11.2, ips: 193.73208 samples/s, eta: 21:42:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:30:57] ppocr INFO: epoch: [353/500], global_step: 142600, lr: 0.000100, edit distance: 0.001065, exp_rate: 0.950040, exp_rate<=1 : 0.982878, exp_rate<=2 : 0.996912, exp_rate<=3 : 0.998799, loss: 0.001393, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19661 s, avg_samples: 56.0, ips: 284.82373 samples/s, eta: 21:41:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:31:18] ppocr INFO: epoch: [353/500], global_step: 142700, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.944128, exp_rate<=1 : 0.979543, exp_rate<=2 : 0.995504, exp_rate<=3 : 0.998210, loss: 0.001360, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20569 s, avg_samples: 56.0, ips: 272.25122 samples/s, eta: 21:41:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:31:36] ppocr INFO: epoch: [353/500], global_step: 142800, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.944902, exp_rate<=1 : 0.979674, exp_rate<=2 : 0.995701, exp_rate<=3 : 0.998408, loss: 0.001346, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18566 s, avg_samples: 56.0, ips: 301.63139 samples/s, eta: 21:41:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:31:56] ppocr INFO: epoch: [353/500], global_step: 142900, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.944180, exp_rate<=1 : 0.979011, exp_rate<=2 : 0.995341, exp_rate<=3 : 0.998361, loss: 0.001426, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19443 s, avg_samples: 56.0, ips: 288.02038 samples/s, eta: 21:40:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:32:16] ppocr INFO: epoch: [353/500], global_step: 143000, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.942861, exp_rate<=1 : 0.978554, exp_rate<=2 : 0.994953, exp_rate<=3 : 0.998317, loss: 0.001928, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19906 s, avg_samples: 56.0, ips: 281.31533 samples/s, eta: 21:40:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:32:35] ppocr INFO: epoch: [353/500], global_step: 143100, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.941765, exp_rate<=1 : 0.978008, exp_rate<=2 : 0.995016, exp_rate<=3 : 0.998314, loss: 0.001747, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18972 s, avg_samples: 56.0, ips: 295.17167 samples/s, eta: 21:40:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:32:54] ppocr INFO: epoch: [353/500], global_step: 143200, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.940808, exp_rate<=1 : 0.977537, exp_rate<=2 : 0.994637, exp_rate<=3 : 0.998288, loss: 0.001465, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19141 s, avg_samples: 56.0, ips: 292.55951 samples/s, eta: 21:39:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:33:13] ppocr INFO: epoch: [353/500], global_step: 143300, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.940265, exp_rate<=1 : 0.977148, exp_rate<=2 : 0.994565, exp_rate<=3 : 0.998180, loss: 0.001675, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19648 s, avg_samples: 56.0, ips: 285.01246 samples/s, eta: 21:39:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:33:31] ppocr INFO: epoch: [353/500], global_step: 143400, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.941000, exp_rate<=1 : 0.977642, exp_rate<=2 : 0.994748, exp_rate<=3 : 0.998254, loss: 0.001487, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17413 s, avg_samples: 56.0, ips: 321.60642 samples/s, eta: 21:39:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:33:49] ppocr INFO: epoch: [353/500], global_step: 143500, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.941118, exp_rate<=1 : 0.977868, exp_rate<=2 : 0.994851, exp_rate<=3 : 0.998339, loss: 0.001291, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18591 s, avg_samples: 56.0, ips: 301.21306 samples/s, eta: 21:38:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:34:10] ppocr INFO: epoch: [353/500], global_step: 143600, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.940275, exp_rate<=1 : 0.977475, exp_rate<=2 : 0.994716, exp_rate<=3 : 0.998304, loss: 0.001878, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20422 s, avg_samples: 56.0, ips: 274.21090 samples/s, eta: 21:38:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:34:28] ppocr INFO: epoch: [353/500], global_step: 143700, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.940375, exp_rate<=1 : 0.977504, exp_rate<=2 : 0.994829, exp_rate<=3 : 0.998365, loss: 0.001643, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17901 s, avg_samples: 56.0, ips: 312.83260 samples/s, eta: 21:38:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:34:48] ppocr INFO: epoch: [353/500], global_step: 143800, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.939671, exp_rate<=1 : 0.977034, exp_rate<=2 : 0.994666, exp_rate<=3 : 0.998304, loss: 0.001326, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20306 s, avg_samples: 56.0, ips: 275.77610 samples/s, eta: 21:37:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:35:06] ppocr INFO: epoch: [353/500], global_step: 143900, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.939179, exp_rate<=1 : 0.977104, exp_rate<=2 : 0.994764, exp_rate<=3 : 0.998347, loss: 0.001780, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17595 s, avg_samples: 56.0, ips: 318.26575 samples/s, eta: 21:37:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:35:24] ppocr INFO: epoch: [353/500], global_step: 144000, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.939352, exp_rate<=1 : 0.976989, exp_rate<=2 : 0.994771, exp_rate<=3 : 0.998340, loss: 0.001448, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18225 s, avg_samples: 56.0, ips: 307.26806 samples/s, eta: 21:37:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:35:45] ppocr INFO: epoch: [353/500], global_step: 144100, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.938559, exp_rate<=1 : 0.976633, exp_rate<=2 : 0.994660, exp_rate<=3 : 0.998266, loss: 0.001792, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20841 s, avg_samples: 56.0, ips: 268.70511 samples/s, eta: 21:36:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:36:04] ppocr INFO: epoch: [353/500], global_step: 144200, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.938344, exp_rate<=1 : 0.976471, exp_rate<=2 : 0.994621, exp_rate<=3 : 0.998230, loss: 0.001601, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18999 s, avg_samples: 56.0, ips: 294.74683 samples/s, eta: 21:36:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:36:23] ppocr INFO: epoch: [353/500], global_step: 144300, lr: 0.000100, edit distance: 0.001383, exp_rate: 0.938128, exp_rate<=1 : 0.976376, exp_rate<=2 : 0.994608, exp_rate<=3 : 0.998205, loss: 0.001609, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18598 s, avg_samples: 56.0, ips: 301.10705 samples/s, eta: 21:36:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:36:41] ppocr INFO: epoch: [353/500], global_step: 144400, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.937582, exp_rate<=1 : 0.976161, exp_rate<=2 : 0.994502, exp_rate<=3 : 0.998153, loss: 0.001864, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18835 s, avg_samples: 56.0, ips: 297.31457 samples/s, eta: 21:35:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:37:00] ppocr INFO: epoch: [353/500], global_step: 144500, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.937533, exp_rate<=1 : 0.976131, exp_rate<=2 : 0.994459, exp_rate<=3 : 0.998128, loss: 0.001874, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18641 s, avg_samples: 56.0, ips: 300.40774 samples/s, eta: 21:35:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:37:19] ppocr INFO: epoch: [353/500], global_step: 144600, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.937230, exp_rate<=1 : 0.976010, exp_rate<=2 : 0.994456, exp_rate<=3 : 0.998135, loss: 0.001573, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18855 s, avg_samples: 56.0, ips: 297.00587 samples/s, eta: 21:35:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:37:39] ppocr INFO: epoch: [353/500], global_step: 144700, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.936226, exp_rate<=1 : 0.975637, exp_rate<=2 : 0.994330, exp_rate<=3 : 0.998080, loss: 0.001640, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20286 s, avg_samples: 56.0, ips: 276.05047 samples/s, eta: 21:34:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:37:59] ppocr INFO: epoch: [353/500], global_step: 144800, lr: 0.000100, edit distance: 0.001419, exp_rate: 0.935617, exp_rate<=1 : 0.975453, exp_rate<=2 : 0.994256, exp_rate<=3 : 0.998060, loss: 0.001490, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20243 s, avg_samples: 56.0, ips: 276.63420 samples/s, eta: 21:34:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:38:18] ppocr INFO: epoch: [353/500], global_step: 144900, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.934945, exp_rate<=1 : 0.975032, exp_rate<=2 : 0.994129, exp_rate<=3 : 0.998016, loss: 0.002228, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18937 s, avg_samples: 56.0, ips: 295.72071 samples/s, eta: 21:34:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:38:37] ppocr INFO: epoch: [353/500], global_step: 145000, lr: 0.000100, edit distance: 0.001447, exp_rate: 0.934443, exp_rate<=1 : 0.974892, exp_rate<=2 : 0.994101, exp_rate<=3 : 0.998001, loss: 0.001717, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18996 s, avg_samples: 56.0, ips: 294.80482 samples/s, eta: 21:33:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:38:58] ppocr INFO: epoch: [353/500], global_step: 145100, lr: 0.000100, edit distance: 0.001457, exp_rate: 0.933834, exp_rate<=1 : 0.974642, exp_rate<=2 : 0.994026, exp_rate<=3 : 0.997994, loss: 0.002127, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20054 s, avg_samples: 56.0, ips: 279.23989 samples/s, eta: 21:33:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:39:16] ppocr INFO: epoch: [353/500], global_step: 145200, lr: 0.000100, edit distance: 0.001456, exp_rate: 0.933934, exp_rate<=1 : 0.974730, exp_rate<=2 : 0.994075, exp_rate<=3 : 0.997990, loss: 0.001716, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18019 s, avg_samples: 56.0, ips: 310.77485 samples/s, eta: 21:33:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:39:20] ppocr INFO: epoch: [353/500], global_step: 145220, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.933865, exp_rate<=1 : 0.974706, exp_rate<=2 : 0.994101, exp_rate<=3 : 0.997997, loss: 0.002326, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03966 s, avg_samples: 11.2, ips: 282.42298 samples/s, eta: 21:33:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:39:22] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 10:39:39] ppocr INFO: epoch: [354/500], global_step: 145300, lr: 0.000100, edit distance: 0.001653, exp_rate: 0.922745, exp_rate<=1 : 0.970744, exp_rate<=2 : 0.992064, exp_rate<=3 : 0.997243, loss: 0.001073, avg_reader_cost: 0.02736 s, avg_batch_cost: 0.18999 s, avg_samples: 44.8, ips: 235.80335 samples/s, eta: 21:33:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:39:57] ppocr INFO: epoch: [354/500], global_step: 145400, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.938356, exp_rate<=1 : 0.976399, exp_rate<=2 : 0.994597, exp_rate<=3 : 0.997942, loss: 0.001382, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18399 s, avg_samples: 56.0, ips: 304.36485 samples/s, eta: 21:32:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:40:17] ppocr INFO: epoch: [354/500], global_step: 145500, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.938578, exp_rate<=1 : 0.976577, exp_rate<=2 : 0.994704, exp_rate<=3 : 0.997947, loss: 0.000858, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19850 s, avg_samples: 56.0, ips: 282.11581 samples/s, eta: 21:32:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:40:36] ppocr INFO: epoch: [354/500], global_step: 145600, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.940070, exp_rate<=1 : 0.976764, exp_rate<=2 : 0.994938, exp_rate<=3 : 0.998137, loss: 0.002066, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18961 s, avg_samples: 56.0, ips: 295.33910 samples/s, eta: 21:32:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:40:55] ppocr INFO: epoch: [354/500], global_step: 145700, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.941172, exp_rate<=1 : 0.977381, exp_rate<=2 : 0.995186, exp_rate<=3 : 0.998190, loss: 0.001679, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18788 s, avg_samples: 56.0, ips: 298.06333 samples/s, eta: 21:31:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:41:13] ppocr INFO: epoch: [354/500], global_step: 145800, lr: 0.000100, edit distance: 0.001308, exp_rate: 0.940189, exp_rate<=1 : 0.977090, exp_rate<=2 : 0.995341, exp_rate<=3 : 0.998281, loss: 0.001702, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.25812 samples/s, eta: 21:31:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:41:33] ppocr INFO: epoch: [354/500], global_step: 145900, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939467, exp_rate<=1 : 0.976968, exp_rate<=2 : 0.995259, exp_rate<=3 : 0.998216, loss: 0.001115, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19113 s, avg_samples: 56.0, ips: 292.99524 samples/s, eta: 21:31:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:41:52] ppocr INFO: epoch: [354/500], global_step: 146000, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.939964, exp_rate<=1 : 0.977705, exp_rate<=2 : 0.995365, exp_rate<=3 : 0.998285, loss: 0.001744, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.35717 samples/s, eta: 21:30:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:42:11] ppocr INFO: epoch: [354/500], global_step: 146100, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.939816, exp_rate<=1 : 0.977374, exp_rate<=2 : 0.995189, exp_rate<=3 : 0.998318, loss: 0.001494, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18969 s, avg_samples: 56.0, ips: 295.21632 samples/s, eta: 21:30:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:42:31] ppocr INFO: epoch: [354/500], global_step: 146200, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.938857, exp_rate<=1 : 0.976991, exp_rate<=2 : 0.994922, exp_rate<=3 : 0.998217, loss: 0.001325, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19804 s, avg_samples: 56.0, ips: 282.77828 samples/s, eta: 21:30:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:42:51] ppocr INFO: epoch: [354/500], global_step: 146300, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.938539, exp_rate<=1 : 0.976905, exp_rate<=2 : 0.994855, exp_rate<=3 : 0.998228, loss: 0.001540, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20076 s, avg_samples: 56.0, ips: 278.94587 samples/s, eta: 21:29:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:43:10] ppocr INFO: epoch: [354/500], global_step: 146400, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.938113, exp_rate<=1 : 0.976694, exp_rate<=2 : 0.994799, exp_rate<=3 : 0.998254, loss: 0.001572, avg_reader_cost: 0.00017 s, avg_batch_cost: 0.19010 s, avg_samples: 56.0, ips: 294.57528 samples/s, eta: 21:29:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:43:30] ppocr INFO: epoch: [354/500], global_step: 146500, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.937929, exp_rate<=1 : 0.976620, exp_rate<=2 : 0.994752, exp_rate<=3 : 0.998237, loss: 0.002085, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19799 s, avg_samples: 56.0, ips: 282.83734 samples/s, eta: 21:29:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:43:48] ppocr INFO: epoch: [354/500], global_step: 146600, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.938009, exp_rate<=1 : 0.976293, exp_rate<=2 : 0.994606, exp_rate<=3 : 0.998176, loss: 0.001279, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17976 s, avg_samples: 56.0, ips: 311.52169 samples/s, eta: 21:28:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:44:06] ppocr INFO: epoch: [354/500], global_step: 146700, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.937740, exp_rate<=1 : 0.976219, exp_rate<=2 : 0.994508, exp_rate<=3 : 0.998170, loss: 0.001978, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18645 s, avg_samples: 56.0, ips: 300.34337 samples/s, eta: 21:28:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:44:26] ppocr INFO: epoch: [354/500], global_step: 146800, lr: 0.000100, edit distance: 0.001393, exp_rate: 0.936306, exp_rate<=1 : 0.975444, exp_rate<=2 : 0.994235, exp_rate<=3 : 0.998038, loss: 0.001973, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19174 s, avg_samples: 56.0, ips: 292.06712 samples/s, eta: 21:28:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:44:45] ppocr INFO: epoch: [354/500], global_step: 146900, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.935542, exp_rate<=1 : 0.975014, exp_rate<=2 : 0.994234, exp_rate<=3 : 0.997990, loss: 0.001517, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19730 s, avg_samples: 56.0, ips: 283.82711 samples/s, eta: 21:27:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:45:05] ppocr INFO: epoch: [354/500], global_step: 147000, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.934911, exp_rate<=1 : 0.974687, exp_rate<=2 : 0.994120, exp_rate<=3 : 0.997875, loss: 0.001908, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19987 s, avg_samples: 56.0, ips: 280.17696 samples/s, eta: 21:27:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:45:24] ppocr INFO: epoch: [354/500], global_step: 147100, lr: 0.000100, edit distance: 0.001441, exp_rate: 0.934424, exp_rate<=1 : 0.974432, exp_rate<=2 : 0.994003, exp_rate<=3 : 0.997804, loss: 0.002272, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18810 s, avg_samples: 56.0, ips: 297.70632 samples/s, eta: 21:27:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:45:44] ppocr INFO: epoch: [354/500], global_step: 147200, lr: 0.000100, edit distance: 0.001454, exp_rate: 0.933613, exp_rate<=1 : 0.974107, exp_rate<=2 : 0.993897, exp_rate<=3 : 0.997771, loss: 0.001834, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19469 s, avg_samples: 56.0, ips: 287.63972 samples/s, eta: 21:26:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:46:03] ppocr INFO: epoch: [354/500], global_step: 147300, lr: 0.000100, edit distance: 0.001456, exp_rate: 0.933310, exp_rate<=1 : 0.973971, exp_rate<=2 : 0.993844, exp_rate<=3 : 0.997765, loss: 0.001770, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19519 s, avg_samples: 56.0, ips: 286.89564 samples/s, eta: 21:26:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:46:23] ppocr INFO: epoch: [354/500], global_step: 147400, lr: 0.000100, edit distance: 0.001475, exp_rate: 0.932890, exp_rate<=1 : 0.973936, exp_rate<=2 : 0.993804, exp_rate<=3 : 0.997729, loss: 0.001589, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19392 s, avg_samples: 56.0, ips: 288.78358 samples/s, eta: 21:26:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:46:42] ppocr INFO: epoch: [354/500], global_step: 147500, lr: 0.000100, edit distance: 0.001482, exp_rate: 0.932410, exp_rate<=1 : 0.973751, exp_rate<=2 : 0.993767, exp_rate<=3 : 0.997743, loss: 0.001842, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19774 s, avg_samples: 56.0, ips: 283.19763 samples/s, eta: 21:26:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:47:00] ppocr INFO: epoch: [354/500], global_step: 147600, lr: 0.000100, edit distance: 0.001489, exp_rate: 0.932300, exp_rate<=1 : 0.973804, exp_rate<=2 : 0.993823, exp_rate<=3 : 0.997778, loss: 0.001816, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17257 s, avg_samples: 56.0, ips: 324.51116 samples/s, eta: 21:25:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:47:19] ppocr INFO: epoch: [354/500], global_step: 147700, lr: 0.000100, edit distance: 0.001493, exp_rate: 0.932554, exp_rate<=1 : 0.973731, exp_rate<=2 : 0.993807, exp_rate<=3 : 0.997789, loss: 0.001277, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18868 s, avg_samples: 56.0, ips: 296.79837 samples/s, eta: 21:25:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:47:39] ppocr INFO: epoch: [354/500], global_step: 147800, lr: 0.000100, edit distance: 0.001492, exp_rate: 0.932457, exp_rate<=1 : 0.973591, exp_rate<=2 : 0.993794, exp_rate<=3 : 0.997764, loss: 0.001594, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20752 s, avg_samples: 56.0, ips: 269.84898 samples/s, eta: 21:25:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:47:58] ppocr INFO: epoch: [354/500], global_step: 147900, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.932508, exp_rate<=1 : 0.973565, exp_rate<=2 : 0.993758, exp_rate<=3 : 0.997733, loss: 0.001478, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18194 s, avg_samples: 56.0, ips: 307.80125 samples/s, eta: 21:24:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:48:09] ppocr INFO: epoch: [354/500], global_step: 147960, lr: 0.000100, edit distance: 0.001490, exp_rate: 0.932696, exp_rate<=1 : 0.973687, exp_rate<=2 : 0.993792, exp_rate<=3 : 0.997757, loss: 0.000998, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11434 s, avg_samples: 33.6, ips: 293.86086 samples/s, eta: 21:24:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:48:11] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 10:48:19] ppocr INFO: epoch: [355/500], global_step: 148000, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.933010, exp_rate<=1 : 0.974944, exp_rate<=2 : 0.991485, exp_rate<=3 : 0.996486, loss: 0.001170, avg_reader_cost: 0.02735 s, avg_batch_cost: 0.10379 s, avg_samples: 22.4, ips: 215.83017 samples/s, eta: 21:24:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:48:39] ppocr INFO: epoch: [355/500], global_step: 148100, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.940477, exp_rate<=1 : 0.977356, exp_rate<=2 : 0.993617, exp_rate<=3 : 0.997812, loss: 0.000935, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19507 s, avg_samples: 56.0, ips: 287.07198 samples/s, eta: 21:24:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:48:58] ppocr INFO: epoch: [355/500], global_step: 148200, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.943756, exp_rate<=1 : 0.977533, exp_rate<=2 : 0.993647, exp_rate<=3 : 0.997986, loss: 0.001024, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18995 s, avg_samples: 56.0, ips: 294.80774 samples/s, eta: 21:23:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:49:18] ppocr INFO: epoch: [355/500], global_step: 148300, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.943426, exp_rate<=1 : 0.978083, exp_rate<=2 : 0.994261, exp_rate<=3 : 0.998168, loss: 0.001512, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19482 s, avg_samples: 56.0, ips: 287.45198 samples/s, eta: 21:23:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:49:36] ppocr INFO: epoch: [355/500], global_step: 148400, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.945140, exp_rate<=1 : 0.979023, exp_rate<=2 : 0.994277, exp_rate<=3 : 0.998092, loss: 0.001852, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18071 s, avg_samples: 56.0, ips: 309.89567 samples/s, eta: 21:23:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:49:55] ppocr INFO: epoch: [355/500], global_step: 148500, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.943818, exp_rate<=1 : 0.978598, exp_rate<=2 : 0.994548, exp_rate<=3 : 0.998194, loss: 0.001619, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19780 s, avg_samples: 56.0, ips: 283.11606 samples/s, eta: 21:22:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:50:16] ppocr INFO: epoch: [355/500], global_step: 148600, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.942223, exp_rate<=1 : 0.977795, exp_rate<=2 : 0.994339, exp_rate<=3 : 0.998068, loss: 0.000884, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20513 s, avg_samples: 56.0, ips: 272.99302 samples/s, eta: 21:22:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:50:35] ppocr INFO: epoch: [355/500], global_step: 148700, lr: 0.000100, edit distance: 0.001393, exp_rate: 0.941365, exp_rate<=1 : 0.976980, exp_rate<=2 : 0.994146, exp_rate<=3 : 0.997925, loss: 0.000881, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19112 s, avg_samples: 56.0, ips: 293.00362 samples/s, eta: 21:22:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:50:55] ppocr INFO: epoch: [355/500], global_step: 148800, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.940634, exp_rate<=1 : 0.977015, exp_rate<=2 : 0.994251, exp_rate<=3 : 0.997958, loss: 0.001456, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19528 s, avg_samples: 56.0, ips: 286.76815 samples/s, eta: 21:21:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:51:14] ppocr INFO: epoch: [355/500], global_step: 148900, lr: 0.000100, edit distance: 0.001367, exp_rate: 0.940630, exp_rate<=1 : 0.977119, exp_rate<=2 : 0.994345, exp_rate<=3 : 0.998006, loss: 0.001101, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19029 s, avg_samples: 56.0, ips: 294.29175 samples/s, eta: 21:21:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:51:32] ppocr INFO: epoch: [355/500], global_step: 149000, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.940710, exp_rate<=1 : 0.977198, exp_rate<=2 : 0.994437, exp_rate<=3 : 0.998092, loss: 0.001736, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18361 s, avg_samples: 56.0, ips: 304.98598 samples/s, eta: 21:21:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:51:51] ppocr INFO: epoch: [355/500], global_step: 149100, lr: 0.000100, edit distance: 0.001383, exp_rate: 0.940730, exp_rate<=1 : 0.977352, exp_rate<=2 : 0.994377, exp_rate<=3 : 0.998075, loss: 0.001263, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18952 s, avg_samples: 56.0, ips: 295.48118 samples/s, eta: 21:20:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:52:10] ppocr INFO: epoch: [355/500], global_step: 149200, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.940347, exp_rate<=1 : 0.977190, exp_rate<=2 : 0.994347, exp_rate<=3 : 0.998098, loss: 0.001688, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18538 s, avg_samples: 56.0, ips: 302.08863 samples/s, eta: 21:20:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:52:30] ppocr INFO: epoch: [355/500], global_step: 149300, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.938917, exp_rate<=1 : 0.976375, exp_rate<=2 : 0.994206, exp_rate<=3 : 0.998100, loss: 0.001745, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20110 s, avg_samples: 56.0, ips: 278.46780 samples/s, eta: 21:20:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:52:48] ppocr INFO: epoch: [355/500], global_step: 149400, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.938615, exp_rate<=1 : 0.976213, exp_rate<=2 : 0.994139, exp_rate<=3 : 0.998096, loss: 0.001511, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18482 s, avg_samples: 56.0, ips: 302.99988 samples/s, eta: 21:19:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:53:07] ppocr INFO: epoch: [355/500], global_step: 149500, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.938346, exp_rate<=1 : 0.976119, exp_rate<=2 : 0.994110, exp_rate<=3 : 0.998072, loss: 0.002134, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19076 s, avg_samples: 56.0, ips: 293.55848 samples/s, eta: 21:19:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:53:26] ppocr INFO: epoch: [355/500], global_step: 149600, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.938540, exp_rate<=1 : 0.976154, exp_rate<=2 : 0.994214, exp_rate<=3 : 0.998094, loss: 0.001245, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18505 s, avg_samples: 56.0, ips: 302.61572 samples/s, eta: 21:19:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:53:47] ppocr INFO: epoch: [355/500], global_step: 149700, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.937162, exp_rate<=1 : 0.975415, exp_rate<=2 : 0.994025, exp_rate<=3 : 0.998036, loss: 0.001494, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21097 s, avg_samples: 56.0, ips: 265.43957 samples/s, eta: 21:18:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:54:06] ppocr INFO: epoch: [355/500], global_step: 149800, lr: 0.000100, edit distance: 0.001414, exp_rate: 0.936664, exp_rate<=1 : 0.975177, exp_rate<=2 : 0.994049, exp_rate<=3 : 0.998088, loss: 0.002083, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18506 s, avg_samples: 56.0, ips: 302.60030 samples/s, eta: 21:18:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:54:25] ppocr INFO: epoch: [355/500], global_step: 149900, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.936290, exp_rate<=1 : 0.975043, exp_rate<=2 : 0.994028, exp_rate<=3 : 0.998045, loss: 0.001770, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19222 s, avg_samples: 56.0, ips: 291.33616 samples/s, eta: 21:18:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:54:45] ppocr INFO: epoch: [355/500], global_step: 150000, lr: 0.000100, edit distance: 0.001419, exp_rate: 0.936187, exp_rate<=1 : 0.975112, exp_rate<=2 : 0.994093, exp_rate<=3 : 0.998025, loss: 0.001353, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20174 s, avg_samples: 56.0, ips: 277.59109 samples/s, eta: 21:18:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:55:05] ppocr INFO: epoch: [355/500], global_step: 150100, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.935614, exp_rate<=1 : 0.974908, exp_rate<=2 : 0.993999, exp_rate<=3 : 0.997936, loss: 0.001769, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19703 s, avg_samples: 56.0, ips: 284.21558 samples/s, eta: 21:17:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:55:24] ppocr INFO: epoch: [355/500], global_step: 150200, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.935474, exp_rate<=1 : 0.974812, exp_rate<=2 : 0.993962, exp_rate<=3 : 0.997902, loss: 0.001490, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19593 s, avg_samples: 56.0, ips: 285.82330 samples/s, eta: 21:17:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:55:43] ppocr INFO: epoch: [355/500], global_step: 150300, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.935640, exp_rate<=1 : 0.974942, exp_rate<=2 : 0.993988, exp_rate<=3 : 0.997880, loss: 0.001438, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18775 s, avg_samples: 56.0, ips: 298.26764 samples/s, eta: 21:17:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:56:02] ppocr INFO: epoch: [355/500], global_step: 150400, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.935325, exp_rate<=1 : 0.974836, exp_rate<=2 : 0.993959, exp_rate<=3 : 0.997889, loss: 0.001866, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18736 s, avg_samples: 56.0, ips: 298.89325 samples/s, eta: 21:16:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:56:21] ppocr INFO: epoch: [355/500], global_step: 150500, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.935029, exp_rate<=1 : 0.974782, exp_rate<=2 : 0.993971, exp_rate<=3 : 0.997912, loss: 0.001449, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18907 s, avg_samples: 56.0, ips: 296.18456 samples/s, eta: 21:16:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:56:39] ppocr INFO: epoch: [355/500], global_step: 150600, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.935163, exp_rate<=1 : 0.974771, exp_rate<=2 : 0.994044, exp_rate<=3 : 0.997954, loss: 0.001534, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18623 s, avg_samples: 56.0, ips: 300.70817 samples/s, eta: 21:16:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:56:58] ppocr INFO: epoch: [355/500], global_step: 150700, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.934947, exp_rate<=1 : 0.974637, exp_rate<=2 : 0.994012, exp_rate<=3 : 0.997933, loss: 0.002027, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18639 s, avg_samples: 56.0, ips: 300.44920 samples/s, eta: 21:15:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:57:00] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 10:57:01] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_355 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 10:57:20] ppocr INFO: epoch: [356/500], global_step: 150800, lr: 0.000100, edit distance: 0.001457, exp_rate: 0.933731, exp_rate<=1 : 0.973411, exp_rate<=2 : 0.992778, exp_rate<=3 : 0.996789, loss: 0.002002, avg_reader_cost: 0.03234 s, avg_batch_cost: 0.21706 s, avg_samples: 56.0, ips: 257.99398 samples/s, eta: 21:15:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:57:40] ppocr INFO: epoch: [356/500], global_step: 150900, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.931694, exp_rate<=1 : 0.974232, exp_rate<=2 : 0.993918, exp_rate<=3 : 0.997409, loss: 0.001383, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19769 s, avg_samples: 56.0, ips: 283.26845 samples/s, eta: 21:15:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:57:59] ppocr INFO: epoch: [356/500], global_step: 151000, lr: 0.000100, edit distance: 0.001481, exp_rate: 0.934271, exp_rate<=1 : 0.975301, exp_rate<=2 : 0.993903, exp_rate<=3 : 0.997524, loss: 0.001370, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19131 s, avg_samples: 56.0, ips: 292.71740 samples/s, eta: 21:14:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:58:20] ppocr INFO: epoch: [356/500], global_step: 151100, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.934915, exp_rate<=1 : 0.975302, exp_rate<=2 : 0.994155, exp_rate<=3 : 0.997737, loss: 0.001319, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20903 s, avg_samples: 56.0, ips: 267.91043 samples/s, eta: 21:14:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:58:39] ppocr INFO: epoch: [356/500], global_step: 151200, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.935468, exp_rate<=1 : 0.975838, exp_rate<=2 : 0.994436, exp_rate<=3 : 0.997823, loss: 0.001433, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19004 s, avg_samples: 56.0, ips: 294.67261 samples/s, eta: 21:14:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:58:58] ppocr INFO: epoch: [356/500], global_step: 151300, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.936575, exp_rate<=1 : 0.975978, exp_rate<=2 : 0.994390, exp_rate<=3 : 0.998004, loss: 0.001556, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19308 s, avg_samples: 56.0, ips: 290.03050 samples/s, eta: 21:13:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:59:17] ppocr INFO: epoch: [356/500], global_step: 151400, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.937797, exp_rate<=1 : 0.976496, exp_rate<=2 : 0.994574, exp_rate<=3 : 0.998112, loss: 0.001211, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19095 s, avg_samples: 56.0, ips: 293.27167 samples/s, eta: 21:13:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:59:37] ppocr INFO: epoch: [356/500], global_step: 151500, lr: 0.000100, edit distance: 0.001377, exp_rate: 0.937069, exp_rate<=1 : 0.975752, exp_rate<=2 : 0.994286, exp_rate<=3 : 0.997976, loss: 0.001778, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20254 s, avg_samples: 56.0, ips: 276.49112 samples/s, eta: 21:13:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 10:59:57] ppocr INFO: epoch: [356/500], global_step: 151600, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.935563, exp_rate<=1 : 0.975229, exp_rate<=2 : 0.994155, exp_rate<=3 : 0.997864, loss: 0.002092, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19357 s, avg_samples: 56.0, ips: 289.29734 samples/s, eta: 21:13:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:00:15] ppocr INFO: epoch: [356/500], global_step: 151700, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.934721, exp_rate<=1 : 0.974934, exp_rate<=2 : 0.994268, exp_rate<=3 : 0.997878, loss: 0.001686, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17763 s, avg_samples: 56.0, ips: 315.25817 samples/s, eta: 21:12:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:00:34] ppocr INFO: epoch: [356/500], global_step: 151800, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.934037, exp_rate<=1 : 0.974967, exp_rate<=2 : 0.994380, exp_rate<=3 : 0.997953, loss: 0.002308, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18900 s, avg_samples: 56.0, ips: 296.29173 samples/s, eta: 21:12:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:00:52] ppocr INFO: epoch: [356/500], global_step: 151900, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.934299, exp_rate<=1 : 0.974919, exp_rate<=2 : 0.994344, exp_rate<=3 : 0.997968, loss: 0.002116, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18103 s, avg_samples: 56.0, ips: 309.33718 samples/s, eta: 21:11:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:01:10] ppocr INFO: epoch: [356/500], global_step: 152000, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.934087, exp_rate<=1 : 0.974806, exp_rate<=2 : 0.994207, exp_rate<=3 : 0.997925, loss: 0.001432, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18609 s, avg_samples: 56.0, ips: 300.93055 samples/s, eta: 21:11:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:01:30] ppocr INFO: epoch: [356/500], global_step: 152100, lr: 0.000100, edit distance: 0.001484, exp_rate: 0.934327, exp_rate<=1 : 0.974845, exp_rate<=2 : 0.994179, exp_rate<=3 : 0.997954, loss: 0.001571, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19659 s, avg_samples: 56.0, ips: 284.85170 samples/s, eta: 21:11:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:01:47] ppocr INFO: epoch: [356/500], global_step: 152200, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.934791, exp_rate<=1 : 0.975097, exp_rate<=2 : 0.994316, exp_rate<=3 : 0.998054, loss: 0.001634, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17523 s, avg_samples: 56.0, ips: 319.57432 samples/s, eta: 21:10:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:02:05] ppocr INFO: epoch: [356/500], global_step: 152300, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.935166, exp_rate<=1 : 0.975213, exp_rate<=2 : 0.994309, exp_rate<=3 : 0.998051, loss: 0.001654, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17618 s, avg_samples: 56.0, ips: 317.84902 samples/s, eta: 21:10:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:02:25] ppocr INFO: epoch: [356/500], global_step: 152400, lr: 0.000100, edit distance: 0.001461, exp_rate: 0.934826, exp_rate<=1 : 0.974966, exp_rate<=2 : 0.994236, exp_rate<=3 : 0.998026, loss: 0.002006, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19563 s, avg_samples: 56.0, ips: 286.24902 samples/s, eta: 21:10:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:02:43] ppocr INFO: epoch: [356/500], global_step: 152500, lr: 0.000100, edit distance: 0.001452, exp_rate: 0.935098, exp_rate<=1 : 0.975046, exp_rate<=2 : 0.994196, exp_rate<=3 : 0.998029, loss: 0.002058, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18484 s, avg_samples: 56.0, ips: 302.96904 samples/s, eta: 21:09:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:03:03] ppocr INFO: epoch: [356/500], global_step: 152600, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.934625, exp_rate<=1 : 0.974825, exp_rate<=2 : 0.994112, exp_rate<=3 : 0.998060, loss: 0.001605, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19980 s, avg_samples: 56.0, ips: 280.27647 samples/s, eta: 21:09:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:03:21] ppocr INFO: epoch: [356/500], global_step: 152700, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.934652, exp_rate<=1 : 0.974823, exp_rate<=2 : 0.994160, exp_rate<=3 : 0.998085, loss: 0.001573, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18116 s, avg_samples: 56.0, ips: 309.11967 samples/s, eta: 21:09:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:03:41] ppocr INFO: epoch: [356/500], global_step: 152800, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.934196, exp_rate<=1 : 0.974641, exp_rate<=2 : 0.994142, exp_rate<=3 : 0.998077, loss: 0.002223, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20121 s, avg_samples: 56.0, ips: 278.31849 samples/s, eta: 21:08:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:04:00] ppocr INFO: epoch: [356/500], global_step: 152900, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.934591, exp_rate<=1 : 0.974787, exp_rate<=2 : 0.994060, exp_rate<=3 : 0.998060, loss: 0.001338, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18627 s, avg_samples: 56.0, ips: 300.64457 samples/s, eta: 21:08:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:04:20] ppocr INFO: epoch: [356/500], global_step: 153000, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.934167, exp_rate<=1 : 0.974692, exp_rate<=2 : 0.994007, exp_rate<=3 : 0.998020, loss: 0.001336, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20085 s, avg_samples: 56.0, ips: 278.81426 samples/s, eta: 21:08:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:04:40] ppocr INFO: epoch: [356/500], global_step: 153100, lr: 0.000100, edit distance: 0.001470, exp_rate: 0.934241, exp_rate<=1 : 0.974694, exp_rate<=2 : 0.994044, exp_rate<=3 : 0.998045, loss: 0.001331, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19267 s, avg_samples: 56.0, ips: 290.64928 samples/s, eta: 21:07:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:04:59] ppocr INFO: epoch: [356/500], global_step: 153200, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.934133, exp_rate<=1 : 0.974659, exp_rate<=2 : 0.994024, exp_rate<=3 : 0.998023, loss: 0.001634, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18993 s, avg_samples: 56.0, ips: 294.83982 samples/s, eta: 21:07:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:05:17] ppocr INFO: epoch: [356/500], global_step: 153300, lr: 0.000100, edit distance: 0.001478, exp_rate: 0.934046, exp_rate<=1 : 0.974594, exp_rate<=2 : 0.994038, exp_rate<=3 : 0.998027, loss: 0.001730, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18848 s, avg_samples: 56.0, ips: 297.11935 samples/s, eta: 21:07:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:05:37] ppocr INFO: epoch: [356/500], global_step: 153400, lr: 0.000100, edit distance: 0.001479, exp_rate: 0.933642, exp_rate<=1 : 0.974449, exp_rate<=2 : 0.994003, exp_rate<=3 : 0.997976, loss: 0.001839, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19460 s, avg_samples: 56.0, ips: 287.77570 samples/s, eta: 21:06:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:05:46] ppocr INFO: epoch: [356/500], global_step: 153440, lr: 0.000100, edit distance: 0.001482, exp_rate: 0.933346, exp_rate<=1 : 0.974295, exp_rate<=2 : 0.993987, exp_rate<=3 : 0.997965, loss: 0.002210, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.08864 s, avg_samples: 22.4, ips: 252.69858 samples/s, eta: 21:06:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:05:48] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 11:06:00] ppocr INFO: epoch: [357/500], global_step: 153500, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.942443, exp_rate<=1 : 0.977876, exp_rate<=2 : 0.995028, exp_rate<=3 : 0.998362, loss: 0.001709, avg_reader_cost: 0.02676 s, avg_batch_cost: 0.14241 s, avg_samples: 33.6, ips: 235.93832 samples/s, eta: 21:06:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:06:20] ppocr INFO: epoch: [357/500], global_step: 153600, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.941894, exp_rate<=1 : 0.977665, exp_rate<=2 : 0.995000, exp_rate<=3 : 0.997977, loss: 0.001573, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19899 s, avg_samples: 56.0, ips: 281.42780 samples/s, eta: 21:06:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:06:39] ppocr INFO: epoch: [357/500], global_step: 153700, lr: 0.000100, edit distance: 0.001175, exp_rate: 0.944968, exp_rate<=1 : 0.979233, exp_rate<=2 : 0.995411, exp_rate<=3 : 0.998563, loss: 0.001787, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18688 s, avg_samples: 56.0, ips: 299.66184 samples/s, eta: 21:06:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:06:58] ppocr INFO: epoch: [357/500], global_step: 153800, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.942892, exp_rate<=1 : 0.978220, exp_rate<=2 : 0.995004, exp_rate<=3 : 0.998558, loss: 0.001460, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19148 s, avg_samples: 56.0, ips: 292.46436 samples/s, eta: 21:05:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:07:16] ppocr INFO: epoch: [357/500], global_step: 153900, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.944266, exp_rate<=1 : 0.978969, exp_rate<=2 : 0.995162, exp_rate<=3 : 0.998507, loss: 0.001859, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18222 s, avg_samples: 56.0, ips: 307.32800 samples/s, eta: 21:05:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:07:34] ppocr INFO: epoch: [357/500], global_step: 154000, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.944658, exp_rate<=1 : 0.979305, exp_rate<=2 : 0.995285, exp_rate<=3 : 0.998597, loss: 0.001722, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17607 s, avg_samples: 56.0, ips: 318.06075 samples/s, eta: 21:05:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:07:54] ppocr INFO: epoch: [357/500], global_step: 154100, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.941772, exp_rate<=1 : 0.977858, exp_rate<=2 : 0.995129, exp_rate<=3 : 0.998588, loss: 0.001929, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19874 s, avg_samples: 56.0, ips: 281.77088 samples/s, eta: 21:04:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:08:13] ppocr INFO: epoch: [357/500], global_step: 154200, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.940674, exp_rate<=1 : 0.977530, exp_rate<=2 : 0.994966, exp_rate<=3 : 0.998520, loss: 0.002096, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19634 s, avg_samples: 56.0, ips: 285.21969 samples/s, eta: 21:04:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:08:32] ppocr INFO: epoch: [357/500], global_step: 154300, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.940214, exp_rate<=1 : 0.977579, exp_rate<=2 : 0.995140, exp_rate<=3 : 0.998604, loss: 0.001618, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18491 s, avg_samples: 56.0, ips: 302.84468 samples/s, eta: 21:04:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:08:51] ppocr INFO: epoch: [357/500], global_step: 154400, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.940501, exp_rate<=1 : 0.977751, exp_rate<=2 : 0.994964, exp_rate<=3 : 0.998566, loss: 0.001729, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18806 s, avg_samples: 56.0, ips: 297.78096 samples/s, eta: 21:03:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:09:11] ppocr INFO: epoch: [357/500], global_step: 154500, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.939294, exp_rate<=1 : 0.976951, exp_rate<=2 : 0.994898, exp_rate<=3 : 0.998435, loss: 0.001494, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19928 s, avg_samples: 56.0, ips: 281.01454 samples/s, eta: 21:03:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:09:30] ppocr INFO: epoch: [357/500], global_step: 154600, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.938094, exp_rate<=1 : 0.976351, exp_rate<=2 : 0.994742, exp_rate<=3 : 0.998307, loss: 0.001651, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19520 s, avg_samples: 56.0, ips: 286.88801 samples/s, eta: 21:03:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:09:50] ppocr INFO: epoch: [357/500], global_step: 154700, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.937908, exp_rate<=1 : 0.976352, exp_rate<=2 : 0.994659, exp_rate<=3 : 0.998229, loss: 0.001213, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19708 s, avg_samples: 56.0, ips: 284.14893 samples/s, eta: 21:02:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:10:08] ppocr INFO: epoch: [357/500], global_step: 154800, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.938599, exp_rate<=1 : 0.976697, exp_rate<=2 : 0.994790, exp_rate<=3 : 0.998285, loss: 0.001814, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18177 s, avg_samples: 56.0, ips: 308.07905 samples/s, eta: 21:02:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:10:28] ppocr INFO: epoch: [357/500], global_step: 154900, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.938485, exp_rate<=1 : 0.976616, exp_rate<=2 : 0.994715, exp_rate<=3 : 0.998214, loss: 0.001398, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19577 s, avg_samples: 56.0, ips: 286.05502 samples/s, eta: 21:02:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:10:47] ppocr INFO: epoch: [357/500], global_step: 155000, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.938334, exp_rate<=1 : 0.976411, exp_rate<=2 : 0.994710, exp_rate<=3 : 0.998206, loss: 0.002472, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19050 s, avg_samples: 56.0, ips: 293.96752 samples/s, eta: 21:01:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:11:07] ppocr INFO: epoch: [357/500], global_step: 155100, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.937026, exp_rate<=1 : 0.975949, exp_rate<=2 : 0.994727, exp_rate<=3 : 0.998225, loss: 0.002249, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19832 s, avg_samples: 56.0, ips: 282.36876 samples/s, eta: 21:01:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:11:24] ppocr INFO: epoch: [357/500], global_step: 155200, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.937207, exp_rate<=1 : 0.975987, exp_rate<=2 : 0.994763, exp_rate<=3 : 0.998258, loss: 0.002019, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17789 s, avg_samples: 56.0, ips: 314.79324 samples/s, eta: 21:01:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:11:44] ppocr INFO: epoch: [357/500], global_step: 155300, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.937141, exp_rate<=1 : 0.975986, exp_rate<=2 : 0.994769, exp_rate<=3 : 0.998299, loss: 0.002008, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19234 s, avg_samples: 56.0, ips: 291.14433 samples/s, eta: 21:00:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:12:03] ppocr INFO: epoch: [357/500], global_step: 155400, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.936741, exp_rate<=1 : 0.975745, exp_rate<=2 : 0.994657, exp_rate<=3 : 0.998265, loss: 0.001139, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19591 s, avg_samples: 56.0, ips: 285.84377 samples/s, eta: 21:00:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:12:24] ppocr INFO: epoch: [357/500], global_step: 155500, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.936020, exp_rate<=1 : 0.975380, exp_rate<=2 : 0.994499, exp_rate<=3 : 0.998159, loss: 0.001468, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20319 s, avg_samples: 56.0, ips: 275.60084 samples/s, eta: 21:00:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:12:41] ppocr INFO: epoch: [357/500], global_step: 155600, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.936527, exp_rate<=1 : 0.975649, exp_rate<=2 : 0.994602, exp_rate<=3 : 0.998170, loss: 0.001744, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17834 s, avg_samples: 56.0, ips: 313.99953 samples/s, eta: 20:59:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:13:02] ppocr INFO: epoch: [357/500], global_step: 155700, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.935268, exp_rate<=1 : 0.974968, exp_rate<=2 : 0.994356, exp_rate<=3 : 0.998087, loss: 0.002541, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20585 s, avg_samples: 56.0, ips: 272.04195 samples/s, eta: 20:59:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:13:21] ppocr INFO: epoch: [357/500], global_step: 155800, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.934891, exp_rate<=1 : 0.974804, exp_rate<=2 : 0.994305, exp_rate<=3 : 0.998055, loss: 0.001996, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18630 s, avg_samples: 56.0, ips: 300.58740 samples/s, eta: 20:59:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:13:40] ppocr INFO: epoch: [357/500], global_step: 155900, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.934503, exp_rate<=1 : 0.974665, exp_rate<=2 : 0.994234, exp_rate<=3 : 0.998034, loss: 0.002406, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19069 s, avg_samples: 56.0, ips: 293.67575 samples/s, eta: 20:58:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:13:58] ppocr INFO: epoch: [357/500], global_step: 156000, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.934278, exp_rate<=1 : 0.974602, exp_rate<=2 : 0.994129, exp_rate<=3 : 0.997981, loss: 0.001154, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18471 s, avg_samples: 56.0, ips: 303.17957 samples/s, eta: 20:58:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:14:19] ppocr INFO: epoch: [357/500], global_step: 156100, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.934083, exp_rate<=1 : 0.974518, exp_rate<=2 : 0.994118, exp_rate<=3 : 0.997999, loss: 0.001245, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20251 s, avg_samples: 56.0, ips: 276.52467 samples/s, eta: 20:58:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:14:34] ppocr INFO: epoch: [357/500], global_step: 156180, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.933517, exp_rate<=1 : 0.974296, exp_rate<=2 : 0.994033, exp_rate<=3 : 0.997920, loss: 0.002504, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.15768 s, avg_samples: 44.8, ips: 284.11875 samples/s, eta: 20:58:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:14:37] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 11:14:41] ppocr INFO: epoch: [358/500], global_step: 156200, lr: 0.000100, edit distance: 0.001680, exp_rate: 0.932341, exp_rate<=1 : 0.975000, exp_rate<=2 : 0.995217, exp_rate<=3 : 0.998917, loss: 0.001822, avg_reader_cost: 0.02814 s, avg_batch_cost: 0.06588 s, avg_samples: 11.2, ips: 170.00451 samples/s, eta: 20:58:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:15:00] ppocr INFO: epoch: [358/500], global_step: 156300, lr: 0.000100, edit distance: 0.001444, exp_rate: 0.934467, exp_rate<=1 : 0.970238, exp_rate<=2 : 0.992225, exp_rate<=3 : 0.996032, loss: 0.002251, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18749 s, avg_samples: 56.0, ips: 298.67983 samples/s, eta: 20:57:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:15:18] ppocr INFO: epoch: [358/500], global_step: 156400, lr: 0.000100, edit distance: 0.001610, exp_rate: 0.930214, exp_rate<=1 : 0.968960, exp_rate<=2 : 0.991029, exp_rate<=3 : 0.996152, loss: 0.003024, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18554 s, avg_samples: 56.0, ips: 301.82504 samples/s, eta: 20:57:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:15:37] ppocr INFO: epoch: [358/500], global_step: 156500, lr: 0.000100, edit distance: 0.001547, exp_rate: 0.931447, exp_rate<=1 : 0.970554, exp_rate<=2 : 0.991431, exp_rate<=3 : 0.996319, loss: 0.001220, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18669 s, avg_samples: 56.0, ips: 299.95586 samples/s, eta: 20:57:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:15:56] ppocr INFO: epoch: [358/500], global_step: 156600, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.935071, exp_rate<=1 : 0.972573, exp_rate<=2 : 0.992390, exp_rate<=3 : 0.996995, loss: 0.001541, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18936 s, avg_samples: 56.0, ips: 295.72711 samples/s, eta: 20:56:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:16:14] ppocr INFO: epoch: [358/500], global_step: 156700, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.937205, exp_rate<=1 : 0.973983, exp_rate<=2 : 0.993074, exp_rate<=3 : 0.997307, loss: 0.001043, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18505 s, avg_samples: 56.0, ips: 302.62012 samples/s, eta: 20:56:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:16:36] ppocr INFO: epoch: [358/500], global_step: 156800, lr: 0.000100, edit distance: 0.001351, exp_rate: 0.936746, exp_rate<=1 : 0.974500, exp_rate<=2 : 0.993335, exp_rate<=3 : 0.997436, loss: 0.001736, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21549 s, avg_samples: 56.0, ips: 259.87322 samples/s, eta: 20:56:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:16:57] ppocr INFO: epoch: [358/500], global_step: 156900, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.935307, exp_rate<=1 : 0.973748, exp_rate<=2 : 0.993147, exp_rate<=3 : 0.997399, loss: 0.001644, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20710 s, avg_samples: 56.0, ips: 270.40671 samples/s, eta: 20:55:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:17:16] ppocr INFO: epoch: [358/500], global_step: 157000, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.935736, exp_rate<=1 : 0.973976, exp_rate<=2 : 0.993357, exp_rate<=3 : 0.997532, loss: 0.001558, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19391 s, avg_samples: 56.0, ips: 288.79742 samples/s, eta: 20:55:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:17:35] ppocr INFO: epoch: [358/500], global_step: 157100, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.934950, exp_rate<=1 : 0.974052, exp_rate<=2 : 0.993529, exp_rate<=3 : 0.997701, loss: 0.002233, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19047 s, avg_samples: 56.0, ips: 294.01455 samples/s, eta: 20:55:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:17:54] ppocr INFO: epoch: [358/500], global_step: 157200, lr: 0.000100, edit distance: 0.001377, exp_rate: 0.935720, exp_rate<=1 : 0.974468, exp_rate<=2 : 0.993849, exp_rate<=3 : 0.997823, loss: 0.001856, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18479 s, avg_samples: 56.0, ips: 303.05465 samples/s, eta: 20:54:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:18:12] ppocr INFO: epoch: [358/500], global_step: 157300, lr: 0.000100, edit distance: 0.001372, exp_rate: 0.936565, exp_rate<=1 : 0.974815, exp_rate<=2 : 0.993997, exp_rate<=3 : 0.997857, loss: 0.001538, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.18499 s, avg_samples: 56.0, ips: 302.71459 samples/s, eta: 20:54:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:18:32] ppocr INFO: epoch: [358/500], global_step: 157400, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.936766, exp_rate<=1 : 0.974925, exp_rate<=2 : 0.994067, exp_rate<=3 : 0.997914, loss: 0.001714, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19267 s, avg_samples: 56.0, ips: 290.65048 samples/s, eta: 20:54:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:18:51] ppocr INFO: epoch: [358/500], global_step: 157500, lr: 0.000100, edit distance: 0.001384, exp_rate: 0.936798, exp_rate<=1 : 0.975068, exp_rate<=2 : 0.994148, exp_rate<=3 : 0.998004, loss: 0.001231, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19088 s, avg_samples: 56.0, ips: 293.38545 samples/s, eta: 20:53:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:19:11] ppocr INFO: epoch: [358/500], global_step: 157600, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.936203, exp_rate<=1 : 0.974718, exp_rate<=2 : 0.994141, exp_rate<=3 : 0.997924, loss: 0.001489, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20194 s, avg_samples: 56.0, ips: 277.31319 samples/s, eta: 20:53:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:19:31] ppocr INFO: epoch: [358/500], global_step: 157700, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.935630, exp_rate<=1 : 0.974502, exp_rate<=2 : 0.993973, exp_rate<=3 : 0.997892, loss: 0.001688, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20384 s, avg_samples: 56.0, ips: 274.73024 samples/s, eta: 20:53:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:19:50] ppocr INFO: epoch: [358/500], global_step: 157800, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.935992, exp_rate<=1 : 0.974604, exp_rate<=2 : 0.993990, exp_rate<=3 : 0.997930, loss: 0.001505, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18787 s, avg_samples: 56.0, ips: 298.08098 samples/s, eta: 20:52:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:20:09] ppocr INFO: epoch: [358/500], global_step: 157900, lr: 0.000100, edit distance: 0.001384, exp_rate: 0.936301, exp_rate<=1 : 0.974790, exp_rate<=2 : 0.994098, exp_rate<=3 : 0.997994, loss: 0.001851, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19181 s, avg_samples: 56.0, ips: 291.95710 samples/s, eta: 20:52:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:20:28] ppocr INFO: epoch: [358/500], global_step: 158000, lr: 0.000100, edit distance: 0.001372, exp_rate: 0.936277, exp_rate<=1 : 0.974866, exp_rate<=2 : 0.994018, exp_rate<=3 : 0.997896, loss: 0.001842, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18969 s, avg_samples: 56.0, ips: 295.21692 samples/s, eta: 20:52:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:20:49] ppocr INFO: epoch: [358/500], global_step: 158100, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.936134, exp_rate<=1 : 0.974885, exp_rate<=2 : 0.993973, exp_rate<=3 : 0.997881, loss: 0.001933, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20615 s, avg_samples: 56.0, ips: 271.64085 samples/s, eta: 20:52:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:21:08] ppocr INFO: epoch: [358/500], global_step: 158200, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.935791, exp_rate<=1 : 0.974621, exp_rate<=2 : 0.993884, exp_rate<=3 : 0.997802, loss: 0.002056, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19472 s, avg_samples: 56.0, ips: 287.59962 samples/s, eta: 20:51:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:21:26] ppocr INFO: epoch: [358/500], global_step: 158300, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.935604, exp_rate<=1 : 0.974691, exp_rate<=2 : 0.994026, exp_rate<=3 : 0.997826, loss: 0.001469, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18039 s, avg_samples: 56.0, ips: 310.43506 samples/s, eta: 20:51:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:21:45] ppocr INFO: epoch: [358/500], global_step: 158400, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.935410, exp_rate<=1 : 0.974671, exp_rate<=2 : 0.994089, exp_rate<=3 : 0.997854, loss: 0.001995, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18224 s, avg_samples: 56.0, ips: 307.27935 samples/s, eta: 20:51:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:22:04] ppocr INFO: epoch: [358/500], global_step: 158500, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.934812, exp_rate<=1 : 0.974369, exp_rate<=2 : 0.994008, exp_rate<=3 : 0.997759, loss: 0.001650, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19244 s, avg_samples: 56.0, ips: 291.00424 samples/s, eta: 20:50:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:22:24] ppocr INFO: epoch: [358/500], global_step: 158600, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.934500, exp_rate<=1 : 0.974162, exp_rate<=2 : 0.993977, exp_rate<=3 : 0.997763, loss: 0.001426, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20028 s, avg_samples: 56.0, ips: 279.60472 samples/s, eta: 20:50:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:22:43] ppocr INFO: epoch: [358/500], global_step: 158700, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.934427, exp_rate<=1 : 0.974124, exp_rate<=2 : 0.994023, exp_rate<=3 : 0.997808, loss: 0.001784, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18784 s, avg_samples: 56.0, ips: 298.12616 samples/s, eta: 20:50:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:23:02] ppocr INFO: epoch: [358/500], global_step: 158800, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.934178, exp_rate<=1 : 0.974107, exp_rate<=2 : 0.994021, exp_rate<=3 : 0.997797, loss: 0.002121, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18774 s, avg_samples: 56.0, ips: 298.28905 samples/s, eta: 20:49:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:23:20] ppocr INFO: epoch: [358/500], global_step: 158900, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.934269, exp_rate<=1 : 0.974178, exp_rate<=2 : 0.994074, exp_rate<=3 : 0.997826, loss: 0.001313, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18517 s, avg_samples: 56.0, ips: 302.42510 samples/s, eta: 20:49:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:23:24] ppocr INFO: epoch: [358/500], global_step: 158920, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.934361, exp_rate<=1 : 0.974204, exp_rate<=2 : 0.994096, exp_rate<=3 : 0.997842, loss: 0.001431, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.03630 s, avg_samples: 11.2, ips: 308.50225 samples/s, eta: 20:49:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:23:26] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 11:23:42] ppocr INFO: epoch: [359/500], global_step: 159000, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.942934, exp_rate<=1 : 0.977858, exp_rate<=2 : 0.995294, exp_rate<=3 : 0.999486, loss: 0.001886, avg_reader_cost: 0.02629 s, avg_batch_cost: 0.18492 s, avg_samples: 44.8, ips: 242.26917 samples/s, eta: 20:49:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:24:02] ppocr INFO: epoch: [359/500], global_step: 159100, lr: 0.000100, edit distance: 0.001358, exp_rate: 0.941200, exp_rate<=1 : 0.977435, exp_rate<=2 : 0.995344, exp_rate<=3 : 0.999115, loss: 0.001105, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19307 s, avg_samples: 56.0, ips: 290.05124 samples/s, eta: 20:48:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:24:22] ppocr INFO: epoch: [359/500], global_step: 159200, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.940185, exp_rate<=1 : 0.977511, exp_rate<=2 : 0.995772, exp_rate<=3 : 0.998857, loss: 0.001276, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20440 s, avg_samples: 56.0, ips: 273.96624 samples/s, eta: 20:48:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:24:41] ppocr INFO: epoch: [359/500], global_step: 159300, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.939907, exp_rate<=1 : 0.977183, exp_rate<=2 : 0.995421, exp_rate<=3 : 0.998554, loss: 0.001824, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18690 s, avg_samples: 56.0, ips: 299.62882 samples/s, eta: 20:48:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:25:01] ppocr INFO: epoch: [359/500], global_step: 159400, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.936724, exp_rate<=1 : 0.975088, exp_rate<=2 : 0.994060, exp_rate<=3 : 0.997675, loss: 0.001074, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20196 s, avg_samples: 56.0, ips: 277.27652 samples/s, eta: 20:47:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:25:20] ppocr INFO: epoch: [359/500], global_step: 159500, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.936489, exp_rate<=1 : 0.975082, exp_rate<=2 : 0.993556, exp_rate<=3 : 0.997527, loss: 0.001119, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19314 s, avg_samples: 56.0, ips: 289.93886 samples/s, eta: 20:47:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:25:40] ppocr INFO: epoch: [359/500], global_step: 159600, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.937713, exp_rate<=1 : 0.975655, exp_rate<=2 : 0.993905, exp_rate<=3 : 0.997753, loss: 0.001646, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19221 s, avg_samples: 56.0, ips: 291.34571 samples/s, eta: 20:47:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:25:58] ppocr INFO: epoch: [359/500], global_step: 159700, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.938792, exp_rate<=1 : 0.976159, exp_rate<=2 : 0.993998, exp_rate<=3 : 0.997791, loss: 0.002095, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18383 s, avg_samples: 56.0, ips: 304.62256 samples/s, eta: 20:46:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:26:16] ppocr INFO: epoch: [359/500], global_step: 159800, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.939015, exp_rate<=1 : 0.976255, exp_rate<=2 : 0.994116, exp_rate<=3 : 0.997785, loss: 0.001812, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18472 s, avg_samples: 56.0, ips: 303.16042 samples/s, eta: 20:46:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:26:35] ppocr INFO: epoch: [359/500], global_step: 159900, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.938883, exp_rate<=1 : 0.976440, exp_rate<=2 : 0.994198, exp_rate<=3 : 0.997809, loss: 0.001961, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18916 s, avg_samples: 56.0, ips: 296.04412 samples/s, eta: 20:46:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:26:54] ppocr INFO: epoch: [359/500], global_step: 160000, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.938624, exp_rate<=1 : 0.976290, exp_rate<=2 : 0.994106, exp_rate<=3 : 0.997833, loss: 0.002384, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19013 s, avg_samples: 56.0, ips: 294.54111 samples/s, eta: 20:45:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:27:15] ppocr INFO: epoch: [359/500], global_step: 160100, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.936706, exp_rate<=1 : 0.975283, exp_rate<=2 : 0.993741, exp_rate<=3 : 0.997555, loss: 0.001820, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20303 s, avg_samples: 56.0, ips: 275.82539 samples/s, eta: 20:45:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:27:34] ppocr INFO: epoch: [359/500], global_step: 160200, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.936418, exp_rate<=1 : 0.975354, exp_rate<=2 : 0.993697, exp_rate<=3 : 0.997552, loss: 0.001575, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19457 s, avg_samples: 56.0, ips: 287.81239 samples/s, eta: 20:45:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:27:53] ppocr INFO: epoch: [359/500], global_step: 160300, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.936676, exp_rate<=1 : 0.975456, exp_rate<=2 : 0.993774, exp_rate<=3 : 0.997590, loss: 0.002017, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18494 s, avg_samples: 56.0, ips: 302.79321 samples/s, eta: 20:44:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:28:11] ppocr INFO: epoch: [359/500], global_step: 160400, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.936014, exp_rate<=1 : 0.975198, exp_rate<=2 : 0.993823, exp_rate<=3 : 0.997637, loss: 0.001857, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18450 s, avg_samples: 56.0, ips: 303.52892 samples/s, eta: 20:44:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:28:29] ppocr INFO: epoch: [359/500], global_step: 160500, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.935793, exp_rate<=1 : 0.974920, exp_rate<=2 : 0.993803, exp_rate<=3 : 0.997669, loss: 0.002028, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17863 s, avg_samples: 56.0, ips: 313.50250 samples/s, eta: 20:44:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:28:49] ppocr INFO: epoch: [359/500], global_step: 160600, lr: 0.000100, edit distance: 0.001419, exp_rate: 0.935054, exp_rate<=1 : 0.974717, exp_rate<=2 : 0.993880, exp_rate<=3 : 0.997734, loss: 0.001592, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20024 s, avg_samples: 56.0, ips: 279.65890 samples/s, eta: 20:43:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:29:08] ppocr INFO: epoch: [359/500], global_step: 160700, lr: 0.000100, edit distance: 0.001416, exp_rate: 0.935376, exp_rate<=1 : 0.974834, exp_rate<=2 : 0.993960, exp_rate<=3 : 0.997767, loss: 0.001796, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19198 s, avg_samples: 56.0, ips: 291.69261 samples/s, eta: 20:43:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:29:27] ppocr INFO: epoch: [359/500], global_step: 160800, lr: 0.000100, edit distance: 0.001413, exp_rate: 0.935374, exp_rate<=1 : 0.974854, exp_rate<=2 : 0.993966, exp_rate<=3 : 0.997814, loss: 0.001134, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19012 s, avg_samples: 56.0, ips: 294.54399 samples/s, eta: 20:43:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:29:47] ppocr INFO: epoch: [359/500], global_step: 160900, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.935026, exp_rate<=1 : 0.974747, exp_rate<=2 : 0.993928, exp_rate<=3 : 0.997798, loss: 0.001580, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19544 s, avg_samples: 56.0, ips: 286.53822 samples/s, eta: 20:43:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:30:06] ppocr INFO: epoch: [359/500], global_step: 161000, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.935132, exp_rate<=1 : 0.974755, exp_rate<=2 : 0.993931, exp_rate<=3 : 0.997788, loss: 0.001807, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18777 s, avg_samples: 56.0, ips: 298.24418 samples/s, eta: 20:42:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:30:25] ppocr INFO: epoch: [359/500], global_step: 161100, lr: 0.000100, edit distance: 0.001419, exp_rate: 0.935227, exp_rate<=1 : 0.974760, exp_rate<=2 : 0.993967, exp_rate<=3 : 0.997819, loss: 0.001991, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19305 s, avg_samples: 56.0, ips: 290.08400 samples/s, eta: 20:42:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:30:44] ppocr INFO: epoch: [359/500], global_step: 161200, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.934823, exp_rate<=1 : 0.974588, exp_rate<=2 : 0.993938, exp_rate<=3 : 0.997806, loss: 0.001427, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19137 s, avg_samples: 56.0, ips: 292.63065 samples/s, eta: 20:42:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:31:04] ppocr INFO: epoch: [359/500], global_step: 161300, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.934617, exp_rate<=1 : 0.974440, exp_rate<=2 : 0.993893, exp_rate<=3 : 0.997797, loss: 0.001552, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19672 s, avg_samples: 56.0, ips: 284.67129 samples/s, eta: 20:41:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:31:23] ppocr INFO: epoch: [359/500], global_step: 161400, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.934507, exp_rate<=1 : 0.974397, exp_rate<=2 : 0.993922, exp_rate<=3 : 0.997825, loss: 0.001749, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19041 s, avg_samples: 56.0, ips: 294.10369 samples/s, eta: 20:41:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:31:42] ppocr INFO: epoch: [359/500], global_step: 161500, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.934688, exp_rate<=1 : 0.974496, exp_rate<=2 : 0.994000, exp_rate<=3 : 0.997867, loss: 0.001820, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18519 s, avg_samples: 56.0, ips: 302.38623 samples/s, eta: 20:41:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:32:00] ppocr INFO: epoch: [359/500], global_step: 161600, lr: 0.000100, edit distance: 0.001444, exp_rate: 0.934777, exp_rate<=1 : 0.974510, exp_rate<=2 : 0.994049, exp_rate<=3 : 0.997874, loss: 0.001812, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18355 s, avg_samples: 56.0, ips: 305.09312 samples/s, eta: 20:40:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:32:11] ppocr INFO: epoch: [359/500], global_step: 161660, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.934812, exp_rate<=1 : 0.974599, exp_rate<=2 : 0.994085, exp_rate<=3 : 0.997891, loss: 0.001710, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.11292 s, avg_samples: 33.6, ips: 297.55279 samples/s, eta: 20:40:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:32:13] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 11:32:21] ppocr INFO: epoch: [360/500], global_step: 161700, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.941882, exp_rate<=1 : 0.974483, exp_rate<=2 : 0.992306, exp_rate<=3 : 0.997231, loss: 0.001133, avg_reader_cost: 0.02626 s, avg_batch_cost: 0.09849 s, avg_samples: 22.4, ips: 227.44532 samples/s, eta: 20:40:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:32:41] ppocr INFO: epoch: [360/500], global_step: 161800, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.941237, exp_rate<=1 : 0.977642, exp_rate<=2 : 0.994748, exp_rate<=3 : 0.998109, loss: 0.001236, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19729 s, avg_samples: 56.0, ips: 283.83944 samples/s, eta: 20:40:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:32:59] ppocr INFO: epoch: [360/500], global_step: 161900, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.944892, exp_rate<=1 : 0.978903, exp_rate<=2 : 0.995359, exp_rate<=3 : 0.998451, loss: 0.001329, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18050 s, avg_samples: 56.0, ips: 310.24196 samples/s, eta: 20:39:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:33:18] ppocr INFO: epoch: [360/500], global_step: 162000, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.944997, exp_rate<=1 : 0.979530, exp_rate<=2 : 0.995563, exp_rate<=3 : 0.998433, loss: 0.000965, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18836 s, avg_samples: 56.0, ips: 297.29539 samples/s, eta: 20:39:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:33:38] ppocr INFO: epoch: [360/500], global_step: 162100, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.942403, exp_rate<=1 : 0.977431, exp_rate<=2 : 0.995313, exp_rate<=3 : 0.998122, loss: 0.001471, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20079 s, avg_samples: 56.0, ips: 278.90471 samples/s, eta: 20:39:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:33:57] ppocr INFO: epoch: [360/500], global_step: 162200, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.939404, exp_rate<=1 : 0.975730, exp_rate<=2 : 0.994120, exp_rate<=3 : 0.997518, loss: 0.001959, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18708 s, avg_samples: 56.0, ips: 299.32920 samples/s, eta: 20:38:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:34:15] ppocr INFO: epoch: [360/500], global_step: 162300, lr: 0.000100, edit distance: 0.001449, exp_rate: 0.938159, exp_rate<=1 : 0.975541, exp_rate<=2 : 0.994048, exp_rate<=3 : 0.997465, loss: 0.001359, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18551 s, avg_samples: 56.0, ips: 301.86651 samples/s, eta: 20:38:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:34:34] ppocr INFO: epoch: [360/500], global_step: 162400, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.938617, exp_rate<=1 : 0.975853, exp_rate<=2 : 0.994161, exp_rate<=3 : 0.997605, loss: 0.001840, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18595 s, avg_samples: 56.0, ips: 301.15434 samples/s, eta: 20:38:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:34:52] ppocr INFO: epoch: [360/500], global_step: 162500, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.937409, exp_rate<=1 : 0.975727, exp_rate<=2 : 0.993983, exp_rate<=3 : 0.997584, loss: 0.001915, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18659 s, avg_samples: 56.0, ips: 300.12710 samples/s, eta: 20:37:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:35:11] ppocr INFO: epoch: [360/500], global_step: 162600, lr: 0.000100, edit distance: 0.001453, exp_rate: 0.937385, exp_rate<=1 : 0.975656, exp_rate<=2 : 0.994070, exp_rate<=3 : 0.997698, loss: 0.001123, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18315 s, avg_samples: 56.0, ips: 305.75628 samples/s, eta: 20:37:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:35:30] ppocr INFO: epoch: [360/500], global_step: 162700, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.937868, exp_rate<=1 : 0.975809, exp_rate<=2 : 0.994160, exp_rate<=3 : 0.997730, loss: 0.002151, avg_reader_cost: 0.00017 s, avg_batch_cost: 0.19308 s, avg_samples: 56.0, ips: 290.03903 samples/s, eta: 20:37:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:35:51] ppocr INFO: epoch: [360/500], global_step: 162800, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.936922, exp_rate<=1 : 0.975436, exp_rate<=2 : 0.993877, exp_rate<=3 : 0.997576, loss: 0.001539, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20539 s, avg_samples: 56.0, ips: 272.65706 samples/s, eta: 20:36:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:36:11] ppocr INFO: epoch: [360/500], global_step: 162900, lr: 0.000100, edit distance: 0.001431, exp_rate: 0.936586, exp_rate<=1 : 0.975413, exp_rate<=2 : 0.993879, exp_rate<=3 : 0.997635, loss: 0.001691, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19918 s, avg_samples: 56.0, ips: 281.15437 samples/s, eta: 20:36:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:36:30] ppocr INFO: epoch: [360/500], global_step: 163000, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.936245, exp_rate<=1 : 0.975306, exp_rate<=2 : 0.993842, exp_rate<=3 : 0.997616, loss: 0.001138, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19083 s, avg_samples: 56.0, ips: 293.45527 samples/s, eta: 20:36:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:36:50] ppocr INFO: epoch: [360/500], global_step: 163100, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.935595, exp_rate<=1 : 0.975036, exp_rate<=2 : 0.993799, exp_rate<=3 : 0.997644, loss: 0.001665, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20398 s, avg_samples: 56.0, ips: 274.53198 samples/s, eta: 20:35:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:37:09] ppocr INFO: epoch: [360/500], global_step: 163200, lr: 0.000100, edit distance: 0.001452, exp_rate: 0.934924, exp_rate<=1 : 0.974582, exp_rate<=2 : 0.993754, exp_rate<=3 : 0.997619, loss: 0.001139, avg_reader_cost: 0.00018 s, avg_batch_cost: 0.19275 s, avg_samples: 56.0, ips: 290.53419 samples/s, eta: 20:35:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:37:29] ppocr INFO: epoch: [360/500], global_step: 163300, lr: 0.000100, edit distance: 0.001458, exp_rate: 0.934633, exp_rate<=1 : 0.974641, exp_rate<=2 : 0.993771, exp_rate<=3 : 0.997608, loss: 0.001828, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19537 s, avg_samples: 56.0, ips: 286.63923 samples/s, eta: 20:35:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:37:48] ppocr INFO: epoch: [360/500], global_step: 163400, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.934082, exp_rate<=1 : 0.974546, exp_rate<=2 : 0.993719, exp_rate<=3 : 0.997617, loss: 0.001954, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19037 s, avg_samples: 56.0, ips: 294.16288 samples/s, eta: 20:34:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:38:08] ppocr INFO: epoch: [360/500], global_step: 163500, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.933540, exp_rate<=1 : 0.974304, exp_rate<=2 : 0.993711, exp_rate<=3 : 0.997678, loss: 0.001442, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20180 s, avg_samples: 56.0, ips: 277.50648 samples/s, eta: 20:34:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:38:27] ppocr INFO: epoch: [360/500], global_step: 163600, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.933870, exp_rate<=1 : 0.974424, exp_rate<=2 : 0.993769, exp_rate<=3 : 0.997723, loss: 0.001231, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18960 s, avg_samples: 56.0, ips: 295.36445 samples/s, eta: 20:34:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:38:46] ppocr INFO: epoch: [360/500], global_step: 163700, lr: 0.000100, edit distance: 0.001467, exp_rate: 0.934122, exp_rate<=1 : 0.974482, exp_rate<=2 : 0.993705, exp_rate<=3 : 0.997680, loss: 0.001746, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18709 s, avg_samples: 56.0, ips: 299.31834 samples/s, eta: 20:33:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:39:07] ppocr INFO: epoch: [360/500], global_step: 163800, lr: 0.000100, edit distance: 0.001457, exp_rate: 0.934079, exp_rate<=1 : 0.974422, exp_rate<=2 : 0.993698, exp_rate<=3 : 0.997688, loss: 0.001015, avg_reader_cost: 0.00023 s, avg_batch_cost: 0.20704 s, avg_samples: 56.0, ips: 270.47575 samples/s, eta: 20:33:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:39:25] ppocr INFO: epoch: [360/500], global_step: 163900, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.933995, exp_rate<=1 : 0.974437, exp_rate<=2 : 0.993725, exp_rate<=3 : 0.997688, loss: 0.001997, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18469 s, avg_samples: 56.0, ips: 303.20988 samples/s, eta: 20:33:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:39:45] ppocr INFO: epoch: [360/500], global_step: 164000, lr: 0.000100, edit distance: 0.001470, exp_rate: 0.933520, exp_rate<=1 : 0.974192, exp_rate<=2 : 0.993648, exp_rate<=3 : 0.997618, loss: 0.001831, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19656 s, avg_samples: 56.0, ips: 284.90592 samples/s, eta: 20:33:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:40:03] ppocr INFO: epoch: [360/500], global_step: 164100, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.933836, exp_rate<=1 : 0.974355, exp_rate<=2 : 0.993759, exp_rate<=3 : 0.997671, loss: 0.002572, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18165 s, avg_samples: 56.0, ips: 308.28559 samples/s, eta: 20:32:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:40:23] ppocr INFO: epoch: [360/500], global_step: 164200, lr: 0.000100, edit distance: 0.001476, exp_rate: 0.933839, exp_rate<=1 : 0.974321, exp_rate<=2 : 0.993724, exp_rate<=3 : 0.997631, loss: 0.001341, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20011 s, avg_samples: 56.0, ips: 279.85215 samples/s, eta: 20:32:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:40:42] ppocr INFO: epoch: [360/500], global_step: 164300, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.933768, exp_rate<=1 : 0.974359, exp_rate<=2 : 0.993746, exp_rate<=3 : 0.997658, loss: 0.001981, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19009 s, avg_samples: 56.0, ips: 294.59933 samples/s, eta: 20:32:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:41:00] ppocr INFO: epoch: [360/500], global_step: 164400, lr: 0.000100, edit distance: 0.001475, exp_rate: 0.933718, exp_rate<=1 : 0.974308, exp_rate<=2 : 0.993817, exp_rate<=3 : 0.997692, loss: 0.001643, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17960 s, avg_samples: 56.0, ips: 311.79755 samples/s, eta: 20:31:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:41:02] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 11:41:03] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_360 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 11:41:23] ppocr INFO: epoch: [361/500], global_step: 164500, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.947201, exp_rate<=1 : 0.980417, exp_rate<=2 : 0.995659, exp_rate<=3 : 0.998803, loss: 0.001453, avg_reader_cost: 0.03377 s, avg_batch_cost: 0.22970 s, avg_samples: 56.0, ips: 243.79148 samples/s, eta: 20:31:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:41:43] ppocr INFO: epoch: [361/500], global_step: 164600, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.947154, exp_rate<=1 : 0.979282, exp_rate<=2 : 0.994559, exp_rate<=3 : 0.998270, loss: 0.001455, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20216 s, avg_samples: 56.0, ips: 277.01339 samples/s, eta: 20:31:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:42:04] ppocr INFO: epoch: [361/500], global_step: 164700, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941202, exp_rate<=1 : 0.976160, exp_rate<=2 : 0.993953, exp_rate<=3 : 0.998083, loss: 0.001695, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20884 s, avg_samples: 56.0, ips: 268.15361 samples/s, eta: 20:30:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:42:24] ppocr INFO: epoch: [361/500], global_step: 164800, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.941317, exp_rate<=1 : 0.976481, exp_rate<=2 : 0.993988, exp_rate<=3 : 0.997894, loss: 0.002012, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19200 s, avg_samples: 56.0, ips: 291.66078 samples/s, eta: 20:30:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:42:41] ppocr INFO: epoch: [361/500], global_step: 164900, lr: 0.000100, edit distance: 0.001390, exp_rate: 0.939968, exp_rate<=1 : 0.975555, exp_rate<=2 : 0.993552, exp_rate<=3 : 0.997498, loss: 0.001329, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17926 s, avg_samples: 56.0, ips: 312.39593 samples/s, eta: 20:30:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:43:00] ppocr INFO: epoch: [361/500], global_step: 165000, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.939277, exp_rate<=1 : 0.975422, exp_rate<=2 : 0.993736, exp_rate<=3 : 0.997713, loss: 0.001436, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18870 s, avg_samples: 56.0, ips: 296.77499 samples/s, eta: 20:29:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:43:18] ppocr INFO: epoch: [361/500], global_step: 165100, lr: 0.000100, edit distance: 0.001406, exp_rate: 0.939467, exp_rate<=1 : 0.976105, exp_rate<=2 : 0.994030, exp_rate<=3 : 0.997776, loss: 0.001838, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17604 s, avg_samples: 56.0, ips: 318.10589 samples/s, eta: 20:29:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:43:35] ppocr INFO: epoch: [361/500], global_step: 165200, lr: 0.000100, edit distance: 0.001414, exp_rate: 0.939822, exp_rate<=1 : 0.976492, exp_rate<=2 : 0.994306, exp_rate<=3 : 0.998005, loss: 0.002489, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17098 s, avg_samples: 56.0, ips: 327.52127 samples/s, eta: 20:29:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:43:54] ppocr INFO: epoch: [361/500], global_step: 165300, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.938658, exp_rate<=1 : 0.976251, exp_rate<=2 : 0.994281, exp_rate<=3 : 0.998016, loss: 0.001454, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18851 s, avg_samples: 56.0, ips: 297.06400 samples/s, eta: 20:28:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:44:14] ppocr INFO: epoch: [361/500], global_step: 165400, lr: 0.000100, edit distance: 0.001458, exp_rate: 0.936482, exp_rate<=1 : 0.974950, exp_rate<=2 : 0.993818, exp_rate<=3 : 0.997802, loss: 0.001694, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20142 s, avg_samples: 56.0, ips: 278.02214 samples/s, eta: 20:28:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:44:33] ppocr INFO: epoch: [361/500], global_step: 165500, lr: 0.000100, edit distance: 0.001444, exp_rate: 0.936489, exp_rate<=1 : 0.975038, exp_rate<=2 : 0.993838, exp_rate<=3 : 0.997740, loss: 0.001214, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18610 s, avg_samples: 56.0, ips: 300.91145 samples/s, eta: 20:28:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:44:51] ppocr INFO: epoch: [361/500], global_step: 165600, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.937290, exp_rate<=1 : 0.975517, exp_rate<=2 : 0.993895, exp_rate<=3 : 0.997778, loss: 0.001301, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18348 s, avg_samples: 56.0, ips: 305.20804 samples/s, eta: 20:27:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:45:10] ppocr INFO: epoch: [361/500], global_step: 165700, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.936784, exp_rate<=1 : 0.975409, exp_rate<=2 : 0.993914, exp_rate<=3 : 0.997786, loss: 0.001151, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18958 s, avg_samples: 56.0, ips: 295.38795 samples/s, eta: 20:27:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:45:29] ppocr INFO: epoch: [361/500], global_step: 165800, lr: 0.000100, edit distance: 0.001452, exp_rate: 0.936872, exp_rate<=1 : 0.975452, exp_rate<=2 : 0.993986, exp_rate<=3 : 0.997847, loss: 0.001448, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19040 s, avg_samples: 56.0, ips: 294.11522 samples/s, eta: 20:27:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:45:47] ppocr INFO: epoch: [361/500], global_step: 165900, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.936846, exp_rate<=1 : 0.975628, exp_rate<=2 : 0.994145, exp_rate<=3 : 0.997926, loss: 0.002265, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17764 s, avg_samples: 56.0, ips: 315.24129 samples/s, eta: 20:26:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:46:06] ppocr INFO: epoch: [361/500], global_step: 166000, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.936766, exp_rate<=1 : 0.975530, exp_rate<=2 : 0.994117, exp_rate<=3 : 0.997940, loss: 0.001287, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19188 s, avg_samples: 56.0, ips: 291.85038 samples/s, eta: 20:26:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:46:27] ppocr INFO: epoch: [361/500], global_step: 166100, lr: 0.000100, edit distance: 0.001475, exp_rate: 0.935675, exp_rate<=1 : 0.974889, exp_rate<=2 : 0.993801, exp_rate<=3 : 0.997782, loss: 0.001439, avg_reader_cost: 0.00018 s, avg_batch_cost: 0.20955 s, avg_samples: 56.0, ips: 267.24250 samples/s, eta: 20:26:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:46:47] ppocr INFO: epoch: [361/500], global_step: 166200, lr: 0.000100, edit distance: 0.001480, exp_rate: 0.935483, exp_rate<=1 : 0.974754, exp_rate<=2 : 0.993687, exp_rate<=3 : 0.997729, loss: 0.002007, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19988 s, avg_samples: 56.0, ips: 280.16700 samples/s, eta: 20:25:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:47:07] ppocr INFO: epoch: [361/500], global_step: 166300, lr: 0.000100, edit distance: 0.001486, exp_rate: 0.935186, exp_rate<=1 : 0.974693, exp_rate<=2 : 0.993582, exp_rate<=3 : 0.997692, loss: 0.001696, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20026 s, avg_samples: 56.0, ips: 279.63527 samples/s, eta: 20:25:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:47:26] ppocr INFO: epoch: [361/500], global_step: 166400, lr: 0.000100, edit distance: 0.001512, exp_rate: 0.934867, exp_rate<=1 : 0.974678, exp_rate<=2 : 0.993661, exp_rate<=3 : 0.997730, loss: 0.002383, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18942 s, avg_samples: 56.0, ips: 295.63241 samples/s, eta: 20:25:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:47:47] ppocr INFO: epoch: [361/500], global_step: 166500, lr: 0.000100, edit distance: 0.001513, exp_rate: 0.934041, exp_rate<=1 : 0.974465, exp_rate<=2 : 0.993518, exp_rate<=3 : 0.997612, loss: 0.001001, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20520 s, avg_samples: 56.0, ips: 272.90510 samples/s, eta: 20:25:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:48:08] ppocr INFO: epoch: [361/500], global_step: 166600, lr: 0.000100, edit distance: 0.001496, exp_rate: 0.933940, exp_rate<=1 : 0.974461, exp_rate<=2 : 0.993529, exp_rate<=3 : 0.997650, loss: 0.001505, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20929 s, avg_samples: 56.0, ips: 267.57730 samples/s, eta: 20:24:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:48:26] ppocr INFO: epoch: [361/500], global_step: 166700, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.934292, exp_rate<=1 : 0.974608, exp_rate<=2 : 0.993560, exp_rate<=3 : 0.997683, loss: 0.001419, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18304 s, avg_samples: 56.0, ips: 305.95090 samples/s, eta: 20:24:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:48:45] ppocr INFO: epoch: [361/500], global_step: 166800, lr: 0.000100, edit distance: 0.001485, exp_rate: 0.934189, exp_rate<=1 : 0.974631, exp_rate<=2 : 0.993578, exp_rate<=3 : 0.997690, loss: 0.001924, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18817 s, avg_samples: 56.0, ips: 297.59772 samples/s, eta: 20:24:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:49:03] ppocr INFO: epoch: [361/500], global_step: 166900, lr: 0.000100, edit distance: 0.001487, exp_rate: 0.934091, exp_rate<=1 : 0.974582, exp_rate<=2 : 0.993537, exp_rate<=3 : 0.997670, loss: 0.001886, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18247 s, avg_samples: 56.0, ips: 306.90648 samples/s, eta: 20:23:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:49:21] ppocr INFO: epoch: [361/500], global_step: 167000, lr: 0.000100, edit distance: 0.001498, exp_rate: 0.933945, exp_rate<=1 : 0.974681, exp_rate<=2 : 0.993569, exp_rate<=3 : 0.997677, loss: 0.001643, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18064 s, avg_samples: 56.0, ips: 310.00476 samples/s, eta: 20:23:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:49:41] ppocr INFO: epoch: [361/500], global_step: 167100, lr: 0.000100, edit distance: 0.001498, exp_rate: 0.933433, exp_rate<=1 : 0.974470, exp_rate<=2 : 0.993572, exp_rate<=3 : 0.997697, loss: 0.002228, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19972 s, avg_samples: 56.0, ips: 280.38818 samples/s, eta: 20:23:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:49:48] ppocr INFO: epoch: [361/500], global_step: 167140, lr: 0.000100, edit distance: 0.001492, exp_rate: 0.933557, exp_rate<=1 : 0.974534, exp_rate<=2 : 0.993624, exp_rate<=3 : 0.997718, loss: 0.001188, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.06895 s, avg_samples: 22.4, ips: 324.87450 samples/s, eta: 20:22:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:49:50] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 11:50:02] ppocr INFO: epoch: [362/500], global_step: 167200, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.946429, exp_rate<=1 : 0.977986, exp_rate<=2 : 0.996707, exp_rate<=3 : 0.999570, loss: 0.001989, avg_reader_cost: 0.02778 s, avg_batch_cost: 0.14210 s, avg_samples: 33.6, ips: 236.45081 samples/s, eta: 20:22:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:50:21] ppocr INFO: epoch: [362/500], global_step: 167300, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.944999, exp_rate<=1 : 0.979705, exp_rate<=2 : 0.995550, exp_rate<=3 : 0.998813, loss: 0.001131, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19199 s, avg_samples: 56.0, ips: 291.68798 samples/s, eta: 20:22:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:50:41] ppocr INFO: epoch: [362/500], global_step: 167400, lr: 0.000100, edit distance: 0.001058, exp_rate: 0.947143, exp_rate<=1 : 0.980962, exp_rate<=2 : 0.995392, exp_rate<=3 : 0.998551, loss: 0.001076, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.36482 samples/s, eta: 20:22:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:51:00] ppocr INFO: epoch: [362/500], global_step: 167500, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.942852, exp_rate<=1 : 0.978318, exp_rate<=2 : 0.994820, exp_rate<=3 : 0.998278, loss: 0.001735, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19338 s, avg_samples: 56.0, ips: 289.57968 samples/s, eta: 20:21:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:51:20] ppocr INFO: epoch: [362/500], global_step: 167600, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.943875, exp_rate<=1 : 0.978941, exp_rate<=2 : 0.995149, exp_rate<=3 : 0.998317, loss: 0.001568, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19365 s, avg_samples: 56.0, ips: 289.18234 samples/s, eta: 20:21:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:51:39] ppocr INFO: epoch: [362/500], global_step: 167700, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.942624, exp_rate<=1 : 0.978169, exp_rate<=2 : 0.994957, exp_rate<=3 : 0.998213, loss: 0.001286, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19497 s, avg_samples: 56.0, ips: 287.22697 samples/s, eta: 20:21:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:51:59] ppocr INFO: epoch: [362/500], global_step: 167800, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942078, exp_rate<=1 : 0.978001, exp_rate<=2 : 0.994865, exp_rate<=3 : 0.998042, loss: 0.001353, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19778 s, avg_samples: 56.0, ips: 283.13902 samples/s, eta: 20:20:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:52:17] ppocr INFO: epoch: [362/500], global_step: 167900, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.942595, exp_rate<=1 : 0.978259, exp_rate<=2 : 0.994831, exp_rate<=3 : 0.998052, loss: 0.001612, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18086 s, avg_samples: 56.0, ips: 309.62799 samples/s, eta: 20:20:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:52:35] ppocr INFO: epoch: [362/500], global_step: 168000, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.943157, exp_rate<=1 : 0.978513, exp_rate<=2 : 0.994863, exp_rate<=3 : 0.998120, loss: 0.001031, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18016 s, avg_samples: 56.0, ips: 310.83924 samples/s, eta: 20:20:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:52:55] ppocr INFO: epoch: [362/500], global_step: 168100, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941929, exp_rate<=1 : 0.978107, exp_rate<=2 : 0.994631, exp_rate<=3 : 0.998013, loss: 0.000926, avg_reader_cost: 0.00018 s, avg_batch_cost: 0.19722 s, avg_samples: 56.0, ips: 283.95179 samples/s, eta: 20:19:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:53:11] ppocr INFO: epoch: [362/500], global_step: 168200, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.943271, exp_rate<=1 : 0.978756, exp_rate<=2 : 0.994859, exp_rate<=3 : 0.998157, loss: 0.001666, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16237 s, avg_samples: 56.0, ips: 344.89804 samples/s, eta: 20:19:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:53:30] ppocr INFO: epoch: [362/500], global_step: 168300, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.943235, exp_rate<=1 : 0.978840, exp_rate<=2 : 0.995006, exp_rate<=3 : 0.998215, loss: 0.001442, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18470 s, avg_samples: 56.0, ips: 303.20121 samples/s, eta: 20:19:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:53:49] ppocr INFO: epoch: [362/500], global_step: 168400, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.942539, exp_rate<=1 : 0.978497, exp_rate<=2 : 0.994802, exp_rate<=3 : 0.998125, loss: 0.001804, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19396 s, avg_samples: 56.0, ips: 288.72145 samples/s, eta: 20:18:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:54:08] ppocr INFO: epoch: [362/500], global_step: 168500, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.941661, exp_rate<=1 : 0.978141, exp_rate<=2 : 0.994803, exp_rate<=3 : 0.998191, loss: 0.002602, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18893 s, avg_samples: 56.0, ips: 296.40998 samples/s, eta: 20:18:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:54:27] ppocr INFO: epoch: [362/500], global_step: 168600, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.940495, exp_rate<=1 : 0.977356, exp_rate<=2 : 0.994600, exp_rate<=3 : 0.998139, loss: 0.002353, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19221 s, avg_samples: 56.0, ips: 291.34729 samples/s, eta: 20:18:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:54:46] ppocr INFO: epoch: [362/500], global_step: 168700, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.939361, exp_rate<=1 : 0.976799, exp_rate<=2 : 0.994415, exp_rate<=3 : 0.998040, loss: 0.001843, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18979 s, avg_samples: 56.0, ips: 295.06048 samples/s, eta: 20:17:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:55:05] ppocr INFO: epoch: [362/500], global_step: 168800, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.938998, exp_rate<=1 : 0.976743, exp_rate<=2 : 0.994385, exp_rate<=3 : 0.998020, loss: 0.002275, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19183 s, avg_samples: 56.0, ips: 291.91859 samples/s, eta: 20:17:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:55:25] ppocr INFO: epoch: [362/500], global_step: 168900, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.938280, exp_rate<=1 : 0.976656, exp_rate<=2 : 0.994365, exp_rate<=3 : 0.997997, loss: 0.001556, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19251 s, avg_samples: 56.0, ips: 290.89188 samples/s, eta: 20:17:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:55:44] ppocr INFO: epoch: [362/500], global_step: 169000, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.937604, exp_rate<=1 : 0.976277, exp_rate<=2 : 0.994278, exp_rate<=3 : 0.997947, loss: 0.001919, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18811 s, avg_samples: 56.0, ips: 297.69529 samples/s, eta: 20:16:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:56:05] ppocr INFO: epoch: [362/500], global_step: 169100, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.936497, exp_rate<=1 : 0.975650, exp_rate<=2 : 0.994113, exp_rate<=3 : 0.997883, loss: 0.001397, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21448 s, avg_samples: 56.0, ips: 261.09265 samples/s, eta: 20:16:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:56:25] ppocr INFO: epoch: [362/500], global_step: 169200, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.936356, exp_rate<=1 : 0.975630, exp_rate<=2 : 0.994180, exp_rate<=3 : 0.997894, loss: 0.001642, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19918 s, avg_samples: 56.0, ips: 281.15238 samples/s, eta: 20:16:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:56:45] ppocr INFO: epoch: [362/500], global_step: 169300, lr: 0.000100, edit distance: 0.001441, exp_rate: 0.935991, exp_rate<=1 : 0.975542, exp_rate<=2 : 0.994172, exp_rate<=3 : 0.997900, loss: 0.002265, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19747 s, avg_samples: 56.0, ips: 283.59424 samples/s, eta: 20:15:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:57:05] ppocr INFO: epoch: [362/500], global_step: 169400, lr: 0.000100, edit distance: 0.001460, exp_rate: 0.935284, exp_rate<=1 : 0.975172, exp_rate<=2 : 0.994104, exp_rate<=3 : 0.997897, loss: 0.002087, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20380 s, avg_samples: 56.0, ips: 274.77694 samples/s, eta: 20:15:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:57:24] ppocr INFO: epoch: [362/500], global_step: 169500, lr: 0.000100, edit distance: 0.001477, exp_rate: 0.934789, exp_rate<=1 : 0.974964, exp_rate<=2 : 0.994015, exp_rate<=3 : 0.997895, loss: 0.001716, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19262 s, avg_samples: 56.0, ips: 290.73530 samples/s, eta: 20:15:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:57:44] ppocr INFO: epoch: [362/500], global_step: 169600, lr: 0.000100, edit distance: 0.001500, exp_rate: 0.934275, exp_rate<=1 : 0.974783, exp_rate<=2 : 0.994026, exp_rate<=3 : 0.997901, loss: 0.002389, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19246 s, avg_samples: 56.0, ips: 290.96620 samples/s, eta: 20:15:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:58:03] ppocr INFO: epoch: [362/500], global_step: 169700, lr: 0.000100, edit distance: 0.001507, exp_rate: 0.933721, exp_rate<=1 : 0.974468, exp_rate<=2 : 0.993976, exp_rate<=3 : 0.997889, loss: 0.002320, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19445 s, avg_samples: 56.0, ips: 287.99846 samples/s, eta: 20:14:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:58:22] ppocr INFO: epoch: [362/500], global_step: 169800, lr: 0.000100, edit distance: 0.001502, exp_rate: 0.933459, exp_rate<=1 : 0.974243, exp_rate<=2 : 0.993988, exp_rate<=3 : 0.997889, loss: 0.001226, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19108 s, avg_samples: 56.0, ips: 293.07241 samples/s, eta: 20:14:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:58:38] ppocr INFO: epoch: [362/500], global_step: 169880, lr: 0.000100, edit distance: 0.001502, exp_rate: 0.933249, exp_rate<=1 : 0.974094, exp_rate<=2 : 0.993909, exp_rate<=3 : 0.997875, loss: 0.001518, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.15569 s, avg_samples: 44.8, ips: 287.74834 samples/s, eta: 20:14:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:58:40] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 11:58:44] ppocr INFO: epoch: [363/500], global_step: 169900, lr: 0.000100, edit distance: 0.001625, exp_rate: 0.933748, exp_rate<=1 : 0.973214, exp_rate<=2 : 0.993564, exp_rate<=3 : 0.998569, loss: 0.001722, avg_reader_cost: 0.02608 s, avg_batch_cost: 0.06529 s, avg_samples: 11.2, ips: 171.54157 samples/s, eta: 20:14:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:59:05] ppocr INFO: epoch: [363/500], global_step: 170000, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.943125, exp_rate<=1 : 0.975597, exp_rate<=2 : 0.994839, exp_rate<=3 : 0.998608, loss: 0.001376, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20290 s, avg_samples: 56.0, ips: 275.99730 samples/s, eta: 20:13:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:59:24] ppocr INFO: epoch: [363/500], global_step: 170100, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.943377, exp_rate<=1 : 0.976871, exp_rate<=2 : 0.995349, exp_rate<=3 : 0.998157, loss: 0.001431, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19567 s, avg_samples: 56.0, ips: 286.19115 samples/s, eta: 20:13:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 11:59:44] ppocr INFO: epoch: [363/500], global_step: 170200, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.944887, exp_rate<=1 : 0.977265, exp_rate<=2 : 0.995427, exp_rate<=3 : 0.998211, loss: 0.001605, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19509 s, avg_samples: 56.0, ips: 287.04134 samples/s, eta: 20:13:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:00:02] ppocr INFO: epoch: [363/500], global_step: 170300, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.945641, exp_rate<=1 : 0.978232, exp_rate<=2 : 0.995459, exp_rate<=3 : 0.998390, loss: 0.001660, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18087 s, avg_samples: 56.0, ips: 309.62220 samples/s, eta: 20:12:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:00:22] ppocr INFO: epoch: [363/500], global_step: 170400, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.944031, exp_rate<=1 : 0.977718, exp_rate<=2 : 0.995492, exp_rate<=3 : 0.998451, loss: 0.001232, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20006 s, avg_samples: 56.0, ips: 279.91156 samples/s, eta: 20:12:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:00:41] ppocr INFO: epoch: [363/500], global_step: 170500, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.942699, exp_rate<=1 : 0.977016, exp_rate<=2 : 0.995320, exp_rate<=3 : 0.998362, loss: 0.001253, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19385 s, avg_samples: 56.0, ips: 288.88255 samples/s, eta: 20:12:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:01:02] ppocr INFO: epoch: [363/500], global_step: 170600, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.940050, exp_rate<=1 : 0.976232, exp_rate<=2 : 0.995145, exp_rate<=3 : 0.998174, loss: 0.001925, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20773 s, avg_samples: 56.0, ips: 269.57596 samples/s, eta: 20:12:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:01:20] ppocr INFO: epoch: [363/500], global_step: 170700, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.940953, exp_rate<=1 : 0.976327, exp_rate<=2 : 0.995218, exp_rate<=3 : 0.998215, loss: 0.001807, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18080 s, avg_samples: 56.0, ips: 309.73470 samples/s, eta: 20:11:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:01:39] ppocr INFO: epoch: [363/500], global_step: 170800, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.940847, exp_rate<=1 : 0.976477, exp_rate<=2 : 0.995321, exp_rate<=3 : 0.998278, loss: 0.001574, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18719 s, avg_samples: 56.0, ips: 299.16288 samples/s, eta: 20:11:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:01:59] ppocr INFO: epoch: [363/500], global_step: 170900, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.940740, exp_rate<=1 : 0.976577, exp_rate<=2 : 0.995212, exp_rate<=3 : 0.998182, loss: 0.001538, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19869 s, avg_samples: 56.0, ips: 281.85248 samples/s, eta: 20:11:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:02:18] ppocr INFO: epoch: [363/500], global_step: 171000, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.940225, exp_rate<=1 : 0.976341, exp_rate<=2 : 0.995166, exp_rate<=3 : 0.998159, loss: 0.001458, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19419 s, avg_samples: 56.0, ips: 288.38277 samples/s, eta: 20:10:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:02:37] ppocr INFO: epoch: [363/500], global_step: 171100, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.939535, exp_rate<=1 : 0.976028, exp_rate<=2 : 0.994747, exp_rate<=3 : 0.997873, loss: 0.001620, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18910 s, avg_samples: 56.0, ips: 296.14199 samples/s, eta: 20:10:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:02:56] ppocr INFO: epoch: [363/500], global_step: 171200, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.939136, exp_rate<=1 : 0.975893, exp_rate<=2 : 0.994703, exp_rate<=3 : 0.997854, loss: 0.001860, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19117 s, avg_samples: 56.0, ips: 292.93587 samples/s, eta: 20:10:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:03:16] ppocr INFO: epoch: [363/500], global_step: 171300, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.938387, exp_rate<=1 : 0.975486, exp_rate<=2 : 0.994673, exp_rate<=3 : 0.997848, loss: 0.001605, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.74981 samples/s, eta: 20:09:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:03:35] ppocr INFO: epoch: [363/500], global_step: 171400, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.937903, exp_rate<=1 : 0.975237, exp_rate<=2 : 0.994597, exp_rate<=3 : 0.997860, loss: 0.001501, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18947 s, avg_samples: 56.0, ips: 295.55598 samples/s, eta: 20:09:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:03:55] ppocr INFO: epoch: [363/500], global_step: 171500, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.937747, exp_rate<=1 : 0.975261, exp_rate<=2 : 0.994595, exp_rate<=3 : 0.997926, loss: 0.002183, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20039 s, avg_samples: 56.0, ips: 279.45127 samples/s, eta: 20:09:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:04:14] ppocr INFO: epoch: [363/500], global_step: 171600, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.937175, exp_rate<=1 : 0.974997, exp_rate<=2 : 0.994581, exp_rate<=3 : 0.997904, loss: 0.001948, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19574 s, avg_samples: 56.0, ips: 286.09213 samples/s, eta: 20:08:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:04:33] ppocr INFO: epoch: [363/500], global_step: 171700, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.937152, exp_rate<=1 : 0.975190, exp_rate<=2 : 0.994593, exp_rate<=3 : 0.997866, loss: 0.002567, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18579 s, avg_samples: 56.0, ips: 301.41423 samples/s, eta: 20:08:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:04:52] ppocr INFO: epoch: [363/500], global_step: 171800, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.936459, exp_rate<=1 : 0.974938, exp_rate<=2 : 0.994500, exp_rate<=3 : 0.997814, loss: 0.002355, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19429 s, avg_samples: 56.0, ips: 288.23276 samples/s, eta: 20:08:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:05:12] ppocr INFO: epoch: [363/500], global_step: 171900, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.936529, exp_rate<=1 : 0.974932, exp_rate<=2 : 0.994569, exp_rate<=3 : 0.997874, loss: 0.002099, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19719 s, avg_samples: 56.0, ips: 283.99054 samples/s, eta: 20:07:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:05:31] ppocr INFO: epoch: [363/500], global_step: 172000, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.935996, exp_rate<=1 : 0.974761, exp_rate<=2 : 0.994477, exp_rate<=3 : 0.997819, loss: 0.002044, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18996 s, avg_samples: 56.0, ips: 294.80001 samples/s, eta: 20:07:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:05:50] ppocr INFO: epoch: [363/500], global_step: 172100, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.936008, exp_rate<=1 : 0.974822, exp_rate<=2 : 0.994467, exp_rate<=3 : 0.997892, loss: 0.001558, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18548 s, avg_samples: 56.0, ips: 301.91970 samples/s, eta: 20:07:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:06:09] ppocr INFO: epoch: [363/500], global_step: 172200, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.935648, exp_rate<=1 : 0.974675, exp_rate<=2 : 0.994343, exp_rate<=3 : 0.997832, loss: 0.001479, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19251 s, avg_samples: 56.0, ips: 290.89271 samples/s, eta: 20:06:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:06:28] ppocr INFO: epoch: [363/500], global_step: 172300, lr: 0.000100, edit distance: 0.001423, exp_rate: 0.935324, exp_rate<=1 : 0.974493, exp_rate<=2 : 0.994304, exp_rate<=3 : 0.997846, loss: 0.001838, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19391 s, avg_samples: 56.0, ips: 288.80003 samples/s, eta: 20:06:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:06:47] ppocr INFO: epoch: [363/500], global_step: 172400, lr: 0.000100, edit distance: 0.001431, exp_rate: 0.935026, exp_rate<=1 : 0.974304, exp_rate<=2 : 0.994227, exp_rate<=3 : 0.997802, loss: 0.001693, avg_reader_cost: 0.00021 s, avg_batch_cost: 0.18569 s, avg_samples: 56.0, ips: 301.58054 samples/s, eta: 20:06:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:07:06] ppocr INFO: epoch: [363/500], global_step: 172500, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.934811, exp_rate<=1 : 0.974234, exp_rate<=2 : 0.994183, exp_rate<=3 : 0.997799, loss: 0.001918, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19150 s, avg_samples: 56.0, ips: 292.43108 samples/s, eta: 20:05:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:07:24] ppocr INFO: epoch: [363/500], global_step: 172600, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.934656, exp_rate<=1 : 0.974200, exp_rate<=2 : 0.994204, exp_rate<=3 : 0.997825, loss: 0.001421, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18148 s, avg_samples: 56.0, ips: 308.57271 samples/s, eta: 20:05:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:07:28] ppocr INFO: epoch: [363/500], global_step: 172620, lr: 0.000100, edit distance: 0.001441, exp_rate: 0.934596, exp_rate<=1 : 0.974174, exp_rate<=2 : 0.994194, exp_rate<=3 : 0.997816, loss: 0.001396, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03946 s, avg_samples: 11.2, ips: 283.84138 samples/s, eta: 20:05:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:07:30] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 12:07:46] ppocr INFO: epoch: [364/500], global_step: 172700, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.939715, exp_rate<=1 : 0.974392, exp_rate<=2 : 0.994006, exp_rate<=3 : 0.997291, loss: 0.001402, avg_reader_cost: 0.02680 s, avg_batch_cost: 0.17696 s, avg_samples: 44.8, ips: 253.16436 samples/s, eta: 20:05:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:08:06] ppocr INFO: epoch: [364/500], global_step: 172800, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.941238, exp_rate<=1 : 0.976278, exp_rate<=2 : 0.994325, exp_rate<=3 : 0.997486, loss: 0.001134, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20236 s, avg_samples: 56.0, ips: 276.73482 samples/s, eta: 20:04:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:08:26] ppocr INFO: epoch: [364/500], global_step: 172900, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.941555, exp_rate<=1 : 0.976884, exp_rate<=2 : 0.994487, exp_rate<=3 : 0.997821, loss: 0.001561, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19491 s, avg_samples: 56.0, ips: 287.30502 samples/s, eta: 20:04:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:08:44] ppocr INFO: epoch: [364/500], global_step: 173000, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.942278, exp_rate<=1 : 0.977456, exp_rate<=2 : 0.994936, exp_rate<=3 : 0.998167, loss: 0.001826, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18569 s, avg_samples: 56.0, ips: 301.57235 samples/s, eta: 20:04:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:09:04] ppocr INFO: epoch: [364/500], global_step: 173100, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.941639, exp_rate<=1 : 0.977328, exp_rate<=2 : 0.995166, exp_rate<=3 : 0.998282, loss: 0.001273, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19829 s, avg_samples: 56.0, ips: 282.41201 samples/s, eta: 20:03:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:09:22] ppocr INFO: epoch: [364/500], global_step: 173200, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.941481, exp_rate<=1 : 0.977498, exp_rate<=2 : 0.995226, exp_rate<=3 : 0.998293, loss: 0.001989, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17543 s, avg_samples: 56.0, ips: 319.21353 samples/s, eta: 20:03:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:09:41] ppocr INFO: epoch: [364/500], global_step: 173300, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.940494, exp_rate<=1 : 0.977008, exp_rate<=2 : 0.994613, exp_rate<=3 : 0.998075, loss: 0.001327, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19721 s, avg_samples: 56.0, ips: 283.95972 samples/s, eta: 20:03:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:10:01] ppocr INFO: epoch: [364/500], global_step: 173400, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.941009, exp_rate<=1 : 0.977357, exp_rate<=2 : 0.994702, exp_rate<=3 : 0.998109, loss: 0.001528, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19016 s, avg_samples: 56.0, ips: 294.48810 samples/s, eta: 20:02:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:10:20] ppocr INFO: epoch: [364/500], global_step: 173500, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.940160, exp_rate<=1 : 0.976743, exp_rate<=2 : 0.994718, exp_rate<=3 : 0.998026, loss: 0.001108, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19560 s, avg_samples: 56.0, ips: 286.29401 samples/s, eta: 20:02:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:10:40] ppocr INFO: epoch: [364/500], global_step: 173600, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.939153, exp_rate<=1 : 0.976148, exp_rate<=2 : 0.994530, exp_rate<=3 : 0.997951, loss: 0.002060, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19613 s, avg_samples: 56.0, ips: 285.52438 samples/s, eta: 20:02:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:10:58] ppocr INFO: epoch: [364/500], global_step: 173700, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.939488, exp_rate<=1 : 0.976019, exp_rate<=2 : 0.994453, exp_rate<=3 : 0.997970, loss: 0.001488, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17829 s, avg_samples: 56.0, ips: 314.09023 samples/s, eta: 20:01:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:11:19] ppocr INFO: epoch: [364/500], global_step: 173800, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.939359, exp_rate<=1 : 0.975765, exp_rate<=2 : 0.994404, exp_rate<=3 : 0.997948, loss: 0.001715, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21037 s, avg_samples: 56.0, ips: 266.19820 samples/s, eta: 20:01:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:11:37] ppocr INFO: epoch: [364/500], global_step: 173900, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.939590, exp_rate<=1 : 0.975930, exp_rate<=2 : 0.994436, exp_rate<=3 : 0.997951, loss: 0.001192, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18216 s, avg_samples: 56.0, ips: 307.42294 samples/s, eta: 20:01:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:11:56] ppocr INFO: epoch: [364/500], global_step: 174000, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.939631, exp_rate<=1 : 0.975784, exp_rate<=2 : 0.994497, exp_rate<=3 : 0.998001, loss: 0.001059, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19086 s, avg_samples: 56.0, ips: 293.41373 samples/s, eta: 20:01:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:12:15] ppocr INFO: epoch: [364/500], global_step: 174100, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.940022, exp_rate<=1 : 0.976085, exp_rate<=2 : 0.994642, exp_rate<=3 : 0.998081, loss: 0.001138, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18954 s, avg_samples: 56.0, ips: 295.44436 samples/s, eta: 20:00:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:12:35] ppocr INFO: epoch: [364/500], global_step: 174200, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.939767, exp_rate<=1 : 0.975947, exp_rate<=2 : 0.994643, exp_rate<=3 : 0.998124, loss: 0.002285, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20176 s, avg_samples: 56.0, ips: 277.55803 samples/s, eta: 20:00:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:12:53] ppocr INFO: epoch: [364/500], global_step: 174300, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.939895, exp_rate<=1 : 0.976075, exp_rate<=2 : 0.994706, exp_rate<=3 : 0.998184, loss: 0.001475, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17779 s, avg_samples: 56.0, ips: 314.97960 samples/s, eta: 20:00:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:13:13] ppocr INFO: epoch: [364/500], global_step: 174400, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.938650, exp_rate<=1 : 0.975430, exp_rate<=2 : 0.994475, exp_rate<=3 : 0.998060, loss: 0.002019, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19876 s, avg_samples: 56.0, ips: 281.74531 samples/s, eta: 19:59:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:13:32] ppocr INFO: epoch: [364/500], global_step: 174500, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.938466, exp_rate<=1 : 0.975354, exp_rate<=2 : 0.994407, exp_rate<=3 : 0.998084, loss: 0.001600, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19054 s, avg_samples: 56.0, ips: 293.90259 samples/s, eta: 19:59:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:13:52] ppocr INFO: epoch: [364/500], global_step: 174600, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.938216, exp_rate<=1 : 0.975117, exp_rate<=2 : 0.994324, exp_rate<=3 : 0.998005, loss: 0.001652, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19895 s, avg_samples: 56.0, ips: 281.48434 samples/s, eta: 19:59:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:14:11] ppocr INFO: epoch: [364/500], global_step: 174700, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.937938, exp_rate<=1 : 0.974983, exp_rate<=2 : 0.994294, exp_rate<=3 : 0.997989, loss: 0.001626, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19540 s, avg_samples: 56.0, ips: 286.58470 samples/s, eta: 19:58:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:14:30] ppocr INFO: epoch: [364/500], global_step: 174800, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.937121, exp_rate<=1 : 0.974638, exp_rate<=2 : 0.994176, exp_rate<=3 : 0.997899, loss: 0.002368, avg_reader_cost: 0.00017 s, avg_batch_cost: 0.19027 s, avg_samples: 56.0, ips: 294.32171 samples/s, eta: 19:58:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:14:49] ppocr INFO: epoch: [364/500], global_step: 174900, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.937017, exp_rate<=1 : 0.974738, exp_rate<=2 : 0.994192, exp_rate<=3 : 0.997941, loss: 0.002144, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18520 s, avg_samples: 56.0, ips: 302.37378 samples/s, eta: 19:58:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:15:08] ppocr INFO: epoch: [364/500], global_step: 175000, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.936662, exp_rate<=1 : 0.974812, exp_rate<=2 : 0.994166, exp_rate<=3 : 0.997954, loss: 0.002113, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19035 s, avg_samples: 56.0, ips: 294.19905 samples/s, eta: 19:57:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:15:27] ppocr INFO: epoch: [364/500], global_step: 175100, lr: 0.000100, edit distance: 0.001372, exp_rate: 0.936432, exp_rate<=1 : 0.974745, exp_rate<=2 : 0.994150, exp_rate<=3 : 0.997961, loss: 0.001835, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19323 s, avg_samples: 56.0, ips: 289.80633 samples/s, eta: 19:57:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:15:46] ppocr INFO: epoch: [364/500], global_step: 175200, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.936165, exp_rate<=1 : 0.974624, exp_rate<=2 : 0.994088, exp_rate<=3 : 0.997933, loss: 0.001812, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18326 s, avg_samples: 56.0, ips: 305.57551 samples/s, eta: 19:57:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:16:06] ppocr INFO: epoch: [364/500], global_step: 175300, lr: 0.000100, edit distance: 0.001387, exp_rate: 0.935833, exp_rate<=1 : 0.974534, exp_rate<=2 : 0.994067, exp_rate<=3 : 0.997909, loss: 0.001796, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19875 s, avg_samples: 56.0, ips: 281.76750 samples/s, eta: 19:56:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:16:18] ppocr INFO: epoch: [364/500], global_step: 175360, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.935523, exp_rate<=1 : 0.974299, exp_rate<=2 : 0.994016, exp_rate<=3 : 0.997865, loss: 0.001169, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11933 s, avg_samples: 33.6, ips: 281.56366 samples/s, eta: 19:56:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:16:20] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 12:16:29] ppocr INFO: epoch: [365/500], global_step: 175400, lr: 0.000100, edit distance: 0.001015, exp_rate: 0.931398, exp_rate<=1 : 0.972256, exp_rate<=2 : 0.993378, exp_rate<=3 : 0.997657, loss: 0.001684, avg_reader_cost: 0.02957 s, avg_batch_cost: 0.11507 s, avg_samples: 22.4, ips: 194.65874 samples/s, eta: 19:56:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:16:47] ppocr INFO: epoch: [365/500], global_step: 175500, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.941359, exp_rate<=1 : 0.978085, exp_rate<=2 : 0.994026, exp_rate<=3 : 0.998370, loss: 0.001898, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18195 s, avg_samples: 56.0, ips: 307.78205 samples/s, eta: 19:56:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:17:07] ppocr INFO: epoch: [365/500], global_step: 175600, lr: 0.000100, edit distance: 0.001095, exp_rate: 0.943370, exp_rate<=1 : 0.979393, exp_rate<=2 : 0.994809, exp_rate<=3 : 0.998334, loss: 0.001196, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19299 s, avg_samples: 56.0, ips: 290.16391 samples/s, eta: 19:55:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:17:27] ppocr INFO: epoch: [365/500], global_step: 175700, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.943379, exp_rate<=1 : 0.978664, exp_rate<=2 : 0.994702, exp_rate<=3 : 0.998231, loss: 0.001399, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19945 s, avg_samples: 56.0, ips: 280.77333 samples/s, eta: 19:55:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:17:46] ppocr INFO: epoch: [365/500], global_step: 175800, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.941324, exp_rate<=1 : 0.977772, exp_rate<=2 : 0.994780, exp_rate<=3 : 0.998330, loss: 0.001579, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19638 s, avg_samples: 56.0, ips: 285.16488 samples/s, eta: 19:55:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:18:06] ppocr INFO: epoch: [365/500], global_step: 175900, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.939776, exp_rate<=1 : 0.977535, exp_rate<=2 : 0.994719, exp_rate<=3 : 0.998367, loss: 0.002294, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19348 s, avg_samples: 56.0, ips: 289.43349 samples/s, eta: 19:55:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:18:25] ppocr INFO: epoch: [365/500], global_step: 176000, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.940151, exp_rate<=1 : 0.977568, exp_rate<=2 : 0.994743, exp_rate<=3 : 0.998442, loss: 0.001577, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19147 s, avg_samples: 56.0, ips: 292.47491 samples/s, eta: 19:54:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:18:44] ppocr INFO: epoch: [365/500], global_step: 176100, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.938348, exp_rate<=1 : 0.976700, exp_rate<=2 : 0.994512, exp_rate<=3 : 0.998182, loss: 0.001665, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19395 s, avg_samples: 56.0, ips: 288.72838 samples/s, eta: 19:54:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:19:03] ppocr INFO: epoch: [365/500], global_step: 176200, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.938626, exp_rate<=1 : 0.976785, exp_rate<=2 : 0.994619, exp_rate<=3 : 0.998220, loss: 0.001451, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18543 s, avg_samples: 56.0, ips: 302.00172 samples/s, eta: 19:54:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:19:21] ppocr INFO: epoch: [365/500], global_step: 176300, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.938259, exp_rate<=1 : 0.976429, exp_rate<=2 : 0.994596, exp_rate<=3 : 0.998208, loss: 0.001557, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18334 s, avg_samples: 56.0, ips: 305.44847 samples/s, eta: 19:53:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:19:40] ppocr INFO: epoch: [365/500], global_step: 176400, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.938353, exp_rate<=1 : 0.976265, exp_rate<=2 : 0.994570, exp_rate<=3 : 0.998177, loss: 0.001358, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19018 s, avg_samples: 56.0, ips: 294.46245 samples/s, eta: 19:53:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:19:59] ppocr INFO: epoch: [365/500], global_step: 176500, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.938686, exp_rate<=1 : 0.976478, exp_rate<=2 : 0.994635, exp_rate<=3 : 0.998231, loss: 0.001104, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19212 s, avg_samples: 56.0, ips: 291.49043 samples/s, eta: 19:53:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:20:19] ppocr INFO: epoch: [365/500], global_step: 176600, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.938451, exp_rate<=1 : 0.976372, exp_rate<=2 : 0.994609, exp_rate<=3 : 0.998213, loss: 0.001668, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19852 s, avg_samples: 56.0, ips: 282.08891 samples/s, eta: 19:52:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:20:38] ppocr INFO: epoch: [365/500], global_step: 176700, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.937859, exp_rate<=1 : 0.976226, exp_rate<=2 : 0.994554, exp_rate<=3 : 0.998186, loss: 0.001441, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18624 s, avg_samples: 56.0, ips: 300.68858 samples/s, eta: 19:52:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:20:57] ppocr INFO: epoch: [365/500], global_step: 176800, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.937331, exp_rate<=1 : 0.976095, exp_rate<=2 : 0.994418, exp_rate<=3 : 0.998115, loss: 0.001863, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18861 s, avg_samples: 56.0, ips: 296.90910 samples/s, eta: 19:52:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:21:15] ppocr INFO: epoch: [365/500], global_step: 176900, lr: 0.000100, edit distance: 0.001341, exp_rate: 0.937076, exp_rate<=1 : 0.976227, exp_rate<=2 : 0.994526, exp_rate<=3 : 0.998132, loss: 0.002139, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18558 s, avg_samples: 56.0, ips: 301.75798 samples/s, eta: 19:51:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:21:35] ppocr INFO: epoch: [365/500], global_step: 177000, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.936654, exp_rate<=1 : 0.976108, exp_rate<=2 : 0.994546, exp_rate<=3 : 0.998169, loss: 0.001349, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19797 s, avg_samples: 56.0, ips: 282.86588 samples/s, eta: 19:51:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:21:54] ppocr INFO: epoch: [365/500], global_step: 177100, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.935975, exp_rate<=1 : 0.975669, exp_rate<=2 : 0.994442, exp_rate<=3 : 0.998122, loss: 0.001744, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19240 s, avg_samples: 56.0, ips: 291.05378 samples/s, eta: 19:51:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:22:14] ppocr INFO: epoch: [365/500], global_step: 177200, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.935582, exp_rate<=1 : 0.975569, exp_rate<=2 : 0.994436, exp_rate<=3 : 0.998131, loss: 0.001463, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19276 s, avg_samples: 56.0, ips: 290.52225 samples/s, eta: 19:50:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:22:34] ppocr INFO: epoch: [365/500], global_step: 177300, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.935252, exp_rate<=1 : 0.975432, exp_rate<=2 : 0.994313, exp_rate<=3 : 0.998116, loss: 0.001348, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19698 s, avg_samples: 56.0, ips: 284.29606 samples/s, eta: 19:50:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:22:53] ppocr INFO: epoch: [365/500], global_step: 177400, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.935282, exp_rate<=1 : 0.975369, exp_rate<=2 : 0.994332, exp_rate<=3 : 0.998156, loss: 0.002257, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19073 s, avg_samples: 56.0, ips: 293.60424 samples/s, eta: 19:50:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:23:12] ppocr INFO: epoch: [365/500], global_step: 177500, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.934980, exp_rate<=1 : 0.975198, exp_rate<=2 : 0.994340, exp_rate<=3 : 0.998186, loss: 0.001481, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18884 s, avg_samples: 56.0, ips: 296.54581 samples/s, eta: 19:49:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:23:30] ppocr INFO: epoch: [365/500], global_step: 177600, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.935009, exp_rate<=1 : 0.975112, exp_rate<=2 : 0.994340, exp_rate<=3 : 0.998191, loss: 0.001737, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18120 s, avg_samples: 56.0, ips: 309.04797 samples/s, eta: 19:49:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:23:48] ppocr INFO: epoch: [365/500], global_step: 177700, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.934862, exp_rate<=1 : 0.975040, exp_rate<=2 : 0.994315, exp_rate<=3 : 0.998144, loss: 0.001891, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18699 s, avg_samples: 56.0, ips: 299.48257 samples/s, eta: 19:49:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:24:07] ppocr INFO: epoch: [365/500], global_step: 177800, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.934504, exp_rate<=1 : 0.974856, exp_rate<=2 : 0.994270, exp_rate<=3 : 0.998123, loss: 0.001471, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18935 s, avg_samples: 56.0, ips: 295.74570 samples/s, eta: 19:48:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:24:25] ppocr INFO: epoch: [365/500], global_step: 177900, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.934304, exp_rate<=1 : 0.974637, exp_rate<=2 : 0.994163, exp_rate<=3 : 0.998075, loss: 0.001237, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.17669 s, avg_samples: 56.0, ips: 316.93426 samples/s, eta: 19:48:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:24:45] ppocr INFO: epoch: [365/500], global_step: 178000, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.934126, exp_rate<=1 : 0.974518, exp_rate<=2 : 0.994100, exp_rate<=3 : 0.998053, loss: 0.001531, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19740 s, avg_samples: 56.0, ips: 283.69279 samples/s, eta: 19:48:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:25:03] ppocr INFO: epoch: [365/500], global_step: 178100, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.934158, exp_rate<=1 : 0.974557, exp_rate<=2 : 0.994117, exp_rate<=3 : 0.998064, loss: 0.002374, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17844 s, avg_samples: 56.0, ips: 313.83256 samples/s, eta: 19:47:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:25:05] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 12:25:05] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_365 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 12:25:25] ppocr INFO: epoch: [366/500], global_step: 178200, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.940819, exp_rate<=1 : 0.979151, exp_rate<=2 : 0.993338, exp_rate<=3 : 0.997224, loss: 0.001821, avg_reader_cost: 0.03228 s, avg_batch_cost: 0.22203 s, avg_samples: 56.0, ips: 252.21790 samples/s, eta: 19:47:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:25:44] ppocr INFO: epoch: [366/500], global_step: 178300, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.940446, exp_rate<=1 : 0.977478, exp_rate<=2 : 0.993626, exp_rate<=3 : 0.997016, loss: 0.001032, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18733 s, avg_samples: 56.0, ips: 298.93727 samples/s, eta: 19:47:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:26:03] ppocr INFO: epoch: [366/500], global_step: 178400, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.944618, exp_rate<=1 : 0.978818, exp_rate<=2 : 0.994591, exp_rate<=3 : 0.997722, loss: 0.000921, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19026 s, avg_samples: 56.0, ips: 294.33128 samples/s, eta: 19:46:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:26:22] ppocr INFO: epoch: [366/500], global_step: 178500, lr: 0.000100, edit distance: 0.001206, exp_rate: 0.943779, exp_rate<=1 : 0.978251, exp_rate<=2 : 0.994551, exp_rate<=3 : 0.997714, loss: 0.001058, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18877 s, avg_samples: 56.0, ips: 296.65843 samples/s, eta: 19:46:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:26:40] ppocr INFO: epoch: [366/500], global_step: 178600, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.944782, exp_rate<=1 : 0.978790, exp_rate<=2 : 0.994856, exp_rate<=3 : 0.998014, loss: 0.001462, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18317 s, avg_samples: 56.0, ips: 305.71999 samples/s, eta: 19:46:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:26:59] ppocr INFO: epoch: [366/500], global_step: 178700, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.944511, exp_rate<=1 : 0.978663, exp_rate<=2 : 0.994829, exp_rate<=3 : 0.998155, loss: 0.001112, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18711 s, avg_samples: 56.0, ips: 299.28771 samples/s, eta: 19:45:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:27:19] ppocr INFO: epoch: [366/500], global_step: 178800, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.943759, exp_rate<=1 : 0.978527, exp_rate<=2 : 0.994617, exp_rate<=3 : 0.998057, loss: 0.001088, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19922 s, avg_samples: 56.0, ips: 281.09719 samples/s, eta: 19:45:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:27:37] ppocr INFO: epoch: [366/500], global_step: 178900, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.943167, exp_rate<=1 : 0.978344, exp_rate<=2 : 0.994547, exp_rate<=3 : 0.998020, loss: 0.001554, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17988 s, avg_samples: 56.0, ips: 311.31106 samples/s, eta: 19:45:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:27:54] ppocr INFO: epoch: [366/500], global_step: 179000, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.942627, exp_rate<=1 : 0.978137, exp_rate<=2 : 0.994604, exp_rate<=3 : 0.998132, loss: 0.001462, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17826 s, avg_samples: 56.0, ips: 314.14481 samples/s, eta: 19:44:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:28:12] ppocr INFO: epoch: [366/500], global_step: 179100, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.942306, exp_rate<=1 : 0.977681, exp_rate<=2 : 0.994757, exp_rate<=3 : 0.998128, loss: 0.001771, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17302 s, avg_samples: 56.0, ips: 323.66080 samples/s, eta: 19:44:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:28:32] ppocr INFO: epoch: [366/500], global_step: 179200, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.940829, exp_rate<=1 : 0.976974, exp_rate<=2 : 0.994600, exp_rate<=3 : 0.998186, loss: 0.001903, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19878 s, avg_samples: 56.0, ips: 281.71759 samples/s, eta: 19:44:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:28:52] ppocr INFO: epoch: [366/500], global_step: 179300, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.940057, exp_rate<=1 : 0.976863, exp_rate<=2 : 0.994507, exp_rate<=3 : 0.998180, loss: 0.001908, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19903 s, avg_samples: 56.0, ips: 281.36076 samples/s, eta: 19:43:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:29:10] ppocr INFO: epoch: [366/500], global_step: 179400, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.939566, exp_rate<=1 : 0.976778, exp_rate<=2 : 0.994424, exp_rate<=3 : 0.998201, loss: 0.001087, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18835 s, avg_samples: 56.0, ips: 297.31390 samples/s, eta: 19:43:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:29:30] ppocr INFO: epoch: [366/500], global_step: 179500, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.939148, exp_rate<=1 : 0.976649, exp_rate<=2 : 0.994362, exp_rate<=3 : 0.998175, loss: 0.001131, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19136 s, avg_samples: 56.0, ips: 292.63831 samples/s, eta: 19:43:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:29:48] ppocr INFO: epoch: [366/500], global_step: 179600, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.939273, exp_rate<=1 : 0.976641, exp_rate<=2 : 0.994447, exp_rate<=3 : 0.998181, loss: 0.002270, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17976 s, avg_samples: 56.0, ips: 311.52471 samples/s, eta: 19:42:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:30:07] ppocr INFO: epoch: [366/500], global_step: 179700, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.938108, exp_rate<=1 : 0.976099, exp_rate<=2 : 0.994436, exp_rate<=3 : 0.998187, loss: 0.001720, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19476 s, avg_samples: 56.0, ips: 287.53140 samples/s, eta: 19:42:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:30:27] ppocr INFO: epoch: [366/500], global_step: 179800, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.937381, exp_rate<=1 : 0.975706, exp_rate<=2 : 0.994361, exp_rate<=3 : 0.998172, loss: 0.001520, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19883 s, avg_samples: 56.0, ips: 281.64109 samples/s, eta: 19:42:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:30:47] ppocr INFO: epoch: [366/500], global_step: 179900, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.936969, exp_rate<=1 : 0.975452, exp_rate<=2 : 0.994258, exp_rate<=3 : 0.998116, loss: 0.001886, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19873 s, avg_samples: 56.0, ips: 281.79235 samples/s, eta: 19:41:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:31:05] ppocr INFO: epoch: [366/500], global_step: 180000, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.937275, exp_rate<=1 : 0.975314, exp_rate<=2 : 0.994227, exp_rate<=3 : 0.998127, loss: 0.001891, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18263 s, avg_samples: 56.0, ips: 306.63907 samples/s, eta: 19:41:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:31:26] ppocr INFO: epoch: [366/500], global_step: 180100, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.936002, exp_rate<=1 : 0.974871, exp_rate<=2 : 0.994081, exp_rate<=3 : 0.998054, loss: 0.001799, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20640 s, avg_samples: 56.0, ips: 271.31509 samples/s, eta: 19:41:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:31:44] ppocr INFO: epoch: [366/500], global_step: 180200, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.935841, exp_rate<=1 : 0.974791, exp_rate<=2 : 0.994042, exp_rate<=3 : 0.998014, loss: 0.001572, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18064 s, avg_samples: 56.0, ips: 310.01116 samples/s, eta: 19:40:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:32:04] ppocr INFO: epoch: [366/500], global_step: 180300, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.935084, exp_rate<=1 : 0.974410, exp_rate<=2 : 0.993867, exp_rate<=3 : 0.997983, loss: 0.002112, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20428 s, avg_samples: 56.0, ips: 274.13530 samples/s, eta: 19:40:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:32:24] ppocr INFO: epoch: [366/500], global_step: 180400, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.934854, exp_rate<=1 : 0.974294, exp_rate<=2 : 0.993723, exp_rate<=3 : 0.997935, loss: 0.001840, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19692 s, avg_samples: 56.0, ips: 284.37838 samples/s, eta: 19:40:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:32:41] ppocr INFO: epoch: [366/500], global_step: 180500, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.934674, exp_rate<=1 : 0.974242, exp_rate<=2 : 0.993690, exp_rate<=3 : 0.997909, loss: 0.002104, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16927 s, avg_samples: 56.0, ips: 330.82757 samples/s, eta: 19:39:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:33:02] ppocr INFO: epoch: [366/500], global_step: 180600, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.934077, exp_rate<=1 : 0.974050, exp_rate<=2 : 0.993654, exp_rate<=3 : 0.997873, loss: 0.002024, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20499 s, avg_samples: 56.0, ips: 273.18054 samples/s, eta: 19:39:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:33:19] ppocr INFO: epoch: [366/500], global_step: 180700, lr: 0.000100, edit distance: 0.001466, exp_rate: 0.934019, exp_rate<=1 : 0.973972, exp_rate<=2 : 0.993654, exp_rate<=3 : 0.997880, loss: 0.001668, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17687 s, avg_samples: 56.0, ips: 316.62375 samples/s, eta: 19:39:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:33:38] ppocr INFO: epoch: [366/500], global_step: 180800, lr: 0.000100, edit distance: 0.001471, exp_rate: 0.933744, exp_rate<=1 : 0.973815, exp_rate<=2 : 0.993657, exp_rate<=3 : 0.997895, loss: 0.002112, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18238 s, avg_samples: 56.0, ips: 307.05003 samples/s, eta: 19:38:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:33:44] ppocr INFO: epoch: [366/500], global_step: 180840, lr: 0.000100, edit distance: 0.001475, exp_rate: 0.933628, exp_rate<=1 : 0.973774, exp_rate<=2 : 0.993669, exp_rate<=3 : 0.997916, loss: 0.002608, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.06718 s, avg_samples: 22.4, ips: 333.43765 samples/s, eta: 19:38:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:33:47] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 12:33:58] ppocr INFO: epoch: [367/500], global_step: 180900, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.939965, exp_rate<=1 : 0.977410, exp_rate<=2 : 0.994873, exp_rate<=3 : 0.998751, loss: 0.001446, avg_reader_cost: 0.02720 s, avg_batch_cost: 0.13843 s, avg_samples: 33.6, ips: 242.72980 samples/s, eta: 19:38:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:34:16] ppocr INFO: epoch: [367/500], global_step: 181000, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.939913, exp_rate<=1 : 0.975964, exp_rate<=2 : 0.994765, exp_rate<=3 : 0.998546, loss: 0.001527, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18226 s, avg_samples: 56.0, ips: 307.25420 samples/s, eta: 19:38:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:34:35] ppocr INFO: epoch: [367/500], global_step: 181100, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.943995, exp_rate<=1 : 0.977492, exp_rate<=2 : 0.995030, exp_rate<=3 : 0.998432, loss: 0.001092, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18187 s, avg_samples: 56.0, ips: 307.90794 samples/s, eta: 19:37:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:34:53] ppocr INFO: epoch: [367/500], global_step: 181200, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.944377, exp_rate<=1 : 0.978373, exp_rate<=2 : 0.995159, exp_rate<=3 : 0.998413, loss: 0.001469, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18428 s, avg_samples: 56.0, ips: 303.89213 samples/s, eta: 19:37:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:35:11] ppocr INFO: epoch: [367/500], global_step: 181300, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.944306, exp_rate<=1 : 0.978357, exp_rate<=2 : 0.995298, exp_rate<=3 : 0.998434, loss: 0.002640, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18276 s, avg_samples: 56.0, ips: 306.41410 samples/s, eta: 19:37:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:35:31] ppocr INFO: epoch: [367/500], global_step: 181400, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.941628, exp_rate<=1 : 0.977590, exp_rate<=2 : 0.994896, exp_rate<=3 : 0.998216, loss: 0.001498, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19218 s, avg_samples: 56.0, ips: 291.38798 samples/s, eta: 19:36:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:35:50] ppocr INFO: epoch: [367/500], global_step: 181500, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.939944, exp_rate<=1 : 0.976892, exp_rate<=2 : 0.994675, exp_rate<=3 : 0.998125, loss: 0.001666, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19896 s, avg_samples: 56.0, ips: 281.45767 samples/s, eta: 19:36:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:36:10] ppocr INFO: epoch: [367/500], global_step: 181600, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.940154, exp_rate<=1 : 0.976567, exp_rate<=2 : 0.994627, exp_rate<=3 : 0.998168, loss: 0.001205, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19029 s, avg_samples: 56.0, ips: 294.29535 samples/s, eta: 19:36:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:36:28] ppocr INFO: epoch: [367/500], global_step: 181700, lr: 0.000100, edit distance: 0.001359, exp_rate: 0.940413, exp_rate<=1 : 0.976673, exp_rate<=2 : 0.994693, exp_rate<=3 : 0.998217, loss: 0.001489, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18961 s, avg_samples: 56.0, ips: 295.34341 samples/s, eta: 19:35:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:36:46] ppocr INFO: epoch: [367/500], global_step: 181800, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.939677, exp_rate<=1 : 0.976755, exp_rate<=2 : 0.994776, exp_rate<=3 : 0.998178, loss: 0.001727, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17985 s, avg_samples: 56.0, ips: 311.36962 samples/s, eta: 19:35:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:37:04] ppocr INFO: epoch: [367/500], global_step: 181900, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.939087, exp_rate<=1 : 0.976684, exp_rate<=2 : 0.994820, exp_rate<=3 : 0.998151, loss: 0.001280, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17803 s, avg_samples: 56.0, ips: 314.54949 samples/s, eta: 19:35:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:37:24] ppocr INFO: epoch: [367/500], global_step: 182000, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.938305, exp_rate<=1 : 0.975891, exp_rate<=2 : 0.994433, exp_rate<=3 : 0.998033, loss: 0.002049, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19314 s, avg_samples: 56.0, ips: 289.94179 samples/s, eta: 19:34:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:37:42] ppocr INFO: epoch: [367/500], global_step: 182100, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.938445, exp_rate<=1 : 0.975990, exp_rate<=2 : 0.994401, exp_rate<=3 : 0.997979, loss: 0.001921, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18391 s, avg_samples: 56.0, ips: 304.50454 samples/s, eta: 19:34:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:38:00] ppocr INFO: epoch: [367/500], global_step: 182200, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.938145, exp_rate<=1 : 0.975595, exp_rate<=2 : 0.994329, exp_rate<=3 : 0.997943, loss: 0.001382, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18159 s, avg_samples: 56.0, ips: 308.37858 samples/s, eta: 19:34:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:38:18] ppocr INFO: epoch: [367/500], global_step: 182300, lr: 0.000100, edit distance: 0.001419, exp_rate: 0.937958, exp_rate<=1 : 0.975610, exp_rate<=2 : 0.994352, exp_rate<=3 : 0.997958, loss: 0.001022, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17908 s, avg_samples: 56.0, ips: 312.71701 samples/s, eta: 19:33:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:38:40] ppocr INFO: epoch: [367/500], global_step: 182400, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.936767, exp_rate<=1 : 0.975061, exp_rate<=2 : 0.994126, exp_rate<=3 : 0.997768, loss: 0.001567, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.22013 s, avg_samples: 56.0, ips: 254.39709 samples/s, eta: 19:33:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:39:01] ppocr INFO: epoch: [367/500], global_step: 182500, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.936012, exp_rate<=1 : 0.974685, exp_rate<=2 : 0.994060, exp_rate<=3 : 0.997795, loss: 0.001343, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20361 s, avg_samples: 56.0, ips: 275.03627 samples/s, eta: 19:33:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:39:19] ppocr INFO: epoch: [367/500], global_step: 182600, lr: 0.000100, edit distance: 0.001452, exp_rate: 0.935920, exp_rate<=1 : 0.974602, exp_rate<=2 : 0.993980, exp_rate<=3 : 0.997765, loss: 0.001762, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17990 s, avg_samples: 56.0, ips: 311.28963 samples/s, eta: 19:32:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:39:38] ppocr INFO: epoch: [367/500], global_step: 182700, lr: 0.000100, edit distance: 0.001464, exp_rate: 0.935238, exp_rate<=1 : 0.974445, exp_rate<=2 : 0.994003, exp_rate<=3 : 0.997780, loss: 0.002063, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18946 s, avg_samples: 56.0, ips: 295.58417 samples/s, eta: 19:32:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:39:57] ppocr INFO: epoch: [367/500], global_step: 182800, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.935144, exp_rate<=1 : 0.974505, exp_rate<=2 : 0.994051, exp_rate<=3 : 0.997789, loss: 0.001846, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19053 s, avg_samples: 56.0, ips: 293.92315 samples/s, eta: 19:32:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:40:15] ppocr INFO: epoch: [367/500], global_step: 182900, lr: 0.000100, edit distance: 0.001472, exp_rate: 0.935103, exp_rate<=1 : 0.974404, exp_rate<=2 : 0.993960, exp_rate<=3 : 0.997768, loss: 0.002453, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18501 s, avg_samples: 56.0, ips: 302.68231 samples/s, eta: 19:31:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:40:34] ppocr INFO: epoch: [367/500], global_step: 183000, lr: 0.000100, edit distance: 0.001489, exp_rate: 0.934619, exp_rate<=1 : 0.974225, exp_rate<=2 : 0.993822, exp_rate<=3 : 0.997709, loss: 0.002949, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18423 s, avg_samples: 56.0, ips: 303.96342 samples/s, eta: 19:31:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:40:53] ppocr INFO: epoch: [367/500], global_step: 183100, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.934321, exp_rate<=1 : 0.974040, exp_rate<=2 : 0.993755, exp_rate<=3 : 0.997723, loss: 0.002225, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19351 s, avg_samples: 56.0, ips: 289.39817 samples/s, eta: 19:31:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:41:11] ppocr INFO: epoch: [367/500], global_step: 183200, lr: 0.000100, edit distance: 0.001482, exp_rate: 0.934554, exp_rate<=1 : 0.974220, exp_rate<=2 : 0.993827, exp_rate<=3 : 0.997767, loss: 0.002379, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18264 s, avg_samples: 56.0, ips: 306.60840 samples/s, eta: 19:30:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:41:30] ppocr INFO: epoch: [367/500], global_step: 183300, lr: 0.000100, edit distance: 0.001494, exp_rate: 0.934216, exp_rate<=1 : 0.973955, exp_rate<=2 : 0.993731, exp_rate<=3 : 0.997716, loss: 0.001742, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18746 s, avg_samples: 56.0, ips: 298.72266 samples/s, eta: 19:30:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:41:50] ppocr INFO: epoch: [367/500], global_step: 183400, lr: 0.000100, edit distance: 0.001506, exp_rate: 0.933770, exp_rate<=1 : 0.973698, exp_rate<=2 : 0.993633, exp_rate<=3 : 0.997648, loss: 0.002131, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19806 s, avg_samples: 56.0, ips: 282.74948 samples/s, eta: 19:30:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:42:09] ppocr INFO: epoch: [367/500], global_step: 183500, lr: 0.000100, edit distance: 0.001510, exp_rate: 0.933528, exp_rate<=1 : 0.973572, exp_rate<=2 : 0.993594, exp_rate<=3 : 0.997611, loss: 0.001166, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18622 s, avg_samples: 56.0, ips: 300.71695 samples/s, eta: 19:29:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:42:24] ppocr INFO: epoch: [367/500], global_step: 183580, lr: 0.000100, edit distance: 0.001499, exp_rate: 0.933930, exp_rate<=1 : 0.973721, exp_rate<=2 : 0.993637, exp_rate<=3 : 0.997621, loss: 0.001427, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15044 s, avg_samples: 44.8, ips: 297.79603 samples/s, eta: 19:29:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:42:26] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 12:42:29] ppocr INFO: epoch: [368/500], global_step: 183600, lr: 0.000100, edit distance: 0.000782, exp_rate: 0.974557, exp_rate<=1 : 0.987753, exp_rate<=2 : 0.996540, exp_rate<=3 : 1.000000, loss: 0.000824, avg_reader_cost: 0.02558 s, avg_batch_cost: 0.05408 s, avg_samples: 11.2, ips: 207.08623 samples/s, eta: 19:29:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:42:48] ppocr INFO: epoch: [368/500], global_step: 183700, lr: 0.000100, edit distance: 0.001051, exp_rate: 0.953033, exp_rate<=1 : 0.981550, exp_rate<=2 : 0.995924, exp_rate<=3 : 0.998199, loss: 0.001694, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19342 s, avg_samples: 56.0, ips: 289.52573 samples/s, eta: 19:29:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:43:07] ppocr INFO: epoch: [368/500], global_step: 183800, lr: 0.000100, edit distance: 0.001116, exp_rate: 0.949713, exp_rate<=1 : 0.980010, exp_rate<=2 : 0.995909, exp_rate<=3 : 0.998227, loss: 0.001530, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19023 s, avg_samples: 56.0, ips: 294.38425 samples/s, eta: 19:28:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:43:26] ppocr INFO: epoch: [368/500], global_step: 183900, lr: 0.000100, edit distance: 0.001069, exp_rate: 0.951217, exp_rate<=1 : 0.980368, exp_rate<=2 : 0.996006, exp_rate<=3 : 0.998275, loss: 0.001133, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18488 s, avg_samples: 56.0, ips: 302.89614 samples/s, eta: 19:28:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:43:44] ppocr INFO: epoch: [368/500], global_step: 184000, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.949961, exp_rate<=1 : 0.979261, exp_rate<=2 : 0.995815, exp_rate<=3 : 0.998525, loss: 0.001491, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17696 s, avg_samples: 56.0, ips: 316.46059 samples/s, eta: 19:28:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:44:03] ppocr INFO: epoch: [368/500], global_step: 184100, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.948803, exp_rate<=1 : 0.979502, exp_rate<=2 : 0.995549, exp_rate<=3 : 0.998538, loss: 0.001433, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19079 s, avg_samples: 56.0, ips: 293.51795 samples/s, eta: 19:27:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:44:21] ppocr INFO: epoch: [368/500], global_step: 184200, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.949005, exp_rate<=1 : 0.979759, exp_rate<=2 : 0.995543, exp_rate<=3 : 0.998526, loss: 0.001831, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18641 s, avg_samples: 56.0, ips: 300.41149 samples/s, eta: 19:27:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:44:40] ppocr INFO: epoch: [368/500], global_step: 184300, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.948879, exp_rate<=1 : 0.979743, exp_rate<=2 : 0.995387, exp_rate<=3 : 0.998521, loss: 0.001226, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18358 s, avg_samples: 56.0, ips: 305.04173 samples/s, eta: 19:27:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:44:59] ppocr INFO: epoch: [368/500], global_step: 184400, lr: 0.000100, edit distance: 0.001157, exp_rate: 0.948222, exp_rate<=1 : 0.979425, exp_rate<=2 : 0.995213, exp_rate<=3 : 0.998390, loss: 0.001537, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19140 s, avg_samples: 56.0, ips: 292.57593 samples/s, eta: 19:26:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:45:18] ppocr INFO: epoch: [368/500], global_step: 184500, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.947302, exp_rate<=1 : 0.979167, exp_rate<=2 : 0.995135, exp_rate<=3 : 0.998290, loss: 0.001172, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19295 s, avg_samples: 56.0, ips: 290.23266 samples/s, eta: 19:26:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:45:37] ppocr INFO: epoch: [368/500], global_step: 184600, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.946199, exp_rate<=1 : 0.978704, exp_rate<=2 : 0.994954, exp_rate<=3 : 0.998213, loss: 0.001456, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19251 s, avg_samples: 56.0, ips: 290.88845 samples/s, eta: 19:26:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:45:56] ppocr INFO: epoch: [368/500], global_step: 184700, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.945659, exp_rate<=1 : 0.978621, exp_rate<=2 : 0.994930, exp_rate<=3 : 0.998215, loss: 0.001258, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18580 s, avg_samples: 56.0, ips: 301.39701 samples/s, eta: 19:25:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:46:15] ppocr INFO: epoch: [368/500], global_step: 184800, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.945094, exp_rate<=1 : 0.978453, exp_rate<=2 : 0.994822, exp_rate<=3 : 0.998097, loss: 0.002173, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18931 s, avg_samples: 56.0, ips: 295.80457 samples/s, eta: 19:25:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:46:34] ppocr INFO: epoch: [368/500], global_step: 184900, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.944388, exp_rate<=1 : 0.978140, exp_rate<=2 : 0.994815, exp_rate<=3 : 0.998113, loss: 0.001598, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18593 s, avg_samples: 56.0, ips: 301.18141 samples/s, eta: 19:25:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:46:53] ppocr INFO: epoch: [368/500], global_step: 185000, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.943006, exp_rate<=1 : 0.977535, exp_rate<=2 : 0.994793, exp_rate<=3 : 0.998125, loss: 0.001984, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19607 s, avg_samples: 56.0, ips: 285.60623 samples/s, eta: 19:24:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:47:12] ppocr INFO: epoch: [368/500], global_step: 185100, lr: 0.000100, edit distance: 0.001270, exp_rate: 0.942477, exp_rate<=1 : 0.977275, exp_rate<=2 : 0.994671, exp_rate<=3 : 0.998111, loss: 0.001554, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18226 s, avg_samples: 56.0, ips: 307.26098 samples/s, eta: 19:24:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:47:31] ppocr INFO: epoch: [368/500], global_step: 185200, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.941591, exp_rate<=1 : 0.977061, exp_rate<=2 : 0.994652, exp_rate<=3 : 0.998137, loss: 0.002052, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19615 s, avg_samples: 56.0, ips: 285.49093 samples/s, eta: 19:24:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:47:51] ppocr INFO: epoch: [368/500], global_step: 185300, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.940070, exp_rate<=1 : 0.976410, exp_rate<=2 : 0.994467, exp_rate<=3 : 0.998061, loss: 0.001809, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19561 s, avg_samples: 56.0, ips: 286.28791 samples/s, eta: 19:23:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:48:10] ppocr INFO: epoch: [368/500], global_step: 185400, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.939577, exp_rate<=1 : 0.976350, exp_rate<=2 : 0.994453, exp_rate<=3 : 0.998040, loss: 0.001365, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19008 s, avg_samples: 56.0, ips: 294.60863 samples/s, eta: 19:23:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:48:29] ppocr INFO: epoch: [368/500], global_step: 185500, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.939665, exp_rate<=1 : 0.976414, exp_rate<=2 : 0.994506, exp_rate<=3 : 0.998057, loss: 0.001634, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19018 s, avg_samples: 56.0, ips: 294.45698 samples/s, eta: 19:23:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:48:47] ppocr INFO: epoch: [368/500], global_step: 185600, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.939326, exp_rate<=1 : 0.976219, exp_rate<=2 : 0.994418, exp_rate<=3 : 0.997997, loss: 0.002403, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18289 s, avg_samples: 56.0, ips: 306.18727 samples/s, eta: 19:22:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:49:05] ppocr INFO: epoch: [368/500], global_step: 185700, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.938577, exp_rate<=1 : 0.976073, exp_rate<=2 : 0.994453, exp_rate<=3 : 0.998029, loss: 0.002346, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17986 s, avg_samples: 56.0, ips: 311.34945 samples/s, eta: 19:22:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:49:25] ppocr INFO: epoch: [368/500], global_step: 185800, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.935197, exp_rate<=1 : 0.974463, exp_rate<=2 : 0.993772, exp_rate<=3 : 0.997702, loss: 0.002568, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19703 s, avg_samples: 56.0, ips: 284.21835 samples/s, eta: 19:22:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:49:44] ppocr INFO: epoch: [368/500], global_step: 185900, lr: 0.000100, edit distance: 0.001441, exp_rate: 0.933752, exp_rate<=1 : 0.973708, exp_rate<=2 : 0.993520, exp_rate<=3 : 0.997583, loss: 0.001235, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19265 s, avg_samples: 56.0, ips: 290.67684 samples/s, eta: 19:21:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:50:02] ppocr INFO: epoch: [368/500], global_step: 186000, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.933791, exp_rate<=1 : 0.973724, exp_rate<=2 : 0.993607, exp_rate<=3 : 0.997637, loss: 0.001622, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17943 s, avg_samples: 56.0, ips: 312.10618 samples/s, eta: 19:21:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:50:21] ppocr INFO: epoch: [368/500], global_step: 186100, lr: 0.000100, edit distance: 0.001449, exp_rate: 0.933542, exp_rate<=1 : 0.973694, exp_rate<=2 : 0.993639, exp_rate<=3 : 0.997665, loss: 0.001328, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18823 s, avg_samples: 56.0, ips: 297.51170 samples/s, eta: 19:21:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:50:40] ppocr INFO: epoch: [368/500], global_step: 186200, lr: 0.000100, edit distance: 0.001453, exp_rate: 0.933386, exp_rate<=1 : 0.973656, exp_rate<=2 : 0.993603, exp_rate<=3 : 0.997645, loss: 0.001168, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18784 s, avg_samples: 56.0, ips: 298.12653 samples/s, eta: 19:20:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:50:59] ppocr INFO: epoch: [368/500], global_step: 186300, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.933178, exp_rate<=1 : 0.973463, exp_rate<=2 : 0.993535, exp_rate<=3 : 0.997620, loss: 0.001744, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19683 s, avg_samples: 56.0, ips: 284.51029 samples/s, eta: 19:20:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:51:03] ppocr INFO: epoch: [368/500], global_step: 186320, lr: 0.000100, edit distance: 0.001452, exp_rate: 0.933296, exp_rate<=1 : 0.973485, exp_rate<=2 : 0.993535, exp_rate<=3 : 0.997615, loss: 0.001249, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03758 s, avg_samples: 11.2, ips: 298.06421 samples/s, eta: 19:20:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:51:05] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 12:51:20] ppocr INFO: epoch: [369/500], global_step: 186400, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.946915, exp_rate<=1 : 0.980294, exp_rate<=2 : 0.995633, exp_rate<=3 : 0.998733, loss: 0.000993, avg_reader_cost: 0.02593 s, avg_batch_cost: 0.17207 s, avg_samples: 44.8, ips: 260.36508 samples/s, eta: 19:20:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:51:40] ppocr INFO: epoch: [369/500], global_step: 186500, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.941900, exp_rate<=1 : 0.977563, exp_rate<=2 : 0.994727, exp_rate<=3 : 0.998477, loss: 0.001471, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19368 s, avg_samples: 56.0, ips: 289.13651 samples/s, eta: 19:19:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:51:58] ppocr INFO: epoch: [369/500], global_step: 186600, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.943588, exp_rate<=1 : 0.978119, exp_rate<=2 : 0.995420, exp_rate<=3 : 0.998675, loss: 0.001398, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18005 s, avg_samples: 56.0, ips: 311.03331 samples/s, eta: 19:19:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:52:17] ppocr INFO: epoch: [369/500], global_step: 186700, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942426, exp_rate<=1 : 0.977537, exp_rate<=2 : 0.994922, exp_rate<=3 : 0.998333, loss: 0.001586, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19498 s, avg_samples: 56.0, ips: 287.21556 samples/s, eta: 19:19:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:52:35] ppocr INFO: epoch: [369/500], global_step: 186800, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.943471, exp_rate<=1 : 0.978310, exp_rate<=2 : 0.995047, exp_rate<=3 : 0.998330, loss: 0.001131, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17784 s, avg_samples: 56.0, ips: 314.88846 samples/s, eta: 19:18:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:52:55] ppocr INFO: epoch: [369/500], global_step: 186900, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.942397, exp_rate<=1 : 0.977580, exp_rate<=2 : 0.994898, exp_rate<=3 : 0.998310, loss: 0.002382, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19969 s, avg_samples: 56.0, ips: 280.43610 samples/s, eta: 19:18:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:53:14] ppocr INFO: epoch: [369/500], global_step: 187000, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.941481, exp_rate<=1 : 0.977241, exp_rate<=2 : 0.994833, exp_rate<=3 : 0.998296, loss: 0.001876, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18975 s, avg_samples: 56.0, ips: 295.13287 samples/s, eta: 19:18:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:53:33] ppocr INFO: epoch: [369/500], global_step: 187100, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.941348, exp_rate<=1 : 0.977349, exp_rate<=2 : 0.995080, exp_rate<=3 : 0.998424, loss: 0.001889, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18739 s, avg_samples: 56.0, ips: 298.84092 samples/s, eta: 19:17:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:53:53] ppocr INFO: epoch: [369/500], global_step: 187200, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.940551, exp_rate<=1 : 0.976869, exp_rate<=2 : 0.994867, exp_rate<=3 : 0.998324, loss: 0.001194, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19976 s, avg_samples: 56.0, ips: 280.34091 samples/s, eta: 19:17:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:54:12] ppocr INFO: epoch: [369/500], global_step: 187300, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.941200, exp_rate<=1 : 0.976853, exp_rate<=2 : 0.994841, exp_rate<=3 : 0.998372, loss: 0.001377, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18995 s, avg_samples: 56.0, ips: 294.81423 samples/s, eta: 19:17:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:54:30] ppocr INFO: epoch: [369/500], global_step: 187400, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.941272, exp_rate<=1 : 0.976982, exp_rate<=2 : 0.994851, exp_rate<=3 : 0.998414, loss: 0.001269, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18527 s, avg_samples: 56.0, ips: 302.25990 samples/s, eta: 19:17:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:54:48] ppocr INFO: epoch: [369/500], global_step: 187500, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.941319, exp_rate<=1 : 0.977146, exp_rate<=2 : 0.994839, exp_rate<=3 : 0.998341, loss: 0.001306, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17968 s, avg_samples: 56.0, ips: 311.65898 samples/s, eta: 19:16:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:55:07] ppocr INFO: epoch: [369/500], global_step: 187600, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.941436, exp_rate<=1 : 0.977230, exp_rate<=2 : 0.994824, exp_rate<=3 : 0.998327, loss: 0.001390, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18393 s, avg_samples: 56.0, ips: 304.47123 samples/s, eta: 19:16:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:55:27] ppocr INFO: epoch: [369/500], global_step: 187700, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.940509, exp_rate<=1 : 0.976806, exp_rate<=2 : 0.994832, exp_rate<=3 : 0.998277, loss: 0.001372, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20188 s, avg_samples: 56.0, ips: 277.39133 samples/s, eta: 19:16:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:55:48] ppocr INFO: epoch: [369/500], global_step: 187800, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.939725, exp_rate<=1 : 0.976593, exp_rate<=2 : 0.994821, exp_rate<=3 : 0.998288, loss: 0.001550, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20516 s, avg_samples: 56.0, ips: 272.95946 samples/s, eta: 19:15:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:56:04] ppocr INFO: epoch: [369/500], global_step: 187900, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.940317, exp_rate<=1 : 0.977055, exp_rate<=2 : 0.994897, exp_rate<=3 : 0.998340, loss: 0.001821, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16662 s, avg_samples: 56.0, ips: 336.08898 samples/s, eta: 19:15:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:56:23] ppocr INFO: epoch: [369/500], global_step: 188000, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.939444, exp_rate<=1 : 0.976469, exp_rate<=2 : 0.994767, exp_rate<=3 : 0.998322, loss: 0.001685, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18757 s, avg_samples: 56.0, ips: 298.55135 samples/s, eta: 19:14:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:56:42] ppocr INFO: epoch: [369/500], global_step: 188100, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.939165, exp_rate<=1 : 0.976384, exp_rate<=2 : 0.994700, exp_rate<=3 : 0.998321, loss: 0.001354, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18525 s, avg_samples: 56.0, ips: 302.29410 samples/s, eta: 19:14:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:56:59] ppocr INFO: epoch: [369/500], global_step: 188200, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.939392, exp_rate<=1 : 0.976512, exp_rate<=2 : 0.994765, exp_rate<=3 : 0.998300, loss: 0.001860, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17206 s, avg_samples: 56.0, ips: 325.47690 samples/s, eta: 19:14:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:57:19] ppocr INFO: epoch: [369/500], global_step: 188300, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.938556, exp_rate<=1 : 0.976212, exp_rate<=2 : 0.994614, exp_rate<=3 : 0.998234, loss: 0.001408, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20059 s, avg_samples: 56.0, ips: 279.17402 samples/s, eta: 19:13:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:57:38] ppocr INFO: epoch: [369/500], global_step: 188400, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.938429, exp_rate<=1 : 0.976213, exp_rate<=2 : 0.994667, exp_rate<=3 : 0.998267, loss: 0.001796, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18609 s, avg_samples: 56.0, ips: 300.92657 samples/s, eta: 19:13:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:57:56] ppocr INFO: epoch: [369/500], global_step: 188500, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.938581, exp_rate<=1 : 0.976158, exp_rate<=2 : 0.994693, exp_rate<=3 : 0.998283, loss: 0.001592, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18466 s, avg_samples: 56.0, ips: 303.26165 samples/s, eta: 19:13:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:58:15] ppocr INFO: epoch: [369/500], global_step: 188600, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.938310, exp_rate<=1 : 0.976015, exp_rate<=2 : 0.994633, exp_rate<=3 : 0.998271, loss: 0.002246, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18471 s, avg_samples: 56.0, ips: 303.18585 samples/s, eta: 19:12:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:58:34] ppocr INFO: epoch: [369/500], global_step: 188700, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.937877, exp_rate<=1 : 0.975809, exp_rate<=2 : 0.994549, exp_rate<=3 : 0.998223, loss: 0.001303, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19154 s, avg_samples: 56.0, ips: 292.37130 samples/s, eta: 19:12:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:58:52] ppocr INFO: epoch: [369/500], global_step: 188800, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.937632, exp_rate<=1 : 0.975753, exp_rate<=2 : 0.994551, exp_rate<=3 : 0.998201, loss: 0.001316, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18365 s, avg_samples: 56.0, ips: 304.92226 samples/s, eta: 19:12:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:59:11] ppocr INFO: epoch: [369/500], global_step: 188900, lr: 0.000100, edit distance: 0.001360, exp_rate: 0.937463, exp_rate<=1 : 0.975723, exp_rate<=2 : 0.994512, exp_rate<=3 : 0.998200, loss: 0.002051, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18737 s, avg_samples: 56.0, ips: 298.86652 samples/s, eta: 19:11:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:59:30] ppocr INFO: epoch: [369/500], global_step: 189000, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.936693, exp_rate<=1 : 0.975460, exp_rate<=2 : 0.994391, exp_rate<=3 : 0.998123, loss: 0.002002, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19355 s, avg_samples: 56.0, ips: 289.32735 samples/s, eta: 19:11:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:59:41] ppocr INFO: epoch: [369/500], global_step: 189060, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.936611, exp_rate<=1 : 0.975395, exp_rate<=2 : 0.994355, exp_rate<=3 : 0.998090, loss: 0.001610, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10647 s, avg_samples: 33.6, ips: 315.56719 samples/s, eta: 19:11:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 12:59:43] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 12:59:51] ppocr INFO: epoch: [370/500], global_step: 189100, lr: 0.000100, edit distance: 0.001161, exp_rate: 0.939628, exp_rate<=1 : 0.975177, exp_rate<=2 : 0.993558, exp_rate<=3 : 0.998829, loss: 0.001188, avg_reader_cost: 0.02529 s, avg_batch_cost: 0.10352 s, avg_samples: 22.4, ips: 216.38980 samples/s, eta: 19:11:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:00:10] ppocr INFO: epoch: [370/500], global_step: 189200, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.940065, exp_rate<=1 : 0.976779, exp_rate<=2 : 0.994272, exp_rate<=3 : 0.998221, loss: 0.001413, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18801 s, avg_samples: 56.0, ips: 297.86364 samples/s, eta: 19:11:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:00:30] ppocr INFO: epoch: [370/500], global_step: 189300, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.940515, exp_rate<=1 : 0.976911, exp_rate<=2 : 0.994451, exp_rate<=3 : 0.997820, loss: 0.001507, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20230 s, avg_samples: 56.0, ips: 276.82135 samples/s, eta: 19:10:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:00:50] ppocr INFO: epoch: [370/500], global_step: 189400, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.941126, exp_rate<=1 : 0.977278, exp_rate<=2 : 0.994111, exp_rate<=3 : 0.997581, loss: 0.001476, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19661 s, avg_samples: 56.0, ips: 284.82851 samples/s, eta: 19:10:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:01:09] ppocr INFO: epoch: [370/500], global_step: 189500, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942167, exp_rate<=1 : 0.977546, exp_rate<=2 : 0.994462, exp_rate<=3 : 0.997679, loss: 0.000955, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18898 s, avg_samples: 56.0, ips: 296.32941 samples/s, eta: 19:10:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:01:27] ppocr INFO: epoch: [370/500], global_step: 189600, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.942985, exp_rate<=1 : 0.977967, exp_rate<=2 : 0.994837, exp_rate<=3 : 0.998047, loss: 0.001408, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18067 s, avg_samples: 56.0, ips: 309.95559 samples/s, eta: 19:09:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:01:46] ppocr INFO: epoch: [370/500], global_step: 189700, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.943328, exp_rate<=1 : 0.978478, exp_rate<=2 : 0.994888, exp_rate<=3 : 0.998068, loss: 0.000819, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18939 s, avg_samples: 56.0, ips: 295.69045 samples/s, eta: 19:09:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:02:05] ppocr INFO: epoch: [370/500], global_step: 189800, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.944143, exp_rate<=1 : 0.978559, exp_rate<=2 : 0.994862, exp_rate<=3 : 0.998017, loss: 0.001260, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18561 s, avg_samples: 56.0, ips: 301.71238 samples/s, eta: 19:09:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:02:24] ppocr INFO: epoch: [370/500], global_step: 189900, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.944438, exp_rate<=1 : 0.978957, exp_rate<=2 : 0.995016, exp_rate<=3 : 0.998017, loss: 0.001489, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19336 s, avg_samples: 56.0, ips: 289.62141 samples/s, eta: 19:08:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:02:43] ppocr INFO: epoch: [370/500], global_step: 190000, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.944236, exp_rate<=1 : 0.979173, exp_rate<=2 : 0.994897, exp_rate<=3 : 0.998099, loss: 0.001349, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18707 s, avg_samples: 56.0, ips: 299.34756 samples/s, eta: 19:08:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:03:02] ppocr INFO: epoch: [370/500], global_step: 190100, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.943549, exp_rate<=1 : 0.978744, exp_rate<=2 : 0.994775, exp_rate<=3 : 0.998055, loss: 0.002059, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18925 s, avg_samples: 56.0, ips: 295.90855 samples/s, eta: 19:08:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:03:21] ppocr INFO: epoch: [370/500], global_step: 190200, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.942115, exp_rate<=1 : 0.978225, exp_rate<=2 : 0.994581, exp_rate<=3 : 0.998021, loss: 0.001396, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19529 s, avg_samples: 56.0, ips: 286.75320 samples/s, eta: 19:07:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:03:40] ppocr INFO: epoch: [370/500], global_step: 190300, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.941218, exp_rate<=1 : 0.978002, exp_rate<=2 : 0.994569, exp_rate<=3 : 0.998069, loss: 0.001793, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19084 s, avg_samples: 56.0, ips: 293.44689 samples/s, eta: 19:07:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:04:00] ppocr INFO: epoch: [370/500], global_step: 190400, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.940498, exp_rate<=1 : 0.977714, exp_rate<=2 : 0.994598, exp_rate<=3 : 0.998094, loss: 0.001283, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19234 s, avg_samples: 56.0, ips: 291.15809 samples/s, eta: 19:07:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:04:19] ppocr INFO: epoch: [370/500], global_step: 190500, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.939745, exp_rate<=1 : 0.977366, exp_rate<=2 : 0.994500, exp_rate<=3 : 0.998071, loss: 0.001479, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19185 s, avg_samples: 56.0, ips: 291.88997 samples/s, eta: 19:06:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:04:37] ppocr INFO: epoch: [370/500], global_step: 190600, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.939471, exp_rate<=1 : 0.977148, exp_rate<=2 : 0.994442, exp_rate<=3 : 0.998010, loss: 0.002121, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18155 s, avg_samples: 56.0, ips: 308.44973 samples/s, eta: 19:06:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:04:56] ppocr INFO: epoch: [370/500], global_step: 190700, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.939208, exp_rate<=1 : 0.976987, exp_rate<=2 : 0.994461, exp_rate<=3 : 0.998002, loss: 0.000919, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18734 s, avg_samples: 56.0, ips: 298.91802 samples/s, eta: 19:06:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:05:15] ppocr INFO: epoch: [370/500], global_step: 190800, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.938591, exp_rate<=1 : 0.976650, exp_rate<=2 : 0.994311, exp_rate<=3 : 0.997903, loss: 0.001822, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19428 s, avg_samples: 56.0, ips: 288.23736 samples/s, eta: 19:05:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:05:34] ppocr INFO: epoch: [370/500], global_step: 190900, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.938423, exp_rate<=1 : 0.976443, exp_rate<=2 : 0.994254, exp_rate<=3 : 0.997892, loss: 0.001477, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18942 s, avg_samples: 56.0, ips: 295.63736 samples/s, eta: 19:05:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:05:53] ppocr INFO: epoch: [370/500], global_step: 191000, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.938462, exp_rate<=1 : 0.976379, exp_rate<=2 : 0.994339, exp_rate<=3 : 0.997918, loss: 0.001383, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19277 s, avg_samples: 56.0, ips: 290.50489 samples/s, eta: 19:05:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:06:12] ppocr INFO: epoch: [370/500], global_step: 191100, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.938211, exp_rate<=1 : 0.976096, exp_rate<=2 : 0.994381, exp_rate<=3 : 0.997938, loss: 0.001474, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18452 s, avg_samples: 56.0, ips: 303.49671 samples/s, eta: 19:04:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:06:29] ppocr INFO: epoch: [370/500], global_step: 191200, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.938489, exp_rate<=1 : 0.976114, exp_rate<=2 : 0.994342, exp_rate<=3 : 0.997934, loss: 0.001256, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16991 s, avg_samples: 56.0, ips: 329.59315 samples/s, eta: 19:04:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:06:47] ppocr INFO: epoch: [370/500], global_step: 191300, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.938245, exp_rate<=1 : 0.976001, exp_rate<=2 : 0.994306, exp_rate<=3 : 0.997907, loss: 0.001902, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17976 s, avg_samples: 56.0, ips: 311.52102 samples/s, eta: 19:04:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:07:05] ppocr INFO: epoch: [370/500], global_step: 191400, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.938131, exp_rate<=1 : 0.975957, exp_rate<=2 : 0.994345, exp_rate<=3 : 0.997954, loss: 0.001505, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18052 s, avg_samples: 56.0, ips: 310.21468 samples/s, eta: 19:03:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:07:24] ppocr INFO: epoch: [370/500], global_step: 191500, lr: 0.000100, edit distance: 0.001358, exp_rate: 0.937430, exp_rate<=1 : 0.975655, exp_rate<=2 : 0.994299, exp_rate<=3 : 0.997927, loss: 0.002409, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18588 s, avg_samples: 56.0, ips: 301.26546 samples/s, eta: 19:03:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:07:42] ppocr INFO: epoch: [370/500], global_step: 191600, lr: 0.000100, edit distance: 0.001377, exp_rate: 0.936761, exp_rate<=1 : 0.975333, exp_rate<=2 : 0.994188, exp_rate<=3 : 0.997843, loss: 0.001478, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18229 s, avg_samples: 56.0, ips: 307.20650 samples/s, eta: 19:02:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:08:00] ppocr INFO: epoch: [370/500], global_step: 191700, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.936782, exp_rate<=1 : 0.975302, exp_rate<=2 : 0.994199, exp_rate<=3 : 0.997863, loss: 0.001582, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18226 s, avg_samples: 56.0, ips: 307.26140 samples/s, eta: 19:02:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:08:19] ppocr INFO: epoch: [370/500], global_step: 191800, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.936387, exp_rate<=1 : 0.975129, exp_rate<=2 : 0.994127, exp_rate<=3 : 0.997861, loss: 0.001549, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.18964 s, avg_samples: 56.0, ips: 295.30384 samples/s, eta: 19:02:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:08:21] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 13:08:22] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_370 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 13:08:42] ppocr INFO: epoch: [371/500], global_step: 191900, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.944258, exp_rate<=1 : 0.976879, exp_rate<=2 : 0.994942, exp_rate<=3 : 0.998956, loss: 0.001139, avg_reader_cost: 0.03087 s, avg_batch_cost: 0.22693 s, avg_samples: 56.0, ips: 246.77284 samples/s, eta: 19:02:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:09:00] ppocr INFO: epoch: [371/500], global_step: 192000, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.940172, exp_rate<=1 : 0.975085, exp_rate<=2 : 0.992658, exp_rate<=3 : 0.997750, loss: 0.001296, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18272 s, avg_samples: 56.0, ips: 306.47298 samples/s, eta: 19:01:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:09:18] ppocr INFO: epoch: [371/500], global_step: 192100, lr: 0.000100, edit distance: 0.001358, exp_rate: 0.939118, exp_rate<=1 : 0.975239, exp_rate<=2 : 0.993238, exp_rate<=3 : 0.997418, loss: 0.001422, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18006 s, avg_samples: 56.0, ips: 311.00234 samples/s, eta: 19:01:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:09:35] ppocr INFO: epoch: [371/500], global_step: 192200, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.940955, exp_rate<=1 : 0.976552, exp_rate<=2 : 0.993842, exp_rate<=3 : 0.997942, loss: 0.000954, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16828 s, avg_samples: 56.0, ips: 332.77170 samples/s, eta: 19:00:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:09:53] ppocr INFO: epoch: [371/500], global_step: 192300, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.940555, exp_rate<=1 : 0.976564, exp_rate<=2 : 0.994357, exp_rate<=3 : 0.998283, loss: 0.001370, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17851 s, avg_samples: 56.0, ips: 313.71465 samples/s, eta: 19:00:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:10:11] ppocr INFO: epoch: [371/500], global_step: 192400, lr: 0.000100, edit distance: 0.001414, exp_rate: 0.940377, exp_rate<=1 : 0.977019, exp_rate<=2 : 0.994494, exp_rate<=3 : 0.998201, loss: 0.001553, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18444 s, avg_samples: 56.0, ips: 303.62783 samples/s, eta: 19:00:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:10:29] ppocr INFO: epoch: [371/500], global_step: 192500, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.939150, exp_rate<=1 : 0.976718, exp_rate<=2 : 0.994462, exp_rate<=3 : 0.998057, loss: 0.001459, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18157 s, avg_samples: 56.0, ips: 308.41982 samples/s, eta: 18:59:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:10:47] ppocr INFO: epoch: [371/500], global_step: 192600, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.939852, exp_rate<=1 : 0.977255, exp_rate<=2 : 0.994646, exp_rate<=3 : 0.998204, loss: 0.001891, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17276 s, avg_samples: 56.0, ips: 324.14475 samples/s, eta: 18:59:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:11:06] ppocr INFO: epoch: [371/500], global_step: 192700, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.939384, exp_rate<=1 : 0.977095, exp_rate<=2 : 0.994671, exp_rate<=3 : 0.998208, loss: 0.001599, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19029 s, avg_samples: 56.0, ips: 294.28427 samples/s, eta: 18:59:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:11:25] ppocr INFO: epoch: [371/500], global_step: 192800, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.938709, exp_rate<=1 : 0.976961, exp_rate<=2 : 0.994626, exp_rate<=3 : 0.998201, loss: 0.001709, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19656 s, avg_samples: 56.0, ips: 284.89693 samples/s, eta: 18:58:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:11:44] ppocr INFO: epoch: [371/500], global_step: 192900, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.938428, exp_rate<=1 : 0.976717, exp_rate<=2 : 0.994452, exp_rate<=3 : 0.998195, loss: 0.002326, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18909 s, avg_samples: 56.0, ips: 296.15701 samples/s, eta: 18:58:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:12:03] ppocr INFO: epoch: [371/500], global_step: 193000, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.938097, exp_rate<=1 : 0.976603, exp_rate<=2 : 0.994510, exp_rate<=3 : 0.998188, loss: 0.001434, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18317 s, avg_samples: 56.0, ips: 305.73149 samples/s, eta: 18:58:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:12:23] ppocr INFO: epoch: [371/500], global_step: 193100, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.937254, exp_rate<=1 : 0.976420, exp_rate<=2 : 0.994288, exp_rate<=3 : 0.998083, loss: 0.001599, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20130 s, avg_samples: 56.0, ips: 278.19668 samples/s, eta: 18:57:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:12:41] ppocr INFO: epoch: [371/500], global_step: 193200, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.936986, exp_rate<=1 : 0.976190, exp_rate<=2 : 0.994380, exp_rate<=3 : 0.998075, loss: 0.001889, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18555 s, avg_samples: 56.0, ips: 301.80089 samples/s, eta: 18:57:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:13:00] ppocr INFO: epoch: [371/500], global_step: 193300, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.937189, exp_rate<=1 : 0.976208, exp_rate<=2 : 0.994353, exp_rate<=3 : 0.998075, loss: 0.001525, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18796 s, avg_samples: 56.0, ips: 297.93512 samples/s, eta: 18:57:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:13:19] ppocr INFO: epoch: [371/500], global_step: 193400, lr: 0.000100, edit distance: 0.001406, exp_rate: 0.937065, exp_rate<=1 : 0.976086, exp_rate<=2 : 0.994300, exp_rate<=3 : 0.998091, loss: 0.001502, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18825 s, avg_samples: 56.0, ips: 297.46968 samples/s, eta: 18:56:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:13:40] ppocr INFO: epoch: [371/500], global_step: 193500, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.936230, exp_rate<=1 : 0.975882, exp_rate<=2 : 0.994153, exp_rate<=3 : 0.998027, loss: 0.001737, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21008 s, avg_samples: 56.0, ips: 266.56703 samples/s, eta: 18:56:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:14:00] ppocr INFO: epoch: [371/500], global_step: 193600, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.936101, exp_rate<=1 : 0.975618, exp_rate<=2 : 0.994018, exp_rate<=3 : 0.998013, loss: 0.001784, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19764 s, avg_samples: 56.0, ips: 283.34452 samples/s, eta: 18:56:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:14:20] ppocr INFO: epoch: [371/500], global_step: 193700, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.935430, exp_rate<=1 : 0.975313, exp_rate<=2 : 0.993999, exp_rate<=3 : 0.997951, loss: 0.001762, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19970 s, avg_samples: 56.0, ips: 280.41685 samples/s, eta: 18:56:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:14:37] ppocr INFO: epoch: [371/500], global_step: 193800, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.935897, exp_rate<=1 : 0.975473, exp_rate<=2 : 0.994025, exp_rate<=3 : 0.997964, loss: 0.001127, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17131 s, avg_samples: 56.0, ips: 326.90075 samples/s, eta: 18:55:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:14:56] ppocr INFO: epoch: [371/500], global_step: 193900, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.936319, exp_rate<=1 : 0.975689, exp_rate<=2 : 0.994149, exp_rate<=3 : 0.997995, loss: 0.002118, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18572 s, avg_samples: 56.0, ips: 301.53170 samples/s, eta: 18:55:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:15:14] ppocr INFO: epoch: [371/500], global_step: 194000, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.935541, exp_rate<=1 : 0.975250, exp_rate<=2 : 0.993916, exp_rate<=3 : 0.997929, loss: 0.002236, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18597 s, avg_samples: 56.0, ips: 301.11880 samples/s, eta: 18:54:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:15:34] ppocr INFO: epoch: [371/500], global_step: 194100, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.934691, exp_rate<=1 : 0.974879, exp_rate<=2 : 0.993798, exp_rate<=3 : 0.997898, loss: 0.002451, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19972 s, avg_samples: 56.0, ips: 280.39949 samples/s, eta: 18:54:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:15:54] ppocr INFO: epoch: [371/500], global_step: 194200, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.934658, exp_rate<=1 : 0.974875, exp_rate<=2 : 0.993808, exp_rate<=3 : 0.997912, loss: 0.002006, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20241 s, avg_samples: 56.0, ips: 276.66071 samples/s, eta: 18:54:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:16:14] ppocr INFO: epoch: [371/500], global_step: 194300, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.934560, exp_rate<=1 : 0.974773, exp_rate<=2 : 0.993792, exp_rate<=3 : 0.997906, loss: 0.001544, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19396 s, avg_samples: 56.0, ips: 288.71503 samples/s, eta: 18:54:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:16:32] ppocr INFO: epoch: [371/500], global_step: 194400, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.934566, exp_rate<=1 : 0.974828, exp_rate<=2 : 0.993896, exp_rate<=3 : 0.997932, loss: 0.001780, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18395 s, avg_samples: 56.0, ips: 304.43376 samples/s, eta: 18:53:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:16:51] ppocr INFO: epoch: [371/500], global_step: 194500, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.934513, exp_rate<=1 : 0.974757, exp_rate<=2 : 0.993957, exp_rate<=3 : 0.997949, loss: 0.001521, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18711 s, avg_samples: 56.0, ips: 299.28947 samples/s, eta: 18:53:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:16:58] ppocr INFO: epoch: [371/500], global_step: 194540, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.934681, exp_rate<=1 : 0.974829, exp_rate<=2 : 0.993999, exp_rate<=3 : 0.997966, loss: 0.001752, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.06788 s, avg_samples: 22.4, ips: 329.99934 samples/s, eta: 18:53:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:17:00] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 13:17:11] ppocr INFO: epoch: [372/500], global_step: 194600, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.950266, exp_rate<=1 : 0.982519, exp_rate<=2 : 0.998470, exp_rate<=3 : 0.999293, loss: 0.001307, avg_reader_cost: 0.02488 s, avg_batch_cost: 0.12725 s, avg_samples: 33.6, ips: 264.03744 samples/s, eta: 18:53:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:17:30] ppocr INFO: epoch: [372/500], global_step: 194700, lr: 0.000100, edit distance: 0.001470, exp_rate: 0.938976, exp_rate<=1 : 0.977115, exp_rate<=2 : 0.995448, exp_rate<=3 : 0.998398, loss: 0.002027, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19410 s, avg_samples: 56.0, ips: 288.50644 samples/s, eta: 18:52:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:17:49] ppocr INFO: epoch: [372/500], global_step: 194800, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.942026, exp_rate<=1 : 0.978806, exp_rate<=2 : 0.995846, exp_rate<=3 : 0.998521, loss: 0.000769, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19233 s, avg_samples: 56.0, ips: 291.16564 samples/s, eta: 18:52:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:18:08] ppocr INFO: epoch: [372/500], global_step: 194900, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.941888, exp_rate<=1 : 0.978327, exp_rate<=2 : 0.995827, exp_rate<=3 : 0.998565, loss: 0.001882, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18775 s, avg_samples: 56.0, ips: 298.27007 samples/s, eta: 18:52:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:18:26] ppocr INFO: epoch: [372/500], global_step: 195000, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.940509, exp_rate<=1 : 0.978221, exp_rate<=2 : 0.995645, exp_rate<=3 : 0.998646, loss: 0.001729, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17861 s, avg_samples: 56.0, ips: 313.52548 samples/s, eta: 18:51:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:18:45] ppocr INFO: epoch: [372/500], global_step: 195100, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.939509, exp_rate<=1 : 0.977772, exp_rate<=2 : 0.995556, exp_rate<=3 : 0.998521, loss: 0.001012, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18718 s, avg_samples: 56.0, ips: 299.18486 samples/s, eta: 18:51:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:19:03] ppocr INFO: epoch: [372/500], global_step: 195200, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.939484, exp_rate<=1 : 0.977449, exp_rate<=2 : 0.995660, exp_rate<=3 : 0.998640, loss: 0.001278, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18554 s, avg_samples: 56.0, ips: 301.82863 samples/s, eta: 18:51:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:19:23] ppocr INFO: epoch: [372/500], global_step: 195300, lr: 0.000100, edit distance: 0.001390, exp_rate: 0.938987, exp_rate<=1 : 0.976906, exp_rate<=2 : 0.995569, exp_rate<=3 : 0.998515, loss: 0.001374, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19392 s, avg_samples: 56.0, ips: 288.77152 samples/s, eta: 18:50:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:19:43] ppocr INFO: epoch: [372/500], global_step: 195400, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.939415, exp_rate<=1 : 0.977027, exp_rate<=2 : 0.995483, exp_rate<=3 : 0.998541, loss: 0.001181, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20055 s, avg_samples: 56.0, ips: 279.23351 samples/s, eta: 18:50:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:20:02] ppocr INFO: epoch: [372/500], global_step: 195500, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.939566, exp_rate<=1 : 0.977104, exp_rate<=2 : 0.995526, exp_rate<=3 : 0.998535, loss: 0.001430, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19678 s, avg_samples: 56.0, ips: 284.58022 samples/s, eta: 18:50:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:20:21] ppocr INFO: epoch: [372/500], global_step: 195600, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.939541, exp_rate<=1 : 0.977089, exp_rate<=2 : 0.995325, exp_rate<=3 : 0.998351, loss: 0.001223, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18061 s, avg_samples: 56.0, ips: 310.05993 samples/s, eta: 18:49:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:20:40] ppocr INFO: epoch: [372/500], global_step: 195700, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.939457, exp_rate<=1 : 0.977117, exp_rate<=2 : 0.995251, exp_rate<=3 : 0.998370, loss: 0.001144, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19007 s, avg_samples: 56.0, ips: 294.63196 samples/s, eta: 18:49:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:20:58] ppocr INFO: epoch: [372/500], global_step: 195800, lr: 0.000100, edit distance: 0.001351, exp_rate: 0.939441, exp_rate<=1 : 0.977021, exp_rate<=2 : 0.995208, exp_rate<=3 : 0.998384, loss: 0.001861, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18147 s, avg_samples: 56.0, ips: 308.58547 samples/s, eta: 18:49:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:21:16] ppocr INFO: epoch: [372/500], global_step: 195900, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.939472, exp_rate<=1 : 0.977025, exp_rate<=2 : 0.995187, exp_rate<=3 : 0.998347, loss: 0.001395, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18509 s, avg_samples: 56.0, ips: 302.55008 samples/s, eta: 18:48:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:21:35] ppocr INFO: epoch: [372/500], global_step: 196000, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.938962, exp_rate<=1 : 0.976918, exp_rate<=2 : 0.995184, exp_rate<=3 : 0.998326, loss: 0.002370, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18376 s, avg_samples: 56.0, ips: 304.74124 samples/s, eta: 18:48:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:21:53] ppocr INFO: epoch: [372/500], global_step: 196100, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.938105, exp_rate<=1 : 0.976587, exp_rate<=2 : 0.995101, exp_rate<=3 : 0.998304, loss: 0.001354, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18462 s, avg_samples: 56.0, ips: 303.33072 samples/s, eta: 18:48:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:22:13] ppocr INFO: epoch: [372/500], global_step: 196200, lr: 0.000100, edit distance: 0.001405, exp_rate: 0.937254, exp_rate<=1 : 0.976207, exp_rate<=2 : 0.994971, exp_rate<=3 : 0.998240, loss: 0.001702, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19568 s, avg_samples: 56.0, ips: 286.18045 samples/s, eta: 18:47:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:22:31] ppocr INFO: epoch: [372/500], global_step: 196300, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.936879, exp_rate<=1 : 0.976078, exp_rate<=2 : 0.994944, exp_rate<=3 : 0.998234, loss: 0.001897, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18632 s, avg_samples: 56.0, ips: 300.55788 samples/s, eta: 18:47:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:22:50] ppocr INFO: epoch: [372/500], global_step: 196400, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.936542, exp_rate<=1 : 0.976054, exp_rate<=2 : 0.994905, exp_rate<=3 : 0.998215, loss: 0.002121, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18765 s, avg_samples: 56.0, ips: 298.42778 samples/s, eta: 18:47:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:23:09] ppocr INFO: epoch: [372/500], global_step: 196500, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.936326, exp_rate<=1 : 0.975986, exp_rate<=2 : 0.994864, exp_rate<=3 : 0.998206, loss: 0.001331, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18538 s, avg_samples: 56.0, ips: 302.08005 samples/s, eta: 18:46:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:23:28] ppocr INFO: epoch: [372/500], global_step: 196600, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.936122, exp_rate<=1 : 0.975894, exp_rate<=2 : 0.994822, exp_rate<=3 : 0.998184, loss: 0.001696, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18948 s, avg_samples: 56.0, ips: 295.54085 samples/s, eta: 18:46:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:23:47] ppocr INFO: epoch: [372/500], global_step: 196700, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.935875, exp_rate<=1 : 0.975654, exp_rate<=2 : 0.994736, exp_rate<=3 : 0.998167, loss: 0.001102, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19358 s, avg_samples: 56.0, ips: 289.29343 samples/s, eta: 18:46:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:24:06] ppocr INFO: epoch: [372/500], global_step: 196800, lr: 0.000100, edit distance: 0.001419, exp_rate: 0.935929, exp_rate<=1 : 0.975637, exp_rate<=2 : 0.994730, exp_rate<=3 : 0.998133, loss: 0.001367, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19190 s, avg_samples: 56.0, ips: 291.81290 samples/s, eta: 18:45:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:24:26] ppocr INFO: epoch: [372/500], global_step: 196900, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.935765, exp_rate<=1 : 0.975622, exp_rate<=2 : 0.994696, exp_rate<=3 : 0.998146, loss: 0.001345, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19281 s, avg_samples: 56.0, ips: 290.44381 samples/s, eta: 18:45:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:24:46] ppocr INFO: epoch: [372/500], global_step: 197000, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.935335, exp_rate<=1 : 0.975232, exp_rate<=2 : 0.994586, exp_rate<=3 : 0.998127, loss: 0.001748, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20252 s, avg_samples: 56.0, ips: 276.51930 samples/s, eta: 18:45:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:25:04] ppocr INFO: epoch: [372/500], global_step: 197100, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.935226, exp_rate<=1 : 0.975115, exp_rate<=2 : 0.994560, exp_rate<=3 : 0.998133, loss: 0.001229, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17926 s, avg_samples: 56.0, ips: 312.40021 samples/s, eta: 18:44:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:25:23] ppocr INFO: epoch: [372/500], global_step: 197200, lr: 0.000100, edit distance: 0.001428, exp_rate: 0.935359, exp_rate<=1 : 0.975194, exp_rate<=2 : 0.994511, exp_rate<=3 : 0.998105, loss: 0.001444, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18950 s, avg_samples: 56.0, ips: 295.51624 samples/s, eta: 18:44:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:25:36] ppocr INFO: epoch: [372/500], global_step: 197280, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.935535, exp_rate<=1 : 0.975173, exp_rate<=2 : 0.994536, exp_rate<=3 : 0.998103, loss: 0.002285, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.13707 s, avg_samples: 44.8, ips: 326.83816 samples/s, eta: 18:44:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:25:39] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 13:25:43] ppocr INFO: epoch: [373/500], global_step: 197300, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.934673, exp_rate<=1 : 0.971534, exp_rate<=2 : 0.993080, exp_rate<=3 : 0.999084, loss: 0.001919, avg_reader_cost: 0.02944 s, avg_batch_cost: 0.06423 s, avg_samples: 11.2, ips: 174.38586 samples/s, eta: 18:44:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:26:01] ppocr INFO: epoch: [373/500], global_step: 197400, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.933161, exp_rate<=1 : 0.974432, exp_rate<=2 : 0.996101, exp_rate<=3 : 0.999199, loss: 0.001439, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18524 s, avg_samples: 56.0, ips: 302.31092 samples/s, eta: 18:43:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:26:20] ppocr INFO: epoch: [373/500], global_step: 197500, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.936166, exp_rate<=1 : 0.975544, exp_rate<=2 : 0.995202, exp_rate<=3 : 0.998533, loss: 0.001365, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18884 s, avg_samples: 56.0, ips: 296.55368 samples/s, eta: 18:43:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:26:38] ppocr INFO: epoch: [373/500], global_step: 197600, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.938233, exp_rate<=1 : 0.977103, exp_rate<=2 : 0.994971, exp_rate<=3 : 0.998275, loss: 0.001517, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17789 s, avg_samples: 56.0, ips: 314.79289 samples/s, eta: 18:43:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:26:58] ppocr INFO: epoch: [373/500], global_step: 197700, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.938298, exp_rate<=1 : 0.977093, exp_rate<=2 : 0.994764, exp_rate<=3 : 0.998371, loss: 0.001144, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19404 s, avg_samples: 56.0, ips: 288.59761 samples/s, eta: 18:42:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:27:16] ppocr INFO: epoch: [373/500], global_step: 197800, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.940318, exp_rate<=1 : 0.977859, exp_rate<=2 : 0.995016, exp_rate<=3 : 0.998407, loss: 0.001655, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.25725 samples/s, eta: 18:42:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:27:37] ppocr INFO: epoch: [373/500], global_step: 197900, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.940530, exp_rate<=1 : 0.977886, exp_rate<=2 : 0.995021, exp_rate<=3 : 0.998429, loss: 0.001542, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20089 s, avg_samples: 56.0, ips: 278.76362 samples/s, eta: 18:42:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:27:56] ppocr INFO: epoch: [373/500], global_step: 198000, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.939701, exp_rate<=1 : 0.977432, exp_rate<=2 : 0.994695, exp_rate<=3 : 0.998311, loss: 0.002188, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19258 s, avg_samples: 56.0, ips: 290.79524 samples/s, eta: 18:41:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:28:14] ppocr INFO: epoch: [373/500], global_step: 198100, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.938747, exp_rate<=1 : 0.976833, exp_rate<=2 : 0.994323, exp_rate<=3 : 0.998127, loss: 0.001424, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17786 s, avg_samples: 56.0, ips: 314.85960 samples/s, eta: 18:41:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:28:33] ppocr INFO: epoch: [373/500], global_step: 198200, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.938508, exp_rate<=1 : 0.976735, exp_rate<=2 : 0.994388, exp_rate<=3 : 0.998116, loss: 0.001462, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19091 s, avg_samples: 56.0, ips: 293.32783 samples/s, eta: 18:41:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:28:51] ppocr INFO: epoch: [373/500], global_step: 198300, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.938769, exp_rate<=1 : 0.976734, exp_rate<=2 : 0.994340, exp_rate<=3 : 0.998104, loss: 0.002073, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18146 s, avg_samples: 56.0, ips: 308.60366 samples/s, eta: 18:40:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:29:10] ppocr INFO: epoch: [373/500], global_step: 198400, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.938293, exp_rate<=1 : 0.976503, exp_rate<=2 : 0.994347, exp_rate<=3 : 0.998095, loss: 0.001584, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19081 s, avg_samples: 56.0, ips: 293.48471 samples/s, eta: 18:40:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:29:29] ppocr INFO: epoch: [373/500], global_step: 198500, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.938174, exp_rate<=1 : 0.976548, exp_rate<=2 : 0.994467, exp_rate<=3 : 0.998114, loss: 0.001261, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19189 s, avg_samples: 56.0, ips: 291.82915 samples/s, eta: 18:40:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:29:48] ppocr INFO: epoch: [373/500], global_step: 198600, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.937835, exp_rate<=1 : 0.976417, exp_rate<=2 : 0.994485, exp_rate<=3 : 0.998039, loss: 0.001308, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18718 s, avg_samples: 56.0, ips: 299.17455 samples/s, eta: 18:39:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:30:07] ppocr INFO: epoch: [373/500], global_step: 198700, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.937677, exp_rate<=1 : 0.976384, exp_rate<=2 : 0.994512, exp_rate<=3 : 0.997968, loss: 0.001594, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19211 s, avg_samples: 56.0, ips: 291.50711 samples/s, eta: 18:39:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:30:26] ppocr INFO: epoch: [373/500], global_step: 198800, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.937284, exp_rate<=1 : 0.976194, exp_rate<=2 : 0.994475, exp_rate<=3 : 0.997963, loss: 0.001600, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19274 s, avg_samples: 56.0, ips: 290.54361 samples/s, eta: 18:39:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:30:45] ppocr INFO: epoch: [373/500], global_step: 198900, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.937323, exp_rate<=1 : 0.976178, exp_rate<=2 : 0.994358, exp_rate<=3 : 0.997967, loss: 0.001543, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18992 s, avg_samples: 56.0, ips: 294.86042 samples/s, eta: 18:38:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:31:03] ppocr INFO: epoch: [373/500], global_step: 199000, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.937578, exp_rate<=1 : 0.976116, exp_rate<=2 : 0.994259, exp_rate<=3 : 0.997922, loss: 0.001091, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17809 s, avg_samples: 56.0, ips: 314.44525 samples/s, eta: 18:38:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:31:22] ppocr INFO: epoch: [373/500], global_step: 199100, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.937648, exp_rate<=1 : 0.976338, exp_rate<=2 : 0.994346, exp_rate<=3 : 0.997965, loss: 0.001906, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18578 s, avg_samples: 56.0, ips: 301.43159 samples/s, eta: 18:38:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:31:41] ppocr INFO: epoch: [373/500], global_step: 199200, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.937221, exp_rate<=1 : 0.976223, exp_rate<=2 : 0.994209, exp_rate<=3 : 0.997917, loss: 0.001452, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18964 s, avg_samples: 56.0, ips: 295.30159 samples/s, eta: 18:37:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:31:58] ppocr INFO: epoch: [373/500], global_step: 199300, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.937422, exp_rate<=1 : 0.976407, exp_rate<=2 : 0.994278, exp_rate<=3 : 0.997939, loss: 0.002133, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16871 s, avg_samples: 56.0, ips: 331.93805 samples/s, eta: 18:37:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:32:17] ppocr INFO: epoch: [373/500], global_step: 199400, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.937142, exp_rate<=1 : 0.976380, exp_rate<=2 : 0.994305, exp_rate<=3 : 0.997972, loss: 0.001416, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18886 s, avg_samples: 56.0, ips: 296.50834 samples/s, eta: 18:37:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:32:35] ppocr INFO: epoch: [373/500], global_step: 199500, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.936980, exp_rate<=1 : 0.976263, exp_rate<=2 : 0.994292, exp_rate<=3 : 0.997980, loss: 0.001191, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18710 s, avg_samples: 56.0, ips: 299.30370 samples/s, eta: 18:36:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:32:54] ppocr INFO: epoch: [373/500], global_step: 199600, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.936813, exp_rate<=1 : 0.976179, exp_rate<=2 : 0.994272, exp_rate<=3 : 0.997955, loss: 0.002023, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18852 s, avg_samples: 56.0, ips: 297.04769 samples/s, eta: 18:36:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:33:12] ppocr INFO: epoch: [373/500], global_step: 199700, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.936814, exp_rate<=1 : 0.976210, exp_rate<=2 : 0.994348, exp_rate<=3 : 0.997997, loss: 0.001702, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17679 s, avg_samples: 56.0, ips: 316.76735 samples/s, eta: 18:36:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:33:31] ppocr INFO: epoch: [373/500], global_step: 199800, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.936702, exp_rate<=1 : 0.976214, exp_rate<=2 : 0.994368, exp_rate<=3 : 0.998013, loss: 0.001207, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19217 s, avg_samples: 56.0, ips: 291.40191 samples/s, eta: 18:35:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:33:50] ppocr INFO: epoch: [373/500], global_step: 199900, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.936461, exp_rate<=1 : 0.976098, exp_rate<=2 : 0.994334, exp_rate<=3 : 0.997987, loss: 0.001384, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19085 s, avg_samples: 56.0, ips: 293.43169 samples/s, eta: 18:35:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:34:11] ppocr INFO: epoch: [373/500], global_step: 200000, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.936247, exp_rate<=1 : 0.976027, exp_rate<=2 : 0.994288, exp_rate<=3 : 0.997971, loss: 0.001573, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20266 s, avg_samples: 56.0, ips: 276.31830 samples/s, eta: 18:35:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:34:15] ppocr INFO: epoch: [373/500], global_step: 200020, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.936165, exp_rate<=1 : 0.976001, exp_rate<=2 : 0.994291, exp_rate<=3 : 0.997976, loss: 0.001608, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.04162 s, avg_samples: 11.2, ips: 269.10491 samples/s, eta: 18:35:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:34:17] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 13:34:32] ppocr INFO: epoch: [374/500], global_step: 200100, lr: 0.000100, edit distance: 0.001157, exp_rate: 0.942110, exp_rate<=1 : 0.976389, exp_rate<=2 : 0.995675, exp_rate<=3 : 0.998293, loss: 0.001559, avg_reader_cost: 0.02498 s, avg_batch_cost: 0.17222 s, avg_samples: 44.8, ips: 260.13393 samples/s, eta: 18:34:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:34:51] ppocr INFO: epoch: [374/500], global_step: 200200, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.943514, exp_rate<=1 : 0.976733, exp_rate<=2 : 0.994868, exp_rate<=3 : 0.998115, loss: 0.001130, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19445 s, avg_samples: 56.0, ips: 287.99357 samples/s, eta: 18:34:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:35:09] ppocr INFO: epoch: [374/500], global_step: 200300, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.944535, exp_rate<=1 : 0.978258, exp_rate<=2 : 0.995619, exp_rate<=3 : 0.998350, loss: 0.001581, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17805 s, avg_samples: 56.0, ips: 314.51777 samples/s, eta: 18:34:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:35:28] ppocr INFO: epoch: [374/500], global_step: 200400, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.942490, exp_rate<=1 : 0.977286, exp_rate<=2 : 0.995348, exp_rate<=3 : 0.998473, loss: 0.002236, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18208 s, avg_samples: 56.0, ips: 307.56102 samples/s, eta: 18:33:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:35:48] ppocr INFO: epoch: [374/500], global_step: 200500, lr: 0.000100, edit distance: 0.001428, exp_rate: 0.931125, exp_rate<=1 : 0.972174, exp_rate<=2 : 0.993268, exp_rate<=3 : 0.997043, loss: 0.001686, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20008 s, avg_samples: 56.0, ips: 279.88978 samples/s, eta: 18:33:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:36:07] ppocr INFO: epoch: [374/500], global_step: 200600, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.931493, exp_rate<=1 : 0.972777, exp_rate<=2 : 0.993593, exp_rate<=3 : 0.997231, loss: 0.001902, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19838 s, avg_samples: 56.0, ips: 282.27944 samples/s, eta: 18:33:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:36:27] ppocr INFO: epoch: [374/500], global_step: 200700, lr: 0.000100, edit distance: 0.001465, exp_rate: 0.931212, exp_rate<=1 : 0.972515, exp_rate<=2 : 0.993207, exp_rate<=3 : 0.997006, loss: 0.002056, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19281 s, avg_samples: 56.0, ips: 290.43402 samples/s, eta: 18:32:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:36:45] ppocr INFO: epoch: [374/500], global_step: 200800, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.931496, exp_rate<=1 : 0.972930, exp_rate<=2 : 0.993325, exp_rate<=3 : 0.997149, loss: 0.001691, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18189 s, avg_samples: 56.0, ips: 307.87517 samples/s, eta: 18:32:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:37:02] ppocr INFO: epoch: [374/500], global_step: 200900, lr: 0.000100, edit distance: 0.001446, exp_rate: 0.932573, exp_rate<=1 : 0.973250, exp_rate<=2 : 0.993723, exp_rate<=3 : 0.997374, loss: 0.001291, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17047 s, avg_samples: 56.0, ips: 328.50868 samples/s, eta: 18:32:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:37:22] ppocr INFO: epoch: [374/500], global_step: 201000, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.932898, exp_rate<=1 : 0.973375, exp_rate<=2 : 0.993870, exp_rate<=3 : 0.997423, loss: 0.001400, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19900 s, avg_samples: 56.0, ips: 281.41059 samples/s, eta: 18:31:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:37:41] ppocr INFO: epoch: [374/500], global_step: 201100, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.933373, exp_rate<=1 : 0.973714, exp_rate<=2 : 0.993939, exp_rate<=3 : 0.997461, loss: 0.001589, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18946 s, avg_samples: 56.0, ips: 295.58033 samples/s, eta: 18:31:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:37:59] ppocr INFO: epoch: [374/500], global_step: 201200, lr: 0.000100, edit distance: 0.001453, exp_rate: 0.932835, exp_rate<=1 : 0.973565, exp_rate<=2 : 0.993880, exp_rate<=3 : 0.997552, loss: 0.002481, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18528 s, avg_samples: 56.0, ips: 302.24995 samples/s, eta: 18:31:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:38:20] ppocr INFO: epoch: [374/500], global_step: 201300, lr: 0.000100, edit distance: 0.001448, exp_rate: 0.932375, exp_rate<=1 : 0.973456, exp_rate<=2 : 0.993846, exp_rate<=3 : 0.997463, loss: 0.001928, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20404 s, avg_samples: 56.0, ips: 274.45423 samples/s, eta: 18:30:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:38:37] ppocr INFO: epoch: [374/500], global_step: 201400, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.932721, exp_rate<=1 : 0.973657, exp_rate<=2 : 0.993902, exp_rate<=3 : 0.997498, loss: 0.001247, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17539 s, avg_samples: 56.0, ips: 319.27939 samples/s, eta: 18:30:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:38:55] ppocr INFO: epoch: [374/500], global_step: 201500, lr: 0.000100, edit distance: 0.001444, exp_rate: 0.932794, exp_rate<=1 : 0.973769, exp_rate<=2 : 0.993983, exp_rate<=3 : 0.997577, loss: 0.001932, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18010 s, avg_samples: 56.0, ips: 310.94110 samples/s, eta: 18:30:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:39:15] ppocr INFO: epoch: [374/500], global_step: 201600, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.933121, exp_rate<=1 : 0.973787, exp_rate<=2 : 0.993983, exp_rate<=3 : 0.997624, loss: 0.001898, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19560 s, avg_samples: 56.0, ips: 286.29862 samples/s, eta: 18:29:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:39:33] ppocr INFO: epoch: [374/500], global_step: 201700, lr: 0.000100, edit distance: 0.001451, exp_rate: 0.933427, exp_rate<=1 : 0.974006, exp_rate<=2 : 0.994133, exp_rate<=3 : 0.997694, loss: 0.001912, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18435 s, avg_samples: 56.0, ips: 303.77243 samples/s, eta: 18:29:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:39:52] ppocr INFO: epoch: [374/500], global_step: 201800, lr: 0.000100, edit distance: 0.001461, exp_rate: 0.933477, exp_rate<=1 : 0.974088, exp_rate<=2 : 0.994174, exp_rate<=3 : 0.997693, loss: 0.001962, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18635 s, avg_samples: 56.0, ips: 300.51701 samples/s, eta: 18:29:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:40:12] ppocr INFO: epoch: [374/500], global_step: 201900, lr: 0.000100, edit distance: 0.001462, exp_rate: 0.933378, exp_rate<=1 : 0.974103, exp_rate<=2 : 0.994159, exp_rate<=3 : 0.997695, loss: 0.002191, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19428 s, avg_samples: 56.0, ips: 288.24674 samples/s, eta: 18:28:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:40:30] ppocr INFO: epoch: [374/500], global_step: 202000, lr: 0.000100, edit distance: 0.001466, exp_rate: 0.933199, exp_rate<=1 : 0.974015, exp_rate<=2 : 0.994042, exp_rate<=3 : 0.997662, loss: 0.002004, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18179 s, avg_samples: 56.0, ips: 308.05014 samples/s, eta: 18:28:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:40:49] ppocr INFO: epoch: [374/500], global_step: 202100, lr: 0.000100, edit distance: 0.001459, exp_rate: 0.933251, exp_rate<=1 : 0.974161, exp_rate<=2 : 0.994101, exp_rate<=3 : 0.997714, loss: 0.001690, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18833 s, avg_samples: 56.0, ips: 297.35273 samples/s, eta: 18:28:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:41:08] ppocr INFO: epoch: [374/500], global_step: 202200, lr: 0.000100, edit distance: 0.001456, exp_rate: 0.933256, exp_rate<=1 : 0.974193, exp_rate<=2 : 0.994085, exp_rate<=3 : 0.997682, loss: 0.002208, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19457 s, avg_samples: 56.0, ips: 287.80762 samples/s, eta: 18:27:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:41:27] ppocr INFO: epoch: [374/500], global_step: 202300, lr: 0.000100, edit distance: 0.001453, exp_rate: 0.933340, exp_rate<=1 : 0.974295, exp_rate<=2 : 0.994149, exp_rate<=3 : 0.997735, loss: 0.001164, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18987 s, avg_samples: 56.0, ips: 294.93199 samples/s, eta: 18:27:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:41:45] ppocr INFO: epoch: [374/500], global_step: 202400, lr: 0.000100, edit distance: 0.001448, exp_rate: 0.933777, exp_rate<=1 : 0.974549, exp_rate<=2 : 0.994233, exp_rate<=3 : 0.997785, loss: 0.002092, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17413 s, avg_samples: 56.0, ips: 321.60695 samples/s, eta: 18:27:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:42:04] ppocr INFO: epoch: [374/500], global_step: 202500, lr: 0.000100, edit distance: 0.001456, exp_rate: 0.933302, exp_rate<=1 : 0.974392, exp_rate<=2 : 0.994231, exp_rate<=3 : 0.997771, loss: 0.002189, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19846 s, avg_samples: 56.0, ips: 282.17704 samples/s, eta: 18:26:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:42:23] ppocr INFO: epoch: [374/500], global_step: 202600, lr: 0.000100, edit distance: 0.001461, exp_rate: 0.933216, exp_rate<=1 : 0.974274, exp_rate<=2 : 0.994225, exp_rate<=3 : 0.997765, loss: 0.002140, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18669 s, avg_samples: 56.0, ips: 299.95701 samples/s, eta: 18:26:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:42:43] ppocr INFO: epoch: [374/500], global_step: 202700, lr: 0.000100, edit distance: 0.001470, exp_rate: 0.932749, exp_rate<=1 : 0.974124, exp_rate<=2 : 0.994221, exp_rate<=3 : 0.997767, loss: 0.001766, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19588 s, avg_samples: 56.0, ips: 285.89338 samples/s, eta: 18:26:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:42:54] ppocr INFO: epoch: [374/500], global_step: 202760, lr: 0.000100, edit distance: 0.001468, exp_rate: 0.932635, exp_rate<=1 : 0.974077, exp_rate<=2 : 0.994250, exp_rate<=3 : 0.997816, loss: 0.001127, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.11383 s, avg_samples: 33.6, ips: 295.18512 samples/s, eta: 18:26:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:42:56] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 13:43:04] ppocr INFO: epoch: [375/500], global_step: 202800, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.941536, exp_rate<=1 : 0.976531, exp_rate<=2 : 0.996853, exp_rate<=3 : 0.998243, loss: 0.001571, avg_reader_cost: 0.02512 s, avg_batch_cost: 0.09987 s, avg_samples: 22.4, ips: 224.29858 samples/s, eta: 18:26:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:43:23] ppocr INFO: epoch: [375/500], global_step: 202900, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.945728, exp_rate<=1 : 0.976972, exp_rate<=2 : 0.995036, exp_rate<=3 : 0.998195, loss: 0.000990, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18821 s, avg_samples: 56.0, ips: 297.54121 samples/s, eta: 18:25:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:43:41] ppocr INFO: epoch: [375/500], global_step: 203000, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.945637, exp_rate<=1 : 0.978385, exp_rate<=2 : 0.995894, exp_rate<=3 : 0.998760, loss: 0.001369, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18190 s, avg_samples: 56.0, ips: 307.86385 samples/s, eta: 18:25:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:44:00] ppocr INFO: epoch: [375/500], global_step: 203100, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.943324, exp_rate<=1 : 0.977835, exp_rate<=2 : 0.995818, exp_rate<=3 : 0.998723, loss: 0.001897, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18981 s, avg_samples: 56.0, ips: 295.03131 samples/s, eta: 18:25:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:44:19] ppocr INFO: epoch: [375/500], global_step: 203200, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.941964, exp_rate<=1 : 0.977621, exp_rate<=2 : 0.995546, exp_rate<=3 : 0.998262, loss: 0.001159, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18708 s, avg_samples: 56.0, ips: 299.33616 samples/s, eta: 18:24:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:44:38] ppocr INFO: epoch: [375/500], global_step: 203300, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.941836, exp_rate<=1 : 0.977700, exp_rate<=2 : 0.995287, exp_rate<=3 : 0.998081, loss: 0.001670, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18968 s, avg_samples: 56.0, ips: 295.24185 samples/s, eta: 18:24:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:44:56] ppocr INFO: epoch: [375/500], global_step: 203400, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.942407, exp_rate<=1 : 0.978490, exp_rate<=2 : 0.995409, exp_rate<=3 : 0.998262, loss: 0.001449, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18040 s, avg_samples: 56.0, ips: 310.42131 samples/s, eta: 18:24:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:45:15] ppocr INFO: epoch: [375/500], global_step: 203500, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.941242, exp_rate<=1 : 0.978356, exp_rate<=2 : 0.995527, exp_rate<=3 : 0.998338, loss: 0.001324, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19446 s, avg_samples: 56.0, ips: 287.97300 samples/s, eta: 18:23:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:45:34] ppocr INFO: epoch: [375/500], global_step: 203600, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.941457, exp_rate<=1 : 0.978305, exp_rate<=2 : 0.995581, exp_rate<=3 : 0.998398, loss: 0.001198, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18415 s, avg_samples: 56.0, ips: 304.10629 samples/s, eta: 18:23:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:45:52] ppocr INFO: epoch: [375/500], global_step: 203700, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.941135, exp_rate<=1 : 0.978317, exp_rate<=2 : 0.995706, exp_rate<=3 : 0.998433, loss: 0.001550, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18507 s, avg_samples: 56.0, ips: 302.58892 samples/s, eta: 18:23:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:46:11] ppocr INFO: epoch: [375/500], global_step: 203800, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.940418, exp_rate<=1 : 0.978083, exp_rate<=2 : 0.995525, exp_rate<=3 : 0.998398, loss: 0.001495, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18498 s, avg_samples: 56.0, ips: 302.74193 samples/s, eta: 18:22:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:46:29] ppocr INFO: epoch: [375/500], global_step: 203900, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.939951, exp_rate<=1 : 0.977749, exp_rate<=2 : 0.995408, exp_rate<=3 : 0.998326, loss: 0.001580, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17814 s, avg_samples: 56.0, ips: 314.35728 samples/s, eta: 18:22:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:46:46] ppocr INFO: epoch: [375/500], global_step: 204000, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.937821, exp_rate<=1 : 0.976152, exp_rate<=2 : 0.994541, exp_rate<=3 : 0.997964, loss: 0.002744, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17631 s, avg_samples: 56.0, ips: 317.63123 samples/s, eta: 18:21:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:47:06] ppocr INFO: epoch: [375/500], global_step: 204100, lr: 0.000100, edit distance: 0.001407, exp_rate: 0.937110, exp_rate<=1 : 0.975850, exp_rate<=2 : 0.994381, exp_rate<=3 : 0.997879, loss: 0.001185, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19208 s, avg_samples: 56.0, ips: 291.55164 samples/s, eta: 18:21:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:47:24] ppocr INFO: epoch: [375/500], global_step: 204200, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.937123, exp_rate<=1 : 0.975744, exp_rate<=2 : 0.994223, exp_rate<=3 : 0.997798, loss: 0.001416, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17978 s, avg_samples: 56.0, ips: 311.48507 samples/s, eta: 18:21:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:47:43] ppocr INFO: epoch: [375/500], global_step: 204300, lr: 0.000100, edit distance: 0.001413, exp_rate: 0.936943, exp_rate<=1 : 0.975568, exp_rate<=2 : 0.994164, exp_rate<=3 : 0.997751, loss: 0.001725, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19075 s, avg_samples: 56.0, ips: 293.57786 samples/s, eta: 18:20:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:48:02] ppocr INFO: epoch: [375/500], global_step: 204400, lr: 0.000100, edit distance: 0.001400, exp_rate: 0.937164, exp_rate<=1 : 0.975604, exp_rate<=2 : 0.994195, exp_rate<=3 : 0.997744, loss: 0.001042, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19059 s, avg_samples: 56.0, ips: 293.82419 samples/s, eta: 18:20:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:48:19] ppocr INFO: epoch: [375/500], global_step: 204500, lr: 0.000100, edit distance: 0.001400, exp_rate: 0.937319, exp_rate<=1 : 0.975677, exp_rate<=2 : 0.994287, exp_rate<=3 : 0.997786, loss: 0.001141, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17489 s, avg_samples: 56.0, ips: 320.20849 samples/s, eta: 18:20:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:48:37] ppocr INFO: epoch: [375/500], global_step: 204600, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.937127, exp_rate<=1 : 0.975666, exp_rate<=2 : 0.994325, exp_rate<=3 : 0.997798, loss: 0.001580, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18149 s, avg_samples: 56.0, ips: 308.56340 samples/s, eta: 18:19:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:48:57] ppocr INFO: epoch: [375/500], global_step: 204700, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.936796, exp_rate<=1 : 0.975400, exp_rate<=2 : 0.994194, exp_rate<=3 : 0.997719, loss: 0.001525, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19673 s, avg_samples: 56.0, ips: 284.64746 samples/s, eta: 18:19:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:49:18] ppocr INFO: epoch: [375/500], global_step: 204800, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.936520, exp_rate<=1 : 0.975215, exp_rate<=2 : 0.994089, exp_rate<=3 : 0.997713, loss: 0.001422, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20415 s, avg_samples: 56.0, ips: 274.30906 samples/s, eta: 18:19:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:49:39] ppocr INFO: epoch: [375/500], global_step: 204900, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.935870, exp_rate<=1 : 0.975071, exp_rate<=2 : 0.994120, exp_rate<=3 : 0.997715, loss: 0.001666, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21675 s, avg_samples: 56.0, ips: 258.35639 samples/s, eta: 18:19:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:49:58] ppocr INFO: epoch: [375/500], global_step: 205000, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.935977, exp_rate<=1 : 0.975182, exp_rate<=2 : 0.994098, exp_rate<=3 : 0.997718, loss: 0.000904, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19089 s, avg_samples: 56.0, ips: 293.36479 samples/s, eta: 18:18:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:50:18] ppocr INFO: epoch: [375/500], global_step: 205100, lr: 0.000100, edit distance: 0.001410, exp_rate: 0.935857, exp_rate<=1 : 0.974992, exp_rate<=2 : 0.994074, exp_rate<=3 : 0.997705, loss: 0.001030, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19659 s, avg_samples: 56.0, ips: 284.86178 samples/s, eta: 18:18:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:50:38] ppocr INFO: epoch: [375/500], global_step: 205200, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.935928, exp_rate<=1 : 0.975098, exp_rate<=2 : 0.994106, exp_rate<=3 : 0.997738, loss: 0.001449, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19695 s, avg_samples: 56.0, ips: 284.33912 samples/s, eta: 18:18:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:50:55] ppocr INFO: epoch: [375/500], global_step: 205300, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.936488, exp_rate<=1 : 0.975303, exp_rate<=2 : 0.994175, exp_rate<=3 : 0.997782, loss: 0.001794, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17655 s, avg_samples: 56.0, ips: 317.18438 samples/s, eta: 18:17:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:51:15] ppocr INFO: epoch: [375/500], global_step: 205400, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.936560, exp_rate<=1 : 0.975317, exp_rate<=2 : 0.994199, exp_rate<=3 : 0.997801, loss: 0.001387, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19124 s, avg_samples: 56.0, ips: 292.83242 samples/s, eta: 18:17:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:51:32] ppocr INFO: epoch: [375/500], global_step: 205500, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.937097, exp_rate<=1 : 0.975550, exp_rate<=2 : 0.994261, exp_rate<=3 : 0.997842, loss: 0.001567, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17773 s, avg_samples: 56.0, ips: 315.08347 samples/s, eta: 18:17:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:51:35] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 13:51:35] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_375 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 13:51:56] ppocr INFO: epoch: [376/500], global_step: 205600, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.941424, exp_rate<=1 : 0.975335, exp_rate<=2 : 0.994883, exp_rate<=3 : 0.997829, loss: 0.001394, avg_reader_cost: 0.03324 s, avg_batch_cost: 0.23729 s, avg_samples: 56.0, ips: 235.99488 samples/s, eta: 18:16:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:52:15] ppocr INFO: epoch: [376/500], global_step: 205700, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.941433, exp_rate<=1 : 0.977324, exp_rate<=2 : 0.995032, exp_rate<=3 : 0.998524, loss: 0.001660, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18682 s, avg_samples: 56.0, ips: 299.75898 samples/s, eta: 18:16:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:52:33] ppocr INFO: epoch: [376/500], global_step: 205800, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.941865, exp_rate<=1 : 0.978652, exp_rate<=2 : 0.995350, exp_rate<=3 : 0.998541, loss: 0.001957, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17843 s, avg_samples: 56.0, ips: 313.85178 samples/s, eta: 18:16:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:52:53] ppocr INFO: epoch: [376/500], global_step: 205900, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.939958, exp_rate<=1 : 0.977158, exp_rate<=2 : 0.994888, exp_rate<=3 : 0.998221, loss: 0.001282, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20250 s, avg_samples: 56.0, ips: 276.54855 samples/s, eta: 18:15:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:53:12] ppocr INFO: epoch: [376/500], global_step: 206000, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.939732, exp_rate<=1 : 0.977038, exp_rate<=2 : 0.994666, exp_rate<=3 : 0.998077, loss: 0.001604, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19057 s, avg_samples: 56.0, ips: 293.84937 samples/s, eta: 18:15:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:53:31] ppocr INFO: epoch: [376/500], global_step: 206100, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.939746, exp_rate<=1 : 0.977514, exp_rate<=2 : 0.994902, exp_rate<=3 : 0.998307, loss: 0.001064, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18980 s, avg_samples: 56.0, ips: 295.04798 samples/s, eta: 18:15:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:53:49] ppocr INFO: epoch: [376/500], global_step: 206200, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.939895, exp_rate<=1 : 0.977850, exp_rate<=2 : 0.994844, exp_rate<=3 : 0.998328, loss: 0.002228, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18320 s, avg_samples: 56.0, ips: 305.67837 samples/s, eta: 18:14:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:54:08] ppocr INFO: epoch: [376/500], global_step: 206300, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.939588, exp_rate<=1 : 0.977619, exp_rate<=2 : 0.994765, exp_rate<=3 : 0.998294, loss: 0.002275, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18179 s, avg_samples: 56.0, ips: 308.04721 samples/s, eta: 18:14:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:54:26] ppocr INFO: epoch: [376/500], global_step: 206400, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.939989, exp_rate<=1 : 0.977714, exp_rate<=2 : 0.994736, exp_rate<=3 : 0.998316, loss: 0.001157, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18361 s, avg_samples: 56.0, ips: 304.98887 samples/s, eta: 18:14:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:54:44] ppocr INFO: epoch: [376/500], global_step: 206500, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.939098, exp_rate<=1 : 0.977198, exp_rate<=2 : 0.994619, exp_rate<=3 : 0.998250, loss: 0.001920, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18418 s, avg_samples: 56.0, ips: 304.04782 samples/s, eta: 18:13:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:55:03] ppocr INFO: epoch: [376/500], global_step: 206600, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.939186, exp_rate<=1 : 0.977210, exp_rate<=2 : 0.994685, exp_rate<=3 : 0.998325, loss: 0.001678, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18625 s, avg_samples: 56.0, ips: 300.67353 samples/s, eta: 18:13:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:55:22] ppocr INFO: epoch: [376/500], global_step: 206700, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.938893, exp_rate<=1 : 0.976833, exp_rate<=2 : 0.994596, exp_rate<=3 : 0.998264, loss: 0.002009, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18493 s, avg_samples: 56.0, ips: 302.82395 samples/s, eta: 18:13:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:55:40] ppocr INFO: epoch: [376/500], global_step: 206800, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.938862, exp_rate<=1 : 0.976576, exp_rate<=2 : 0.994570, exp_rate<=3 : 0.998269, loss: 0.001782, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17968 s, avg_samples: 56.0, ips: 311.66687 samples/s, eta: 18:12:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:55:58] ppocr INFO: epoch: [376/500], global_step: 206900, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.938195, exp_rate<=1 : 0.976560, exp_rate<=2 : 0.994661, exp_rate<=3 : 0.998295, loss: 0.001833, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18494 s, avg_samples: 56.0, ips: 302.80641 samples/s, eta: 18:12:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:56:16] ppocr INFO: epoch: [376/500], global_step: 207000, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.937542, exp_rate<=1 : 0.976256, exp_rate<=2 : 0.994618, exp_rate<=3 : 0.998278, loss: 0.002303, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17820 s, avg_samples: 56.0, ips: 314.24577 samples/s, eta: 18:12:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:56:35] ppocr INFO: epoch: [376/500], global_step: 207100, lr: 0.000100, edit distance: 0.001367, exp_rate: 0.936857, exp_rate<=1 : 0.976069, exp_rate<=2 : 0.994554, exp_rate<=3 : 0.998226, loss: 0.001813, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19407 s, avg_samples: 56.0, ips: 288.55532 samples/s, eta: 18:11:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:56:56] ppocr INFO: epoch: [376/500], global_step: 207200, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.935959, exp_rate<=1 : 0.975500, exp_rate<=2 : 0.994283, exp_rate<=3 : 0.998045, loss: 0.001612, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20220 s, avg_samples: 56.0, ips: 276.95175 samples/s, eta: 18:11:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:57:15] ppocr INFO: epoch: [376/500], global_step: 207300, lr: 0.000100, edit distance: 0.001390, exp_rate: 0.935769, exp_rate<=1 : 0.975289, exp_rate<=2 : 0.994319, exp_rate<=3 : 0.998045, loss: 0.001493, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19018 s, avg_samples: 56.0, ips: 294.46325 samples/s, eta: 18:11:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:57:34] ppocr INFO: epoch: [376/500], global_step: 207400, lr: 0.000100, edit distance: 0.001416, exp_rate: 0.935186, exp_rate<=1 : 0.975093, exp_rate<=2 : 0.994289, exp_rate<=3 : 0.998054, loss: 0.002308, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19496 s, avg_samples: 56.0, ips: 287.23908 samples/s, eta: 18:10:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:57:53] ppocr INFO: epoch: [376/500], global_step: 207500, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.935228, exp_rate<=1 : 0.975083, exp_rate<=2 : 0.994251, exp_rate<=3 : 0.998017, loss: 0.001378, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19207 s, avg_samples: 56.0, ips: 291.56568 samples/s, eta: 18:10:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:58:13] ppocr INFO: epoch: [376/500], global_step: 207600, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.934890, exp_rate<=1 : 0.974827, exp_rate<=2 : 0.994220, exp_rate<=3 : 0.998049, loss: 0.001663, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19559 s, avg_samples: 56.0, ips: 286.31077 samples/s, eta: 18:10:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:58:31] ppocr INFO: epoch: [376/500], global_step: 207700, lr: 0.000100, edit distance: 0.001430, exp_rate: 0.935329, exp_rate<=1 : 0.975070, exp_rate<=2 : 0.994320, exp_rate<=3 : 0.998102, loss: 0.001416, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17525 s, avg_samples: 56.0, ips: 319.53507 samples/s, eta: 18:09:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:58:50] ppocr INFO: epoch: [376/500], global_step: 207800, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.934789, exp_rate<=1 : 0.974845, exp_rate<=2 : 0.994256, exp_rate<=3 : 0.998090, loss: 0.001729, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19552 s, avg_samples: 56.0, ips: 286.41224 samples/s, eta: 18:09:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:59:09] ppocr INFO: epoch: [376/500], global_step: 207900, lr: 0.000100, edit distance: 0.001431, exp_rate: 0.934928, exp_rate<=1 : 0.974925, exp_rate<=2 : 0.994253, exp_rate<=3 : 0.998119, loss: 0.001232, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18794 s, avg_samples: 56.0, ips: 297.96984 samples/s, eta: 18:09:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:59:28] ppocr INFO: epoch: [376/500], global_step: 208000, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.934734, exp_rate<=1 : 0.974825, exp_rate<=2 : 0.994259, exp_rate<=3 : 0.998105, loss: 0.001600, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19135 s, avg_samples: 56.0, ips: 292.65982 samples/s, eta: 18:08:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 13:59:46] ppocr INFO: epoch: [376/500], global_step: 208100, lr: 0.000100, edit distance: 0.001457, exp_rate: 0.934565, exp_rate<=1 : 0.974805, exp_rate<=2 : 0.994244, exp_rate<=3 : 0.998097, loss: 0.002465, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17808 s, avg_samples: 56.0, ips: 314.46971 samples/s, eta: 18:08:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:00:03] ppocr INFO: epoch: [376/500], global_step: 208200, lr: 0.000100, edit distance: 0.001474, exp_rate: 0.934581, exp_rate<=1 : 0.974798, exp_rate<=2 : 0.994276, exp_rate<=3 : 0.998112, loss: 0.002208, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17381 s, avg_samples: 56.0, ips: 322.18425 samples/s, eta: 18:08:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:00:10] ppocr INFO: epoch: [376/500], global_step: 208240, lr: 0.000100, edit distance: 0.001480, exp_rate: 0.934632, exp_rate<=1 : 0.974875, exp_rate<=2 : 0.994290, exp_rate<=3 : 0.998108, loss: 0.001633, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.06928 s, avg_samples: 22.4, ips: 323.30958 samples/s, eta: 18:08:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:00:12] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 14:00:24] ppocr INFO: epoch: [377/500], global_step: 208300, lr: 0.000100, edit distance: 0.001529, exp_rate: 0.942860, exp_rate<=1 : 0.978242, exp_rate<=2 : 0.996817, exp_rate<=3 : 0.999293, loss: 0.001101, avg_reader_cost: 0.02436 s, avg_batch_cost: 0.13689 s, avg_samples: 33.6, ips: 245.45450 samples/s, eta: 18:07:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:00:43] ppocr INFO: epoch: [377/500], global_step: 208400, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.940026, exp_rate<=1 : 0.977414, exp_rate<=2 : 0.996285, exp_rate<=3 : 0.999378, loss: 0.001523, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19198 s, avg_samples: 56.0, ips: 291.69405 samples/s, eta: 18:07:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:01:04] ppocr INFO: epoch: [377/500], global_step: 208500, lr: 0.000100, edit distance: 0.001407, exp_rate: 0.936785, exp_rate<=1 : 0.976500, exp_rate<=2 : 0.995805, exp_rate<=3 : 0.998778, loss: 0.002234, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20403 s, avg_samples: 56.0, ips: 274.46579 samples/s, eta: 18:07:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:01:21] ppocr INFO: epoch: [377/500], global_step: 208600, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.937639, exp_rate<=1 : 0.976920, exp_rate<=2 : 0.995441, exp_rate<=3 : 0.998539, loss: 0.001593, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17620 s, avg_samples: 56.0, ips: 317.81821 samples/s, eta: 18:06:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:01:40] ppocr INFO: epoch: [377/500], global_step: 208700, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.936595, exp_rate<=1 : 0.976383, exp_rate<=2 : 0.995073, exp_rate<=3 : 0.998391, loss: 0.001469, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19246 s, avg_samples: 56.0, ips: 290.96425 samples/s, eta: 18:06:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:01:59] ppocr INFO: epoch: [377/500], global_step: 208800, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.936456, exp_rate<=1 : 0.976126, exp_rate<=2 : 0.994886, exp_rate<=3 : 0.998387, loss: 0.001370, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18956 s, avg_samples: 56.0, ips: 295.42385 samples/s, eta: 18:06:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:02:16] ppocr INFO: epoch: [377/500], global_step: 208900, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.938073, exp_rate<=1 : 0.976896, exp_rate<=2 : 0.994945, exp_rate<=3 : 0.998518, loss: 0.001719, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16667 s, avg_samples: 56.0, ips: 335.98845 samples/s, eta: 18:05:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:02:34] ppocr INFO: epoch: [377/500], global_step: 209000, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.938199, exp_rate<=1 : 0.976988, exp_rate<=2 : 0.994927, exp_rate<=3 : 0.998489, loss: 0.001487, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18140 s, avg_samples: 56.0, ips: 308.70429 samples/s, eta: 18:05:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:02:54] ppocr INFO: epoch: [377/500], global_step: 209100, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.936993, exp_rate<=1 : 0.975996, exp_rate<=2 : 0.994552, exp_rate<=3 : 0.998294, loss: 0.001082, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20152 s, avg_samples: 56.0, ips: 277.89283 samples/s, eta: 18:05:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:03:15] ppocr INFO: epoch: [377/500], global_step: 209200, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.935991, exp_rate<=1 : 0.975614, exp_rate<=2 : 0.994476, exp_rate<=3 : 0.998281, loss: 0.001336, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20215 s, avg_samples: 56.0, ips: 277.02105 samples/s, eta: 18:04:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:03:33] ppocr INFO: epoch: [377/500], global_step: 209300, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.936350, exp_rate<=1 : 0.975900, exp_rate<=2 : 0.994584, exp_rate<=3 : 0.998345, loss: 0.001724, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18377 s, avg_samples: 56.0, ips: 304.73639 samples/s, eta: 18:04:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:03:51] ppocr INFO: epoch: [377/500], global_step: 209400, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.937093, exp_rate<=1 : 0.976139, exp_rate<=2 : 0.994721, exp_rate<=3 : 0.998295, loss: 0.001644, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18339 s, avg_samples: 56.0, ips: 305.36784 samples/s, eta: 18:04:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:04:12] ppocr INFO: epoch: [377/500], global_step: 209500, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.936679, exp_rate<=1 : 0.975889, exp_rate<=2 : 0.994531, exp_rate<=3 : 0.998152, loss: 0.001390, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20520 s, avg_samples: 56.0, ips: 272.89983 samples/s, eta: 18:04:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:04:30] ppocr INFO: epoch: [377/500], global_step: 209600, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.937926, exp_rate<=1 : 0.976338, exp_rate<=2 : 0.994699, exp_rate<=3 : 0.998190, loss: 0.001606, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17537 s, avg_samples: 56.0, ips: 319.31779 samples/s, eta: 18:03:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:04:49] ppocr INFO: epoch: [377/500], global_step: 209700, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.938070, exp_rate<=1 : 0.976244, exp_rate<=2 : 0.994599, exp_rate<=3 : 0.998116, loss: 0.001935, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19662 s, avg_samples: 56.0, ips: 284.81231 samples/s, eta: 18:03:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:05:08] ppocr INFO: epoch: [377/500], global_step: 209800, lr: 0.000100, edit distance: 0.001372, exp_rate: 0.937612, exp_rate<=1 : 0.976106, exp_rate<=2 : 0.994566, exp_rate<=3 : 0.998134, loss: 0.001986, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18743 s, avg_samples: 56.0, ips: 298.77397 samples/s, eta: 18:03:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:05:26] ppocr INFO: epoch: [377/500], global_step: 209900, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.937524, exp_rate<=1 : 0.976144, exp_rate<=2 : 0.994601, exp_rate<=3 : 0.998148, loss: 0.001609, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17878 s, avg_samples: 56.0, ips: 313.23466 samples/s, eta: 18:02:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:05:45] ppocr INFO: epoch: [377/500], global_step: 210000, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.937283, exp_rate<=1 : 0.976095, exp_rate<=2 : 0.994691, exp_rate<=3 : 0.998178, loss: 0.001603, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18620 s, avg_samples: 56.0, ips: 300.75686 samples/s, eta: 18:02:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:06:03] ppocr INFO: epoch: [377/500], global_step: 210100, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.937136, exp_rate<=1 : 0.976079, exp_rate<=2 : 0.994623, exp_rate<=3 : 0.998136, loss: 0.001439, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18705 s, avg_samples: 56.0, ips: 299.37810 samples/s, eta: 18:01:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:06:22] ppocr INFO: epoch: [377/500], global_step: 210200, lr: 0.000100, edit distance: 0.001390, exp_rate: 0.936930, exp_rate<=1 : 0.976050, exp_rate<=2 : 0.994511, exp_rate<=3 : 0.998073, loss: 0.001486, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18630 s, avg_samples: 56.0, ips: 300.58518 samples/s, eta: 18:01:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:06:40] ppocr INFO: epoch: [377/500], global_step: 210300, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.936636, exp_rate<=1 : 0.975842, exp_rate<=2 : 0.994504, exp_rate<=3 : 0.998039, loss: 0.002185, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18149 s, avg_samples: 56.0, ips: 308.55914 samples/s, eta: 18:01:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:06:59] ppocr INFO: epoch: [377/500], global_step: 210400, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.936467, exp_rate<=1 : 0.975670, exp_rate<=2 : 0.994463, exp_rate<=3 : 0.998016, loss: 0.001753, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18934 s, avg_samples: 56.0, ips: 295.76113 samples/s, eta: 18:00:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:07:18] ppocr INFO: epoch: [377/500], global_step: 210500, lr: 0.000100, edit distance: 0.001393, exp_rate: 0.936596, exp_rate<=1 : 0.975694, exp_rate<=2 : 0.994491, exp_rate<=3 : 0.998028, loss: 0.001595, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18567 s, avg_samples: 56.0, ips: 301.60229 samples/s, eta: 18:00:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:07:37] ppocr INFO: epoch: [377/500], global_step: 210600, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.936484, exp_rate<=1 : 0.975669, exp_rate<=2 : 0.994455, exp_rate<=3 : 0.998018, loss: 0.001451, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19279 s, avg_samples: 56.0, ips: 290.47680 samples/s, eta: 18:00:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:07:55] ppocr INFO: epoch: [377/500], global_step: 210700, lr: 0.000100, edit distance: 0.001400, exp_rate: 0.936446, exp_rate<=1 : 0.975545, exp_rate<=2 : 0.994424, exp_rate<=3 : 0.997997, loss: 0.001787, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18534 s, avg_samples: 56.0, ips: 302.15212 samples/s, eta: 17:59:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:08:15] ppocr INFO: epoch: [377/500], global_step: 210800, lr: 0.000100, edit distance: 0.001400, exp_rate: 0.936445, exp_rate<=1 : 0.975539, exp_rate<=2 : 0.994392, exp_rate<=3 : 0.997977, loss: 0.001325, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19036 s, avg_samples: 56.0, ips: 294.17594 samples/s, eta: 17:59:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:08:33] ppocr INFO: epoch: [377/500], global_step: 210900, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.936793, exp_rate<=1 : 0.975607, exp_rate<=2 : 0.994407, exp_rate<=3 : 0.998017, loss: 0.001317, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18249 s, avg_samples: 56.0, ips: 306.87072 samples/s, eta: 17:59:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:08:47] ppocr INFO: epoch: [377/500], global_step: 210980, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.936702, exp_rate<=1 : 0.975578, exp_rate<=2 : 0.994379, exp_rate<=3 : 0.997995, loss: 0.002338, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.14479 s, avg_samples: 44.8, ips: 309.40657 samples/s, eta: 17:59:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:08:50] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 14:08:53] ppocr INFO: epoch: [378/500], global_step: 211000, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.953827, exp_rate<=1 : 0.988095, exp_rate<=2 : 0.994969, exp_rate<=3 : 1.000000, loss: 0.001347, avg_reader_cost: 0.02828 s, avg_batch_cost: 0.06061 s, avg_samples: 11.2, ips: 184.79342 samples/s, eta: 17:59:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:09:13] ppocr INFO: epoch: [378/500], global_step: 211100, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.940159, exp_rate<=1 : 0.978525, exp_rate<=2 : 0.994493, exp_rate<=3 : 0.998072, loss: 0.001453, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19418 s, avg_samples: 56.0, ips: 288.38552 samples/s, eta: 17:58:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:09:32] ppocr INFO: epoch: [378/500], global_step: 211200, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.936814, exp_rate<=1 : 0.976488, exp_rate<=2 : 0.993807, exp_rate<=3 : 0.997540, loss: 0.001302, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18980 s, avg_samples: 56.0, ips: 295.04462 samples/s, eta: 17:58:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:09:50] ppocr INFO: epoch: [378/500], global_step: 211300, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.938901, exp_rate<=1 : 0.977234, exp_rate<=2 : 0.994468, exp_rate<=3 : 0.998067, loss: 0.001305, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17692 s, avg_samples: 56.0, ips: 316.52845 samples/s, eta: 17:58:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:10:07] ppocr INFO: epoch: [378/500], global_step: 211400, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.940426, exp_rate<=1 : 0.978494, exp_rate<=2 : 0.994852, exp_rate<=3 : 0.998402, loss: 0.002032, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17869 s, avg_samples: 56.0, ips: 313.39311 samples/s, eta: 17:57:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:10:25] ppocr INFO: epoch: [378/500], global_step: 211500, lr: 0.000100, edit distance: 0.001398, exp_rate: 0.938655, exp_rate<=1 : 0.977752, exp_rate<=2 : 0.994843, exp_rate<=3 : 0.998370, loss: 0.001301, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17961 s, avg_samples: 56.0, ips: 311.78830 samples/s, eta: 17:57:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:10:45] ppocr INFO: epoch: [378/500], global_step: 211600, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.937867, exp_rate<=1 : 0.976914, exp_rate<=2 : 0.994474, exp_rate<=3 : 0.998201, loss: 0.001089, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19417 s, avg_samples: 56.0, ips: 288.40429 samples/s, eta: 17:56:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:11:05] ppocr INFO: epoch: [378/500], global_step: 211700, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.937015, exp_rate<=1 : 0.976095, exp_rate<=2 : 0.993872, exp_rate<=3 : 0.997960, loss: 0.001770, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20489 s, avg_samples: 56.0, ips: 273.32174 samples/s, eta: 17:56:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:11:24] ppocr INFO: epoch: [378/500], global_step: 211800, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.937290, exp_rate<=1 : 0.976238, exp_rate<=2 : 0.993930, exp_rate<=3 : 0.997959, loss: 0.001426, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18392 s, avg_samples: 56.0, ips: 304.48309 samples/s, eta: 17:56:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:11:44] ppocr INFO: epoch: [378/500], global_step: 211900, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.936244, exp_rate<=1 : 0.975597, exp_rate<=2 : 0.993802, exp_rate<=3 : 0.997904, loss: 0.001070, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20653 s, avg_samples: 56.0, ips: 271.14932 samples/s, eta: 17:56:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:12:02] ppocr INFO: epoch: [378/500], global_step: 212000, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.937195, exp_rate<=1 : 0.976008, exp_rate<=2 : 0.994033, exp_rate<=3 : 0.998003, loss: 0.001028, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17682 s, avg_samples: 56.0, ips: 316.70992 samples/s, eta: 17:55:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:12:23] ppocr INFO: epoch: [378/500], global_step: 212100, lr: 0.000100, edit distance: 0.001403, exp_rate: 0.936278, exp_rate<=1 : 0.975584, exp_rate<=2 : 0.993843, exp_rate<=3 : 0.997849, loss: 0.001199, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21232 s, avg_samples: 56.0, ips: 263.75487 samples/s, eta: 17:55:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:12:42] ppocr INFO: epoch: [378/500], global_step: 212200, lr: 0.000100, edit distance: 0.001390, exp_rate: 0.937241, exp_rate<=1 : 0.975848, exp_rate<=2 : 0.993991, exp_rate<=3 : 0.997859, loss: 0.001144, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18393 s, avg_samples: 56.0, ips: 304.47099 samples/s, eta: 17:55:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:13:00] ppocr INFO: epoch: [378/500], global_step: 212300, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.937428, exp_rate<=1 : 0.976036, exp_rate<=2 : 0.994166, exp_rate<=3 : 0.997902, loss: 0.001373, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18262 s, avg_samples: 56.0, ips: 306.65410 samples/s, eta: 17:54:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:13:18] ppocr INFO: epoch: [378/500], global_step: 212400, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.937706, exp_rate<=1 : 0.976207, exp_rate<=2 : 0.994348, exp_rate<=3 : 0.997989, loss: 0.001717, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18074 s, avg_samples: 56.0, ips: 309.83079 samples/s, eta: 17:54:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:13:36] ppocr INFO: epoch: [378/500], global_step: 212500, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.938088, exp_rate<=1 : 0.976350, exp_rate<=2 : 0.994442, exp_rate<=3 : 0.998026, loss: 0.001054, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17941 s, avg_samples: 56.0, ips: 312.14137 samples/s, eta: 17:54:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:13:54] ppocr INFO: epoch: [378/500], global_step: 212600, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.937578, exp_rate<=1 : 0.976290, exp_rate<=2 : 0.994419, exp_rate<=3 : 0.998073, loss: 0.001835, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18257 s, avg_samples: 56.0, ips: 306.72481 samples/s, eta: 17:53:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:14:12] ppocr INFO: epoch: [378/500], global_step: 212700, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.937594, exp_rate<=1 : 0.976359, exp_rate<=2 : 0.994477, exp_rate<=3 : 0.998119, loss: 0.001270, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17443 s, avg_samples: 56.0, ips: 321.04505 samples/s, eta: 17:53:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:14:32] ppocr INFO: epoch: [378/500], global_step: 212800, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.937137, exp_rate<=1 : 0.976134, exp_rate<=2 : 0.994405, exp_rate<=3 : 0.998051, loss: 0.001617, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20225 s, avg_samples: 56.0, ips: 276.89186 samples/s, eta: 17:53:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:14:52] ppocr INFO: epoch: [378/500], global_step: 212900, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.936857, exp_rate<=1 : 0.975975, exp_rate<=2 : 0.994372, exp_rate<=3 : 0.998065, loss: 0.001815, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19716 s, avg_samples: 56.0, ips: 284.02890 samples/s, eta: 17:52:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:15:10] ppocr INFO: epoch: [378/500], global_step: 213000, lr: 0.000100, edit distance: 0.001383, exp_rate: 0.936700, exp_rate<=1 : 0.975702, exp_rate<=2 : 0.994299, exp_rate<=3 : 0.998044, loss: 0.001729, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17874 s, avg_samples: 56.0, ips: 313.30882 samples/s, eta: 17:52:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:15:28] ppocr INFO: epoch: [378/500], global_step: 213100, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.936743, exp_rate<=1 : 0.975838, exp_rate<=2 : 0.994386, exp_rate<=3 : 0.998072, loss: 0.001126, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18206 s, avg_samples: 56.0, ips: 307.59279 samples/s, eta: 17:52:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:15:48] ppocr INFO: epoch: [378/500], global_step: 213200, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.936168, exp_rate<=1 : 0.975539, exp_rate<=2 : 0.994271, exp_rate<=3 : 0.997981, loss: 0.001980, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19561 s, avg_samples: 56.0, ips: 286.28349 samples/s, eta: 17:51:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:16:06] ppocr INFO: epoch: [378/500], global_step: 213300, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.935836, exp_rate<=1 : 0.975359, exp_rate<=2 : 0.994245, exp_rate<=3 : 0.997960, loss: 0.001611, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18203 s, avg_samples: 56.0, ips: 307.64229 samples/s, eta: 17:51:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:16:24] ppocr INFO: epoch: [378/500], global_step: 213400, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.935867, exp_rate<=1 : 0.975299, exp_rate<=2 : 0.994226, exp_rate<=3 : 0.997973, loss: 0.001580, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18274 s, avg_samples: 56.0, ips: 306.45426 samples/s, eta: 17:51:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:16:43] ppocr INFO: epoch: [378/500], global_step: 213500, lr: 0.000100, edit distance: 0.001398, exp_rate: 0.935794, exp_rate<=1 : 0.975288, exp_rate<=2 : 0.994187, exp_rate<=3 : 0.997936, loss: 0.001719, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18664 s, avg_samples: 56.0, ips: 300.04988 samples/s, eta: 17:50:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:17:01] ppocr INFO: epoch: [378/500], global_step: 213600, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.935489, exp_rate<=1 : 0.975123, exp_rate<=2 : 0.994175, exp_rate<=3 : 0.997900, loss: 0.001879, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18682 s, avg_samples: 56.0, ips: 299.75141 samples/s, eta: 17:50:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:17:21] ppocr INFO: epoch: [378/500], global_step: 213700, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.935318, exp_rate<=1 : 0.975091, exp_rate<=2 : 0.994193, exp_rate<=3 : 0.997909, loss: 0.002055, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19081 s, avg_samples: 56.0, ips: 293.49081 samples/s, eta: 17:50:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:17:24] ppocr INFO: epoch: [378/500], global_step: 213720, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.935274, exp_rate<=1 : 0.975091, exp_rate<=2 : 0.994180, exp_rate<=3 : 0.997901, loss: 0.001704, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03246 s, avg_samples: 11.2, ips: 345.03513 samples/s, eta: 17:49:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:17:26] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 14:17:41] ppocr INFO: epoch: [379/500], global_step: 213800, lr: 0.000100, edit distance: 0.001095, exp_rate: 0.945646, exp_rate<=1 : 0.981893, exp_rate<=2 : 0.996503, exp_rate<=3 : 0.999747, loss: 0.001555, avg_reader_cost: 0.02555 s, avg_batch_cost: 0.17009 s, avg_samples: 44.8, ips: 263.39024 samples/s, eta: 17:49:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:17:59] ppocr INFO: epoch: [379/500], global_step: 213900, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.946739, exp_rate<=1 : 0.982038, exp_rate<=2 : 0.996203, exp_rate<=3 : 0.999057, loss: 0.001423, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17916 s, avg_samples: 56.0, ips: 312.57302 samples/s, eta: 17:49:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:18:17] ppocr INFO: epoch: [379/500], global_step: 214000, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.942269, exp_rate<=1 : 0.980173, exp_rate<=2 : 0.995221, exp_rate<=3 : 0.998955, loss: 0.001842, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17980 s, avg_samples: 56.0, ips: 311.45348 samples/s, eta: 17:49:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:18:35] ppocr INFO: epoch: [379/500], global_step: 214100, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.941711, exp_rate<=1 : 0.979480, exp_rate<=2 : 0.995141, exp_rate<=3 : 0.998768, loss: 0.001728, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18211 s, avg_samples: 56.0, ips: 307.50560 samples/s, eta: 17:48:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:18:53] ppocr INFO: epoch: [379/500], global_step: 214200, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.942391, exp_rate<=1 : 0.979186, exp_rate<=2 : 0.995066, exp_rate<=3 : 0.998672, loss: 0.001054, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18179 s, avg_samples: 56.0, ips: 308.04728 samples/s, eta: 17:48:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:19:13] ppocr INFO: epoch: [379/500], global_step: 214300, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.941212, exp_rate<=1 : 0.978550, exp_rate<=2 : 0.994822, exp_rate<=3 : 0.998659, loss: 0.002612, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19760 s, avg_samples: 56.0, ips: 283.39817 samples/s, eta: 17:48:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:19:31] ppocr INFO: epoch: [379/500], global_step: 214400, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.939671, exp_rate<=1 : 0.977642, exp_rate<=2 : 0.994297, exp_rate<=3 : 0.998296, loss: 0.001663, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18450 s, avg_samples: 56.0, ips: 303.51901 samples/s, eta: 17:47:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:19:50] ppocr INFO: epoch: [379/500], global_step: 214500, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.940357, exp_rate<=1 : 0.978100, exp_rate<=2 : 0.994603, exp_rate<=3 : 0.998414, loss: 0.001250, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18235 s, avg_samples: 56.0, ips: 307.09548 samples/s, eta: 17:47:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:20:08] ppocr INFO: epoch: [379/500], global_step: 214600, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.940288, exp_rate<=1 : 0.977814, exp_rate<=2 : 0.994456, exp_rate<=3 : 0.998258, loss: 0.001627, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18454 s, avg_samples: 56.0, ips: 303.45116 samples/s, eta: 17:47:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:20:26] ppocr INFO: epoch: [379/500], global_step: 214700, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.941188, exp_rate<=1 : 0.978380, exp_rate<=2 : 0.994615, exp_rate<=3 : 0.998302, loss: 0.001008, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17892 s, avg_samples: 56.0, ips: 312.99434 samples/s, eta: 17:46:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:20:44] ppocr INFO: epoch: [379/500], global_step: 214800, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.941406, exp_rate<=1 : 0.978625, exp_rate<=2 : 0.994734, exp_rate<=3 : 0.998333, loss: 0.001080, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18266 s, avg_samples: 56.0, ips: 306.57977 samples/s, eta: 17:46:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:21:05] ppocr INFO: epoch: [379/500], global_step: 214900, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.941421, exp_rate<=1 : 0.978660, exp_rate<=2 : 0.994774, exp_rate<=3 : 0.998340, loss: 0.001164, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20533 s, avg_samples: 56.0, ips: 272.73389 samples/s, eta: 17:46:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:21:24] ppocr INFO: epoch: [379/500], global_step: 215000, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.940903, exp_rate<=1 : 0.978328, exp_rate<=2 : 0.994771, exp_rate<=3 : 0.998270, loss: 0.001655, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19288 s, avg_samples: 56.0, ips: 290.33984 samples/s, eta: 17:45:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:21:43] ppocr INFO: epoch: [379/500], global_step: 215100, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.940094, exp_rate<=1 : 0.977745, exp_rate<=2 : 0.994625, exp_rate<=3 : 0.998265, loss: 0.001458, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18557 s, avg_samples: 56.0, ips: 301.77139 samples/s, eta: 17:45:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:22:02] ppocr INFO: epoch: [379/500], global_step: 215200, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.939750, exp_rate<=1 : 0.977372, exp_rate<=2 : 0.994509, exp_rate<=3 : 0.998230, loss: 0.001953, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19323 s, avg_samples: 56.0, ips: 289.81589 samples/s, eta: 17:45:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:22:20] ppocr INFO: epoch: [379/500], global_step: 215300, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.939503, exp_rate<=1 : 0.977305, exp_rate<=2 : 0.994451, exp_rate<=3 : 0.998150, loss: 0.001901, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17902 s, avg_samples: 56.0, ips: 312.80581 samples/s, eta: 17:44:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:22:38] ppocr INFO: epoch: [379/500], global_step: 215400, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.939593, exp_rate<=1 : 0.977370, exp_rate<=2 : 0.994457, exp_rate<=3 : 0.998157, loss: 0.001160, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17804 s, avg_samples: 56.0, ips: 314.53098 samples/s, eta: 17:44:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:22:58] ppocr INFO: epoch: [379/500], global_step: 215500, lr: 0.000100, edit distance: 0.001359, exp_rate: 0.938503, exp_rate<=1 : 0.976879, exp_rate<=2 : 0.994281, exp_rate<=3 : 0.998078, loss: 0.001975, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20482 s, avg_samples: 56.0, ips: 273.41546 samples/s, eta: 17:44:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:23:17] ppocr INFO: epoch: [379/500], global_step: 215600, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.937962, exp_rate<=1 : 0.976644, exp_rate<=2 : 0.994297, exp_rate<=3 : 0.998103, loss: 0.001751, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18573 s, avg_samples: 56.0, ips: 301.50601 samples/s, eta: 17:43:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:23:37] ppocr INFO: epoch: [379/500], global_step: 215700, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.937865, exp_rate<=1 : 0.976571, exp_rate<=2 : 0.994261, exp_rate<=3 : 0.998081, loss: 0.002017, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20021 s, avg_samples: 56.0, ips: 279.71035 samples/s, eta: 17:43:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:23:56] ppocr INFO: epoch: [379/500], global_step: 215800, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.937338, exp_rate<=1 : 0.976203, exp_rate<=2 : 0.994108, exp_rate<=3 : 0.998006, loss: 0.001972, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19124 s, avg_samples: 56.0, ips: 292.81969 samples/s, eta: 17:43:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:24:15] ppocr INFO: epoch: [379/500], global_step: 215900, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.937475, exp_rate<=1 : 0.976223, exp_rate<=2 : 0.994138, exp_rate<=3 : 0.998020, loss: 0.001234, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18690 s, avg_samples: 56.0, ips: 299.61772 samples/s, eta: 17:42:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:24:34] ppocr INFO: epoch: [379/500], global_step: 216000, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.937282, exp_rate<=1 : 0.976183, exp_rate<=2 : 0.994122, exp_rate<=3 : 0.998027, loss: 0.001804, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19374 s, avg_samples: 56.0, ips: 289.05410 samples/s, eta: 17:42:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:24:53] ppocr INFO: epoch: [379/500], global_step: 216100, lr: 0.000100, edit distance: 0.001383, exp_rate: 0.936666, exp_rate<=1 : 0.976045, exp_rate<=2 : 0.994109, exp_rate<=3 : 0.998033, loss: 0.001345, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18833 s, avg_samples: 56.0, ips: 297.35437 samples/s, eta: 17:42:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:25:13] ppocr INFO: epoch: [379/500], global_step: 216200, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.936168, exp_rate<=1 : 0.975921, exp_rate<=2 : 0.994127, exp_rate<=3 : 0.998019, loss: 0.001427, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20342 s, avg_samples: 56.0, ips: 275.29753 samples/s, eta: 17:41:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:25:32] ppocr INFO: epoch: [379/500], global_step: 216300, lr: 0.000100, edit distance: 0.001377, exp_rate: 0.936234, exp_rate<=1 : 0.975937, exp_rate<=2 : 0.994197, exp_rate<=3 : 0.998055, loss: 0.001484, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18114 s, avg_samples: 56.0, ips: 309.15000 samples/s, eta: 17:41:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:25:51] ppocr INFO: epoch: [379/500], global_step: 216400, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.936271, exp_rate<=1 : 0.975963, exp_rate<=2 : 0.994243, exp_rate<=3 : 0.998081, loss: 0.001314, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19819 s, avg_samples: 56.0, ips: 282.55425 samples/s, eta: 17:41:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:26:03] ppocr INFO: epoch: [379/500], global_step: 216460, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.936374, exp_rate<=1 : 0.976040, exp_rate<=2 : 0.994277, exp_rate<=3 : 0.998095, loss: 0.001733, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11237 s, avg_samples: 33.6, ips: 299.01524 samples/s, eta: 17:41:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:26:05] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 14:26:13] ppocr INFO: epoch: [380/500], global_step: 216500, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.944161, exp_rate<=1 : 0.979041, exp_rate<=2 : 0.995315, exp_rate<=3 : 0.998243, loss: 0.001358, avg_reader_cost: 0.02549 s, avg_batch_cost: 0.10069 s, avg_samples: 22.4, ips: 222.46013 samples/s, eta: 17:40:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:26:31] ppocr INFO: epoch: [380/500], global_step: 216600, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.942477, exp_rate<=1 : 0.977695, exp_rate<=2 : 0.994138, exp_rate<=3 : 0.998402, loss: 0.001269, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18079 s, avg_samples: 56.0, ips: 309.75357 samples/s, eta: 17:40:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:26:49] ppocr INFO: epoch: [380/500], global_step: 216700, lr: 0.000100, edit distance: 0.001308, exp_rate: 0.941676, exp_rate<=1 : 0.977862, exp_rate<=2 : 0.994734, exp_rate<=3 : 0.998373, loss: 0.001340, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17904 s, avg_samples: 56.0, ips: 312.77377 samples/s, eta: 17:40:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:27:09] ppocr INFO: epoch: [380/500], global_step: 216800, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.942204, exp_rate<=1 : 0.977679, exp_rate<=2 : 0.994300, exp_rate<=3 : 0.998244, loss: 0.001564, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20196 s, avg_samples: 56.0, ips: 277.28375 samples/s, eta: 17:39:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:27:27] ppocr INFO: epoch: [380/500], global_step: 216900, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.942629, exp_rate<=1 : 0.978718, exp_rate<=2 : 0.994748, exp_rate<=3 : 0.998299, loss: 0.001262, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18390 s, avg_samples: 56.0, ips: 304.51795 samples/s, eta: 17:39:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:27:45] ppocr INFO: epoch: [380/500], global_step: 217000, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.944070, exp_rate<=1 : 0.979616, exp_rate<=2 : 0.995061, exp_rate<=3 : 0.998424, loss: 0.001478, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17697 s, avg_samples: 56.0, ips: 316.43371 samples/s, eta: 17:39:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:28:03] ppocr INFO: epoch: [380/500], global_step: 217100, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.944630, exp_rate<=1 : 0.979705, exp_rate<=2 : 0.995152, exp_rate<=3 : 0.998422, loss: 0.001382, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18184 s, avg_samples: 56.0, ips: 307.97026 samples/s, eta: 17:38:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:28:22] ppocr INFO: epoch: [380/500], global_step: 217200, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.944415, exp_rate<=1 : 0.979443, exp_rate<=2 : 0.995325, exp_rate<=3 : 0.998509, loss: 0.001723, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18779 s, avg_samples: 56.0, ips: 298.20573 samples/s, eta: 17:38:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:28:41] ppocr INFO: epoch: [380/500], global_step: 217300, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.944318, exp_rate<=1 : 0.979412, exp_rate<=2 : 0.995312, exp_rate<=3 : 0.998470, loss: 0.001252, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18499 s, avg_samples: 56.0, ips: 302.71729 samples/s, eta: 17:38:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:28:59] ppocr INFO: epoch: [380/500], global_step: 217400, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.943940, exp_rate<=1 : 0.979229, exp_rate<=2 : 0.995259, exp_rate<=3 : 0.998409, loss: 0.000924, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18336 s, avg_samples: 56.0, ips: 305.40610 samples/s, eta: 17:37:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:29:16] ppocr INFO: epoch: [380/500], global_step: 217500, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.943882, exp_rate<=1 : 0.979178, exp_rate<=2 : 0.995299, exp_rate<=3 : 0.998426, loss: 0.001471, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17214 s, avg_samples: 56.0, ips: 325.31862 samples/s, eta: 17:37:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:29:35] ppocr INFO: epoch: [380/500], global_step: 217600, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.943366, exp_rate<=1 : 0.979085, exp_rate<=2 : 0.995243, exp_rate<=3 : 0.998364, loss: 0.001611, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18990 s, avg_samples: 56.0, ips: 294.89495 samples/s, eta: 17:37:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:29:56] ppocr INFO: epoch: [380/500], global_step: 217700, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.941512, exp_rate<=1 : 0.978249, exp_rate<=2 : 0.994954, exp_rate<=3 : 0.998227, loss: 0.001465, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20527 s, avg_samples: 56.0, ips: 272.81504 samples/s, eta: 17:36:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:30:14] ppocr INFO: epoch: [380/500], global_step: 217800, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.941500, exp_rate<=1 : 0.978239, exp_rate<=2 : 0.994949, exp_rate<=3 : 0.998208, loss: 0.001272, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18492 s, avg_samples: 56.0, ips: 302.83603 samples/s, eta: 17:36:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:30:33] ppocr INFO: epoch: [380/500], global_step: 217900, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.941083, exp_rate<=1 : 0.978007, exp_rate<=2 : 0.994916, exp_rate<=3 : 0.998184, loss: 0.001663, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18276 s, avg_samples: 56.0, ips: 306.40495 samples/s, eta: 17:36:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:30:53] ppocr INFO: epoch: [380/500], global_step: 218000, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.940107, exp_rate<=1 : 0.977626, exp_rate<=2 : 0.994757, exp_rate<=3 : 0.998113, loss: 0.001960, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20241 s, avg_samples: 56.0, ips: 276.66213 samples/s, eta: 17:35:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:31:12] ppocr INFO: epoch: [380/500], global_step: 218100, lr: 0.000100, edit distance: 0.001277, exp_rate: 0.939688, exp_rate<=1 : 0.977351, exp_rate<=2 : 0.994765, exp_rate<=3 : 0.998159, loss: 0.000919, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18865 s, avg_samples: 56.0, ips: 296.84679 samples/s, eta: 17:35:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:31:30] ppocr INFO: epoch: [380/500], global_step: 218200, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.939799, exp_rate<=1 : 0.977437, exp_rate<=2 : 0.994815, exp_rate<=3 : 0.998143, loss: 0.001185, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17971 s, avg_samples: 56.0, ips: 311.61602 samples/s, eta: 17:35:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:31:48] ppocr INFO: epoch: [380/500], global_step: 218300, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939733, exp_rate<=1 : 0.977440, exp_rate<=2 : 0.994703, exp_rate<=3 : 0.998166, loss: 0.001163, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18675 s, avg_samples: 56.0, ips: 299.86867 samples/s, eta: 17:34:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:32:09] ppocr INFO: epoch: [380/500], global_step: 218400, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.939488, exp_rate<=1 : 0.977188, exp_rate<=2 : 0.994600, exp_rate<=3 : 0.998091, loss: 0.001328, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20149 s, avg_samples: 56.0, ips: 277.92839 samples/s, eta: 17:34:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:32:28] ppocr INFO: epoch: [380/500], global_step: 218500, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939363, exp_rate<=1 : 0.977132, exp_rate<=2 : 0.994612, exp_rate<=3 : 0.998083, loss: 0.001538, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19819 s, avg_samples: 56.0, ips: 282.55590 samples/s, eta: 17:34:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:32:48] ppocr INFO: epoch: [380/500], global_step: 218600, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.938640, exp_rate<=1 : 0.976766, exp_rate<=2 : 0.994495, exp_rate<=3 : 0.998085, loss: 0.001996, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19795 s, avg_samples: 56.0, ips: 282.89450 samples/s, eta: 17:34:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:33:06] ppocr INFO: epoch: [380/500], global_step: 218700, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.938418, exp_rate<=1 : 0.976591, exp_rate<=2 : 0.994447, exp_rate<=3 : 0.998079, loss: 0.001757, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17796 s, avg_samples: 56.0, ips: 314.67044 samples/s, eta: 17:33:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:33:25] ppocr INFO: epoch: [380/500], global_step: 218800, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.938163, exp_rate<=1 : 0.976436, exp_rate<=2 : 0.994399, exp_rate<=3 : 0.998070, loss: 0.001779, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19265 s, avg_samples: 56.0, ips: 290.68025 samples/s, eta: 17:33:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:33:45] ppocr INFO: epoch: [380/500], global_step: 218900, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.937849, exp_rate<=1 : 0.976387, exp_rate<=2 : 0.994400, exp_rate<=3 : 0.998044, loss: 0.002228, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19148 s, avg_samples: 56.0, ips: 292.45596 samples/s, eta: 17:33:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:34:03] ppocr INFO: epoch: [380/500], global_step: 219000, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.937421, exp_rate<=1 : 0.976248, exp_rate<=2 : 0.994341, exp_rate<=3 : 0.997983, loss: 0.001300, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18273 s, avg_samples: 56.0, ips: 306.46210 samples/s, eta: 17:32:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:34:21] ppocr INFO: epoch: [380/500], global_step: 219100, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.937104, exp_rate<=1 : 0.976127, exp_rate<=2 : 0.994399, exp_rate<=3 : 0.997992, loss: 0.001539, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18233 s, avg_samples: 56.0, ips: 307.13243 samples/s, eta: 17:32:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:34:40] ppocr INFO: epoch: [380/500], global_step: 219200, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.936812, exp_rate<=1 : 0.976042, exp_rate<=2 : 0.994433, exp_rate<=3 : 0.998005, loss: 0.001763, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18827 s, avg_samples: 56.0, ips: 297.44488 samples/s, eta: 17:32:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:34:43] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 14:34:43] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_380 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 14:35:04] ppocr INFO: epoch: [381/500], global_step: 219300, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.930779, exp_rate<=1 : 0.970993, exp_rate<=2 : 0.992256, exp_rate<=3 : 0.997120, loss: 0.001387, avg_reader_cost: 0.03773 s, avg_batch_cost: 0.24435 s, avg_samples: 56.0, ips: 229.17979 samples/s, eta: 17:31:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:35:23] ppocr INFO: epoch: [381/500], global_step: 219400, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.938736, exp_rate<=1 : 0.974333, exp_rate<=2 : 0.993438, exp_rate<=3 : 0.997673, loss: 0.002092, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18524 s, avg_samples: 56.0, ips: 302.31149 samples/s, eta: 17:31:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:35:40] ppocr INFO: epoch: [381/500], global_step: 219500, lr: 0.000100, edit distance: 0.001100, exp_rate: 0.943190, exp_rate<=1 : 0.977385, exp_rate<=2 : 0.994624, exp_rate<=3 : 0.998060, loss: 0.001506, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17100 s, avg_samples: 56.0, ips: 327.48211 samples/s, eta: 17:31:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:35:59] ppocr INFO: epoch: [381/500], global_step: 219600, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.942308, exp_rate<=1 : 0.977021, exp_rate<=2 : 0.994169, exp_rate<=3 : 0.997839, loss: 0.001417, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18747 s, avg_samples: 56.0, ips: 298.71701 samples/s, eta: 17:30:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:36:18] ppocr INFO: epoch: [381/500], global_step: 219700, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.941721, exp_rate<=1 : 0.976789, exp_rate<=2 : 0.993993, exp_rate<=3 : 0.997738, loss: 0.001584, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18753 s, avg_samples: 56.0, ips: 298.61692 samples/s, eta: 17:30:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:36:38] ppocr INFO: epoch: [381/500], global_step: 219800, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.940692, exp_rate<=1 : 0.976579, exp_rate<=2 : 0.994155, exp_rate<=3 : 0.997762, loss: 0.001328, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20165 s, avg_samples: 56.0, ips: 277.70755 samples/s, eta: 17:30:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:36:57] ppocr INFO: epoch: [381/500], global_step: 219900, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.940324, exp_rate<=1 : 0.976651, exp_rate<=2 : 0.994260, exp_rate<=3 : 0.997879, loss: 0.001812, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18756 s, avg_samples: 56.0, ips: 298.57632 samples/s, eta: 17:29:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:37:16] ppocr INFO: epoch: [381/500], global_step: 220000, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.939912, exp_rate<=1 : 0.976618, exp_rate<=2 : 0.994385, exp_rate<=3 : 0.997888, loss: 0.001464, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19559 s, avg_samples: 56.0, ips: 286.31764 samples/s, eta: 17:29:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:37:34] ppocr INFO: epoch: [381/500], global_step: 220100, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.940217, exp_rate<=1 : 0.976671, exp_rate<=2 : 0.994376, exp_rate<=3 : 0.997932, loss: 0.001731, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18199 s, avg_samples: 56.0, ips: 307.70348 samples/s, eta: 17:29:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:37:54] ppocr INFO: epoch: [381/500], global_step: 220200, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.939095, exp_rate<=1 : 0.976341, exp_rate<=2 : 0.994356, exp_rate<=3 : 0.997896, loss: 0.001688, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19840 s, avg_samples: 56.0, ips: 282.26223 samples/s, eta: 17:28:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:38:12] ppocr INFO: epoch: [381/500], global_step: 220300, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.939207, exp_rate<=1 : 0.976371, exp_rate<=2 : 0.994458, exp_rate<=3 : 0.997934, loss: 0.001716, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17845 s, avg_samples: 56.0, ips: 313.80851 samples/s, eta: 17:28:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:38:31] ppocr INFO: epoch: [381/500], global_step: 220400, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.938805, exp_rate<=1 : 0.976168, exp_rate<=2 : 0.994298, exp_rate<=3 : 0.997915, loss: 0.001546, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19044 s, avg_samples: 56.0, ips: 294.05747 samples/s, eta: 17:28:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:38:50] ppocr INFO: epoch: [381/500], global_step: 220500, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.938873, exp_rate<=1 : 0.976354, exp_rate<=2 : 0.994279, exp_rate<=3 : 0.997908, loss: 0.001933, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18789 s, avg_samples: 56.0, ips: 298.04897 samples/s, eta: 17:27:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:39:09] ppocr INFO: epoch: [381/500], global_step: 220600, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.938033, exp_rate<=1 : 0.975855, exp_rate<=2 : 0.994315, exp_rate<=3 : 0.997921, loss: 0.001708, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18898 s, avg_samples: 56.0, ips: 296.32966 samples/s, eta: 17:27:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:39:27] ppocr INFO: epoch: [381/500], global_step: 220700, lr: 0.000100, edit distance: 0.001341, exp_rate: 0.937887, exp_rate<=1 : 0.975702, exp_rate<=2 : 0.994338, exp_rate<=3 : 0.997879, loss: 0.001236, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18089 s, avg_samples: 56.0, ips: 309.58371 samples/s, eta: 17:27:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:39:44] ppocr INFO: epoch: [381/500], global_step: 220800, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.938216, exp_rate<=1 : 0.975907, exp_rate<=2 : 0.994459, exp_rate<=3 : 0.997938, loss: 0.001469, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17421 s, avg_samples: 56.0, ips: 321.45856 samples/s, eta: 17:26:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:40:04] ppocr INFO: epoch: [381/500], global_step: 220900, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.937825, exp_rate<=1 : 0.975662, exp_rate<=2 : 0.994543, exp_rate<=3 : 0.997957, loss: 0.001654, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20024 s, avg_samples: 56.0, ips: 279.66415 samples/s, eta: 17:26:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:40:23] ppocr INFO: epoch: [381/500], global_step: 221000, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.937872, exp_rate<=1 : 0.975839, exp_rate<=2 : 0.994593, exp_rate<=3 : 0.998015, loss: 0.001689, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18962 s, avg_samples: 56.0, ips: 295.32895 samples/s, eta: 17:26:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:40:42] ppocr INFO: epoch: [381/500], global_step: 221100, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.937903, exp_rate<=1 : 0.975890, exp_rate<=2 : 0.994650, exp_rate<=3 : 0.998058, loss: 0.001103, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18452 s, avg_samples: 56.0, ips: 303.49621 samples/s, eta: 17:25:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:41:01] ppocr INFO: epoch: [381/500], global_step: 221200, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.937446, exp_rate<=1 : 0.975750, exp_rate<=2 : 0.994587, exp_rate<=3 : 0.998058, loss: 0.001689, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18916 s, avg_samples: 56.0, ips: 296.05176 samples/s, eta: 17:25:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:41:21] ppocr INFO: epoch: [381/500], global_step: 221300, lr: 0.000100, edit distance: 0.001341, exp_rate: 0.936924, exp_rate<=1 : 0.975424, exp_rate<=2 : 0.994575, exp_rate<=3 : 0.998061, loss: 0.000933, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20195 s, avg_samples: 56.0, ips: 277.29489 samples/s, eta: 17:25:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:41:40] ppocr INFO: epoch: [381/500], global_step: 221400, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.936811, exp_rate<=1 : 0.975406, exp_rate<=2 : 0.994563, exp_rate<=3 : 0.998050, loss: 0.003009, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18806 s, avg_samples: 56.0, ips: 297.77809 samples/s, eta: 17:24:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:41:59] ppocr INFO: epoch: [381/500], global_step: 221500, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.936131, exp_rate<=1 : 0.975172, exp_rate<=2 : 0.994536, exp_rate<=3 : 0.998047, loss: 0.001323, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18663 s, avg_samples: 56.0, ips: 300.05568 samples/s, eta: 17:24:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:42:16] ppocr INFO: epoch: [381/500], global_step: 221600, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.936088, exp_rate<=1 : 0.975211, exp_rate<=2 : 0.994586, exp_rate<=3 : 0.998086, loss: 0.001769, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17408 s, avg_samples: 56.0, ips: 321.68596 samples/s, eta: 17:24:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:42:34] ppocr INFO: epoch: [381/500], global_step: 221700, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.935828, exp_rate<=1 : 0.975070, exp_rate<=2 : 0.994486, exp_rate<=3 : 0.998016, loss: 0.001939, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18228 s, avg_samples: 56.0, ips: 307.21879 samples/s, eta: 17:23:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:42:53] ppocr INFO: epoch: [381/500], global_step: 221800, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.935403, exp_rate<=1 : 0.974888, exp_rate<=2 : 0.994367, exp_rate<=3 : 0.997980, loss: 0.001377, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18260 s, avg_samples: 56.0, ips: 306.68815 samples/s, eta: 17:23:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:43:10] ppocr INFO: epoch: [381/500], global_step: 221900, lr: 0.000100, edit distance: 0.001392, exp_rate: 0.935437, exp_rate<=1 : 0.974870, exp_rate<=2 : 0.994342, exp_rate<=3 : 0.997969, loss: 0.001348, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16968 s, avg_samples: 56.0, ips: 330.02426 samples/s, eta: 17:23:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:43:17] ppocr INFO: epoch: [381/500], global_step: 221940, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.935294, exp_rate<=1 : 0.974814, exp_rate<=2 : 0.994356, exp_rate<=3 : 0.997953, loss: 0.002322, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07412 s, avg_samples: 22.4, ips: 302.20681 samples/s, eta: 17:23:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:43:19] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 14:43:31] ppocr INFO: epoch: [382/500], global_step: 222000, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.932897, exp_rate<=1 : 0.972258, exp_rate<=2 : 0.992457, exp_rate<=3 : 0.998918, loss: 0.001364, avg_reader_cost: 0.02605 s, avg_batch_cost: 0.13709 s, avg_samples: 33.6, ips: 245.09196 samples/s, eta: 17:22:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:43:49] ppocr INFO: epoch: [382/500], global_step: 222100, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.943819, exp_rate<=1 : 0.977969, exp_rate<=2 : 0.994202, exp_rate<=3 : 0.998603, loss: 0.001181, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17842 s, avg_samples: 56.0, ips: 313.85831 samples/s, eta: 17:22:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:44:09] ppocr INFO: epoch: [382/500], global_step: 222200, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.940476, exp_rate<=1 : 0.976861, exp_rate<=2 : 0.994260, exp_rate<=3 : 0.998405, loss: 0.001394, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20460 s, avg_samples: 56.0, ips: 273.69856 samples/s, eta: 17:22:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:44:27] ppocr INFO: epoch: [382/500], global_step: 222300, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.942399, exp_rate<=1 : 0.977470, exp_rate<=2 : 0.994619, exp_rate<=3 : 0.998421, loss: 0.001449, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18003 s, avg_samples: 56.0, ips: 311.06313 samples/s, eta: 17:21:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:44:45] ppocr INFO: epoch: [382/500], global_step: 222400, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.940786, exp_rate<=1 : 0.976711, exp_rate<=2 : 0.994621, exp_rate<=3 : 0.998523, loss: 0.002502, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17558 s, avg_samples: 56.0, ips: 318.94403 samples/s, eta: 17:21:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:45:03] ppocr INFO: epoch: [382/500], global_step: 222500, lr: 0.000100, edit distance: 0.001443, exp_rate: 0.939964, exp_rate<=1 : 0.976196, exp_rate<=2 : 0.994707, exp_rate<=3 : 0.998463, loss: 0.001414, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17989 s, avg_samples: 56.0, ips: 311.30278 samples/s, eta: 17:21:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:45:21] ppocr INFO: epoch: [382/500], global_step: 222600, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.939722, exp_rate<=1 : 0.976296, exp_rate<=2 : 0.994620, exp_rate<=3 : 0.998315, loss: 0.001971, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18689 s, avg_samples: 56.0, ips: 299.64711 samples/s, eta: 17:20:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:45:42] ppocr INFO: epoch: [382/500], global_step: 222700, lr: 0.000100, edit distance: 0.001466, exp_rate: 0.938339, exp_rate<=1 : 0.975687, exp_rate<=2 : 0.994449, exp_rate<=3 : 0.998311, loss: 0.002094, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20214 s, avg_samples: 56.0, ips: 277.03313 samples/s, eta: 17:20:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:46:02] ppocr INFO: epoch: [382/500], global_step: 222800, lr: 0.000100, edit distance: 0.001469, exp_rate: 0.937931, exp_rate<=1 : 0.975519, exp_rate<=2 : 0.994129, exp_rate<=3 : 0.998217, loss: 0.001438, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19984 s, avg_samples: 56.0, ips: 280.21873 samples/s, eta: 17:20:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:46:20] ppocr INFO: epoch: [382/500], global_step: 222900, lr: 0.000100, edit distance: 0.001455, exp_rate: 0.938309, exp_rate<=1 : 0.975701, exp_rate<=2 : 0.994091, exp_rate<=3 : 0.998171, loss: 0.001665, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18518 s, avg_samples: 56.0, ips: 302.40987 samples/s, eta: 17:19:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:46:38] ppocr INFO: epoch: [382/500], global_step: 223000, lr: 0.000100, edit distance: 0.001451, exp_rate: 0.938506, exp_rate<=1 : 0.975665, exp_rate<=2 : 0.994204, exp_rate<=3 : 0.998142, loss: 0.001420, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17884 s, avg_samples: 56.0, ips: 313.12177 samples/s, eta: 17:19:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:46:56] ppocr INFO: epoch: [382/500], global_step: 223100, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.938813, exp_rate<=1 : 0.975862, exp_rate<=2 : 0.994213, exp_rate<=3 : 0.998156, loss: 0.001629, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18392 s, avg_samples: 56.0, ips: 304.47215 samples/s, eta: 17:19:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:47:15] ppocr INFO: epoch: [382/500], global_step: 223200, lr: 0.000100, edit distance: 0.001418, exp_rate: 0.939326, exp_rate<=1 : 0.976121, exp_rate<=2 : 0.994302, exp_rate<=3 : 0.998181, loss: 0.001646, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18375 s, avg_samples: 56.0, ips: 304.75968 samples/s, eta: 17:18:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:47:35] ppocr INFO: epoch: [382/500], global_step: 223300, lr: 0.000100, edit distance: 0.001428, exp_rate: 0.938092, exp_rate<=1 : 0.975651, exp_rate<=2 : 0.994160, exp_rate<=3 : 0.998142, loss: 0.002024, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20316 s, avg_samples: 56.0, ips: 275.63906 samples/s, eta: 17:18:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:47:53] ppocr INFO: epoch: [382/500], global_step: 223400, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.938252, exp_rate<=1 : 0.975749, exp_rate<=2 : 0.993998, exp_rate<=3 : 0.998066, loss: 0.001721, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18130 s, avg_samples: 56.0, ips: 308.87364 samples/s, eta: 17:18:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:48:10] ppocr INFO: epoch: [382/500], global_step: 223500, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.938554, exp_rate<=1 : 0.975968, exp_rate<=2 : 0.994103, exp_rate<=3 : 0.998064, loss: 0.001323, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17019 s, avg_samples: 56.0, ips: 329.04206 samples/s, eta: 17:17:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:48:27] ppocr INFO: epoch: [382/500], global_step: 223600, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.938853, exp_rate<=1 : 0.976113, exp_rate<=2 : 0.994201, exp_rate<=3 : 0.998150, loss: 0.001541, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17113 s, avg_samples: 56.0, ips: 327.24565 samples/s, eta: 17:17:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:48:46] ppocr INFO: epoch: [382/500], global_step: 223700, lr: 0.000100, edit distance: 0.001406, exp_rate: 0.939048, exp_rate<=1 : 0.976012, exp_rate<=2 : 0.994221, exp_rate<=3 : 0.998184, loss: 0.000848, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18517 s, avg_samples: 56.0, ips: 302.42622 samples/s, eta: 17:17:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:49:04] ppocr INFO: epoch: [382/500], global_step: 223800, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.939163, exp_rate<=1 : 0.975896, exp_rate<=2 : 0.994179, exp_rate<=3 : 0.998085, loss: 0.002140, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18436 s, avg_samples: 56.0, ips: 303.75225 samples/s, eta: 17:16:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:49:24] ppocr INFO: epoch: [382/500], global_step: 223900, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.938971, exp_rate<=1 : 0.975850, exp_rate<=2 : 0.994215, exp_rate<=3 : 0.998041, loss: 0.001279, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19767 s, avg_samples: 56.0, ips: 283.29828 samples/s, eta: 17:16:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:49:43] ppocr INFO: epoch: [382/500], global_step: 224000, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.938827, exp_rate<=1 : 0.975854, exp_rate<=2 : 0.994262, exp_rate<=3 : 0.998067, loss: 0.002135, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18866 s, avg_samples: 56.0, ips: 296.82839 samples/s, eta: 17:16:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:50:03] ppocr INFO: epoch: [382/500], global_step: 224100, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.938711, exp_rate<=1 : 0.975637, exp_rate<=2 : 0.994208, exp_rate<=3 : 0.998060, loss: 0.001350, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19650 s, avg_samples: 56.0, ips: 284.98263 samples/s, eta: 17:15:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:50:21] ppocr INFO: epoch: [382/500], global_step: 224200, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.939087, exp_rate<=1 : 0.975717, exp_rate<=2 : 0.994197, exp_rate<=3 : 0.998055, loss: 0.001601, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18427 s, avg_samples: 56.0, ips: 303.90647 samples/s, eta: 17:15:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:50:40] ppocr INFO: epoch: [382/500], global_step: 224300, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.939220, exp_rate<=1 : 0.975854, exp_rate<=2 : 0.994231, exp_rate<=3 : 0.998078, loss: 0.001262, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18325 s, avg_samples: 56.0, ips: 305.59910 samples/s, eta: 17:15:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:50:59] ppocr INFO: epoch: [382/500], global_step: 224400, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.939153, exp_rate<=1 : 0.975881, exp_rate<=2 : 0.994235, exp_rate<=3 : 0.998069, loss: 0.001657, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19670 s, avg_samples: 56.0, ips: 284.70072 samples/s, eta: 17:14:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:51:19] ppocr INFO: epoch: [382/500], global_step: 224500, lr: 0.000100, edit distance: 0.001370, exp_rate: 0.938631, exp_rate<=1 : 0.975670, exp_rate<=2 : 0.994232, exp_rate<=3 : 0.998098, loss: 0.001307, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19647 s, avg_samples: 56.0, ips: 285.03770 samples/s, eta: 17:14:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:51:38] ppocr INFO: epoch: [382/500], global_step: 224600, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.937874, exp_rate<=1 : 0.975419, exp_rate<=2 : 0.994122, exp_rate<=3 : 0.998010, loss: 0.001509, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19512 s, avg_samples: 56.0, ips: 287.00511 samples/s, eta: 17:14:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:51:54] ppocr INFO: epoch: [382/500], global_step: 224680, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.937382, exp_rate<=1 : 0.975190, exp_rate<=2 : 0.994064, exp_rate<=3 : 0.997998, loss: 0.002076, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15720 s, avg_samples: 44.8, ips: 284.98933 samples/s, eta: 17:14:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:51:56] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 14:52:01] ppocr INFO: epoch: [383/500], global_step: 224700, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.941008, exp_rate<=1 : 0.978256, exp_rate<=2 : 0.994390, exp_rate<=3 : 0.998675, loss: 0.001950, avg_reader_cost: 0.02657 s, avg_batch_cost: 0.06529 s, avg_samples: 11.2, ips: 171.54087 samples/s, eta: 17:14:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:52:20] ppocr INFO: epoch: [383/500], global_step: 224800, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.942718, exp_rate<=1 : 0.977622, exp_rate<=2 : 0.994334, exp_rate<=3 : 0.997874, loss: 0.001537, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18988 s, avg_samples: 56.0, ips: 294.92620 samples/s, eta: 17:13:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:52:38] ppocr INFO: epoch: [383/500], global_step: 224900, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.941664, exp_rate<=1 : 0.977261, exp_rate<=2 : 0.994425, exp_rate<=3 : 0.997869, loss: 0.001090, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18562 s, avg_samples: 56.0, ips: 301.69689 samples/s, eta: 17:13:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:52:56] ppocr INFO: epoch: [383/500], global_step: 225000, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.942419, exp_rate<=1 : 0.978175, exp_rate<=2 : 0.994882, exp_rate<=3 : 0.998110, loss: 0.001267, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18116 s, avg_samples: 56.0, ips: 309.12263 samples/s, eta: 17:13:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:53:16] ppocr INFO: epoch: [383/500], global_step: 225100, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.942294, exp_rate<=1 : 0.978467, exp_rate<=2 : 0.994625, exp_rate<=3 : 0.998023, loss: 0.001247, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19288 s, avg_samples: 56.0, ips: 290.33295 samples/s, eta: 17:12:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:53:34] ppocr INFO: epoch: [383/500], global_step: 225200, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.942566, exp_rate<=1 : 0.978206, exp_rate<=2 : 0.994697, exp_rate<=3 : 0.998139, loss: 0.001845, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17968 s, avg_samples: 56.0, ips: 311.66007 samples/s, eta: 17:12:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:53:53] ppocr INFO: epoch: [383/500], global_step: 225300, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.941953, exp_rate<=1 : 0.978047, exp_rate<=2 : 0.994539, exp_rate<=3 : 0.998040, loss: 0.001236, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19229 s, avg_samples: 56.0, ips: 291.22561 samples/s, eta: 17:12:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:54:12] ppocr INFO: epoch: [383/500], global_step: 225400, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.940313, exp_rate<=1 : 0.977704, exp_rate<=2 : 0.994619, exp_rate<=3 : 0.998067, loss: 0.001691, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19304 s, avg_samples: 56.0, ips: 290.09396 samples/s, eta: 17:11:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:54:30] ppocr INFO: epoch: [383/500], global_step: 225500, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.940127, exp_rate<=1 : 0.977864, exp_rate<=2 : 0.994870, exp_rate<=3 : 0.998189, loss: 0.001915, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17165 s, avg_samples: 56.0, ips: 326.23638 samples/s, eta: 17:11:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:54:49] ppocr INFO: epoch: [383/500], global_step: 225600, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.939333, exp_rate<=1 : 0.977271, exp_rate<=2 : 0.994772, exp_rate<=3 : 0.998166, loss: 0.002013, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19196 s, avg_samples: 56.0, ips: 291.73259 samples/s, eta: 17:11:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:55:07] ppocr INFO: epoch: [383/500], global_step: 225700, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.939038, exp_rate<=1 : 0.977115, exp_rate<=2 : 0.994610, exp_rate<=3 : 0.997981, loss: 0.001300, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18477 s, avg_samples: 56.0, ips: 303.08161 samples/s, eta: 17:10:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:55:25] ppocr INFO: epoch: [383/500], global_step: 225800, lr: 0.000100, edit distance: 0.001377, exp_rate: 0.938622, exp_rate<=1 : 0.977168, exp_rate<=2 : 0.994564, exp_rate<=3 : 0.998017, loss: 0.001524, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18135 s, avg_samples: 56.0, ips: 308.78978 samples/s, eta: 17:10:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:55:44] ppocr INFO: epoch: [383/500], global_step: 225900, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.938534, exp_rate<=1 : 0.977058, exp_rate<=2 : 0.994719, exp_rate<=3 : 0.998038, loss: 0.000942, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18630 s, avg_samples: 56.0, ips: 300.59774 samples/s, eta: 17:10:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:56:04] ppocr INFO: epoch: [383/500], global_step: 226000, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.938685, exp_rate<=1 : 0.977011, exp_rate<=2 : 0.994663, exp_rate<=3 : 0.998036, loss: 0.001335, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.19747 s, avg_samples: 56.0, ips: 283.58522 samples/s, eta: 17:09:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:56:23] ppocr INFO: epoch: [383/500], global_step: 226100, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.938337, exp_rate<=1 : 0.976919, exp_rate<=2 : 0.994583, exp_rate<=3 : 0.997903, loss: 0.001772, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19236 s, avg_samples: 56.0, ips: 291.12440 samples/s, eta: 17:09:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:56:41] ppocr INFO: epoch: [383/500], global_step: 226200, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.938765, exp_rate<=1 : 0.977088, exp_rate<=2 : 0.994677, exp_rate<=3 : 0.997925, loss: 0.001421, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18216 s, avg_samples: 56.0, ips: 307.41601 samples/s, eta: 17:09:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:57:00] ppocr INFO: epoch: [383/500], global_step: 226300, lr: 0.000100, edit distance: 0.001343, exp_rate: 0.938660, exp_rate<=1 : 0.977119, exp_rate<=2 : 0.994583, exp_rate<=3 : 0.997915, loss: 0.001470, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19135 s, avg_samples: 56.0, ips: 292.66428 samples/s, eta: 17:08:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:57:19] ppocr INFO: epoch: [383/500], global_step: 226400, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.938501, exp_rate<=1 : 0.976878, exp_rate<=2 : 0.994460, exp_rate<=3 : 0.997858, loss: 0.001522, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18484 s, avg_samples: 56.0, ips: 302.96061 samples/s, eta: 17:08:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:57:39] ppocr INFO: epoch: [383/500], global_step: 226500, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.938106, exp_rate<=1 : 0.976499, exp_rate<=2 : 0.994318, exp_rate<=3 : 0.997830, loss: 0.001314, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19901 s, avg_samples: 56.0, ips: 281.39426 samples/s, eta: 17:08:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:57:59] ppocr INFO: epoch: [383/500], global_step: 226600, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.937811, exp_rate<=1 : 0.976281, exp_rate<=2 : 0.994216, exp_rate<=3 : 0.997766, loss: 0.001011, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19957 s, avg_samples: 56.0, ips: 280.60799 samples/s, eta: 17:07:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:58:18] ppocr INFO: epoch: [383/500], global_step: 226700, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.937660, exp_rate<=1 : 0.976238, exp_rate<=2 : 0.994239, exp_rate<=3 : 0.997770, loss: 0.002120, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18667 s, avg_samples: 56.0, ips: 299.99427 samples/s, eta: 17:07:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:58:35] ppocr INFO: epoch: [383/500], global_step: 226800, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.937339, exp_rate<=1 : 0.976203, exp_rate<=2 : 0.994238, exp_rate<=3 : 0.997749, loss: 0.001582, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17355 s, avg_samples: 56.0, ips: 322.67824 samples/s, eta: 17:07:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:58:54] ppocr INFO: epoch: [383/500], global_step: 226900, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.937009, exp_rate<=1 : 0.976058, exp_rate<=2 : 0.994186, exp_rate<=3 : 0.997778, loss: 0.001866, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19242 s, avg_samples: 56.0, ips: 291.03728 samples/s, eta: 17:06:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:59:12] ppocr INFO: epoch: [383/500], global_step: 227000, lr: 0.000100, edit distance: 0.001358, exp_rate: 0.936551, exp_rate<=1 : 0.975922, exp_rate<=2 : 0.994194, exp_rate<=3 : 0.997813, loss: 0.001160, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18111 s, avg_samples: 56.0, ips: 309.21206 samples/s, eta: 17:06:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:59:31] ppocr INFO: epoch: [383/500], global_step: 227100, lr: 0.000100, edit distance: 0.001360, exp_rate: 0.936717, exp_rate<=1 : 0.976014, exp_rate<=2 : 0.994279, exp_rate<=3 : 0.997852, loss: 0.001806, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18337 s, avg_samples: 56.0, ips: 305.38927 samples/s, eta: 17:06:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 14:59:49] ppocr INFO: epoch: [383/500], global_step: 227200, lr: 0.000100, edit distance: 0.001359, exp_rate: 0.936716, exp_rate<=1 : 0.975960, exp_rate<=2 : 0.994236, exp_rate<=3 : 0.997831, loss: 0.001179, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18126 s, avg_samples: 56.0, ips: 308.94916 samples/s, eta: 17:05:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:00:07] ppocr INFO: epoch: [383/500], global_step: 227300, lr: 0.000100, edit distance: 0.001368, exp_rate: 0.936550, exp_rate<=1 : 0.975859, exp_rate<=2 : 0.994184, exp_rate<=3 : 0.997809, loss: 0.001858, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18330 s, avg_samples: 56.0, ips: 305.51489 samples/s, eta: 17:05:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:00:26] ppocr INFO: epoch: [383/500], global_step: 227400, lr: 0.000100, edit distance: 0.001393, exp_rate: 0.936200, exp_rate<=1 : 0.975733, exp_rate<=2 : 0.994129, exp_rate<=3 : 0.997795, loss: 0.002315, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18571 s, avg_samples: 56.0, ips: 301.54061 samples/s, eta: 17:05:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:00:29] ppocr INFO: epoch: [383/500], global_step: 227420, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.936008, exp_rate<=1 : 0.975598, exp_rate<=2 : 0.994094, exp_rate<=3 : 0.997790, loss: 0.001860, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03225 s, avg_samples: 11.2, ips: 347.27762 samples/s, eta: 17:04:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:00:31] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 15:00:48] ppocr INFO: epoch: [384/500], global_step: 227500, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.936525, exp_rate<=1 : 0.979226, exp_rate<=2 : 0.996059, exp_rate<=3 : 0.998562, loss: 0.001396, avg_reader_cost: 0.02627 s, avg_batch_cost: 0.18662 s, avg_samples: 44.8, ips: 240.05763 samples/s, eta: 17:04:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:01:08] ppocr INFO: epoch: [384/500], global_step: 227600, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.940372, exp_rate<=1 : 0.978383, exp_rate<=2 : 0.995731, exp_rate<=3 : 0.998126, loss: 0.001692, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20289 s, avg_samples: 56.0, ips: 276.00513 samples/s, eta: 17:04:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:01:27] ppocr INFO: epoch: [384/500], global_step: 227700, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.936384, exp_rate<=1 : 0.976941, exp_rate<=2 : 0.994564, exp_rate<=3 : 0.998020, loss: 0.001496, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19435 s, avg_samples: 56.0, ips: 288.14197 samples/s, eta: 17:04:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:01:46] ppocr INFO: epoch: [384/500], global_step: 227800, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.937274, exp_rate<=1 : 0.977277, exp_rate<=2 : 0.995018, exp_rate<=3 : 0.998024, loss: 0.002077, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18604 s, avg_samples: 56.0, ips: 301.00505 samples/s, eta: 17:03:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:02:05] ppocr INFO: epoch: [384/500], global_step: 227900, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.938919, exp_rate<=1 : 0.977896, exp_rate<=2 : 0.995208, exp_rate<=3 : 0.998161, loss: 0.001498, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18632 s, avg_samples: 56.0, ips: 300.55565 samples/s, eta: 17:03:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:02:23] ppocr INFO: epoch: [384/500], global_step: 228000, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.938876, exp_rate<=1 : 0.977921, exp_rate<=2 : 0.995140, exp_rate<=3 : 0.998139, loss: 0.001717, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17942 s, avg_samples: 56.0, ips: 312.11221 samples/s, eta: 17:03:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:02:41] ppocr INFO: epoch: [384/500], global_step: 228100, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.939400, exp_rate<=1 : 0.978204, exp_rate<=2 : 0.995078, exp_rate<=3 : 0.998160, loss: 0.001411, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18466 s, avg_samples: 56.0, ips: 303.25562 samples/s, eta: 17:02:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:02:59] ppocr INFO: epoch: [384/500], global_step: 228200, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.939901, exp_rate<=1 : 0.977849, exp_rate<=2 : 0.994981, exp_rate<=3 : 0.998112, loss: 0.002293, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18306 s, avg_samples: 56.0, ips: 305.91343 samples/s, eta: 17:02:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:03:18] ppocr INFO: epoch: [384/500], global_step: 228300, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.938848, exp_rate<=1 : 0.977217, exp_rate<=2 : 0.994866, exp_rate<=3 : 0.998152, loss: 0.001524, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18823 s, avg_samples: 56.0, ips: 297.50903 samples/s, eta: 17:02:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:03:38] ppocr INFO: epoch: [384/500], global_step: 228400, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.937800, exp_rate<=1 : 0.976657, exp_rate<=2 : 0.994697, exp_rate<=3 : 0.998049, loss: 0.001574, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19523 s, avg_samples: 56.0, ips: 286.84552 samples/s, eta: 17:01:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:03:56] ppocr INFO: epoch: [384/500], global_step: 228500, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.937358, exp_rate<=1 : 0.976454, exp_rate<=2 : 0.994647, exp_rate<=3 : 0.998021, loss: 0.001860, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18042 s, avg_samples: 56.0, ips: 310.38121 samples/s, eta: 17:01:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:04:12] ppocr INFO: epoch: [384/500], global_step: 228600, lr: 0.000100, edit distance: 0.001358, exp_rate: 0.937957, exp_rate<=1 : 0.976716, exp_rate<=2 : 0.994767, exp_rate<=3 : 0.998078, loss: 0.001330, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16528 s, avg_samples: 56.0, ips: 338.82919 samples/s, eta: 17:01:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:04:30] ppocr INFO: epoch: [384/500], global_step: 228700, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.937984, exp_rate<=1 : 0.976693, exp_rate<=2 : 0.994774, exp_rate<=3 : 0.998093, loss: 0.001395, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17917 s, avg_samples: 56.0, ips: 312.54836 samples/s, eta: 17:00:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:04:49] ppocr INFO: epoch: [384/500], global_step: 228800, lr: 0.000100, edit distance: 0.001358, exp_rate: 0.937633, exp_rate<=1 : 0.976392, exp_rate<=2 : 0.994726, exp_rate<=3 : 0.998126, loss: 0.001347, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18546 s, avg_samples: 56.0, ips: 301.95187 samples/s, eta: 17:00:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:05:08] ppocr INFO: epoch: [384/500], global_step: 228900, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.937315, exp_rate<=1 : 0.976050, exp_rate<=2 : 0.994598, exp_rate<=3 : 0.998095, loss: 0.001193, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19168 s, avg_samples: 56.0, ips: 292.15875 samples/s, eta: 17:00:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:05:26] ppocr INFO: epoch: [384/500], global_step: 229000, lr: 0.000100, edit distance: 0.001368, exp_rate: 0.937429, exp_rate<=1 : 0.976179, exp_rate<=2 : 0.994700, exp_rate<=3 : 0.998135, loss: 0.001563, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17477 s, avg_samples: 56.0, ips: 320.42672 samples/s, eta: 16:59:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:05:45] ppocr INFO: epoch: [384/500], global_step: 229100, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.937167, exp_rate<=1 : 0.976087, exp_rate<=2 : 0.994587, exp_rate<=3 : 0.998108, loss: 0.001554, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18837 s, avg_samples: 56.0, ips: 297.29417 samples/s, eta: 16:59:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:06:03] ppocr INFO: epoch: [384/500], global_step: 229200, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.937376, exp_rate<=1 : 0.976100, exp_rate<=2 : 0.994604, exp_rate<=3 : 0.998131, loss: 0.001576, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18616 s, avg_samples: 56.0, ips: 300.81530 samples/s, eta: 16:59:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:06:20] ppocr INFO: epoch: [384/500], global_step: 229300, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.937869, exp_rate<=1 : 0.976340, exp_rate<=2 : 0.994760, exp_rate<=3 : 0.998166, loss: 0.002288, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.17115 s, avg_samples: 56.0, ips: 327.20614 samples/s, eta: 16:58:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:06:39] ppocr INFO: epoch: [384/500], global_step: 229400, lr: 0.000100, edit distance: 0.001372, exp_rate: 0.937502, exp_rate<=1 : 0.975976, exp_rate<=2 : 0.994589, exp_rate<=3 : 0.998122, loss: 0.001003, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18962 s, avg_samples: 56.0, ips: 295.33047 samples/s, eta: 16:58:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:06:58] ppocr INFO: epoch: [384/500], global_step: 229500, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.937017, exp_rate<=1 : 0.975673, exp_rate<=2 : 0.994505, exp_rate<=3 : 0.998074, loss: 0.001855, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18929 s, avg_samples: 56.0, ips: 295.83999 samples/s, eta: 16:58:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:07:18] ppocr INFO: epoch: [384/500], global_step: 229600, lr: 0.000100, edit distance: 0.001395, exp_rate: 0.936367, exp_rate<=1 : 0.975343, exp_rate<=2 : 0.994413, exp_rate<=3 : 0.998029, loss: 0.001554, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19782 s, avg_samples: 56.0, ips: 283.07973 samples/s, eta: 16:57:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:07:36] ppocr INFO: epoch: [384/500], global_step: 229700, lr: 0.000100, edit distance: 0.001393, exp_rate: 0.936432, exp_rate<=1 : 0.975271, exp_rate<=2 : 0.994411, exp_rate<=3 : 0.998012, loss: 0.001611, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18365 s, avg_samples: 56.0, ips: 304.91990 samples/s, eta: 16:57:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:07:56] ppocr INFO: epoch: [384/500], global_step: 229800, lr: 0.000100, edit distance: 0.001386, exp_rate: 0.936470, exp_rate<=1 : 0.975133, exp_rate<=2 : 0.994421, exp_rate<=3 : 0.998025, loss: 0.001630, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19943 s, avg_samples: 56.0, ips: 280.80563 samples/s, eta: 16:57:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:08:15] ppocr INFO: epoch: [384/500], global_step: 229900, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.936489, exp_rate<=1 : 0.975211, exp_rate<=2 : 0.994458, exp_rate<=3 : 0.998045, loss: 0.001213, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18988 s, avg_samples: 56.0, ips: 294.92478 samples/s, eta: 16:56:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:08:34] ppocr INFO: epoch: [384/500], global_step: 230000, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.936696, exp_rate<=1 : 0.975306, exp_rate<=2 : 0.994480, exp_rate<=3 : 0.998041, loss: 0.001375, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18973 s, avg_samples: 56.0, ips: 295.16279 samples/s, eta: 16:56:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:08:54] ppocr INFO: epoch: [384/500], global_step: 230100, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.936318, exp_rate<=1 : 0.975183, exp_rate<=2 : 0.994382, exp_rate<=3 : 0.998002, loss: 0.001169, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19278 s, avg_samples: 56.0, ips: 290.48163 samples/s, eta: 16:56:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:09:06] ppocr INFO: epoch: [384/500], global_step: 230160, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.936300, exp_rate<=1 : 0.975197, exp_rate<=2 : 0.994356, exp_rate<=3 : 0.998005, loss: 0.001597, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.12201 s, avg_samples: 33.6, ips: 275.39078 samples/s, eta: 16:56:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:09:08] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 15:09:17] ppocr INFO: epoch: [385/500], global_step: 230200, lr: 0.000100, edit distance: 0.001031, exp_rate: 0.940860, exp_rate<=1 : 0.974761, exp_rate<=2 : 0.994048, exp_rate<=3 : 0.998161, loss: 0.001687, avg_reader_cost: 0.02688 s, avg_batch_cost: 0.11126 s, avg_samples: 22.4, ips: 201.33547 samples/s, eta: 16:55:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:09:36] ppocr INFO: epoch: [385/500], global_step: 230300, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.941205, exp_rate<=1 : 0.977712, exp_rate<=2 : 0.995117, exp_rate<=3 : 0.998167, loss: 0.000782, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18708 s, avg_samples: 56.0, ips: 299.32991 samples/s, eta: 16:55:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:09:53] ppocr INFO: epoch: [385/500], global_step: 230400, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.946118, exp_rate<=1 : 0.979147, exp_rate<=2 : 0.995555, exp_rate<=3 : 0.998451, loss: 0.000762, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17337 s, avg_samples: 56.0, ips: 323.00269 samples/s, eta: 16:55:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:10:13] ppocr INFO: epoch: [385/500], global_step: 230500, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.945626, exp_rate<=1 : 0.978717, exp_rate<=2 : 0.995529, exp_rate<=3 : 0.998340, loss: 0.001327, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19946 s, avg_samples: 56.0, ips: 280.75370 samples/s, eta: 16:54:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:10:32] ppocr INFO: epoch: [385/500], global_step: 230600, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.945152, exp_rate<=1 : 0.978351, exp_rate<=2 : 0.995238, exp_rate<=3 : 0.998246, loss: 0.001642, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18431 s, avg_samples: 56.0, ips: 303.83860 samples/s, eta: 16:54:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:10:49] ppocr INFO: epoch: [385/500], global_step: 230700, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.944733, exp_rate<=1 : 0.978716, exp_rate<=2 : 0.995098, exp_rate<=3 : 0.998127, loss: 0.001720, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17621 s, avg_samples: 56.0, ips: 317.80186 samples/s, eta: 16:54:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:11:08] ppocr INFO: epoch: [385/500], global_step: 230800, lr: 0.000100, edit distance: 0.001277, exp_rate: 0.944733, exp_rate<=1 : 0.978529, exp_rate<=2 : 0.995191, exp_rate<=3 : 0.998221, loss: 0.001476, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18337 s, avg_samples: 56.0, ips: 305.39919 samples/s, eta: 16:53:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:11:25] ppocr INFO: epoch: [385/500], global_step: 230900, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.944684, exp_rate<=1 : 0.978564, exp_rate<=2 : 0.995384, exp_rate<=3 : 0.998183, loss: 0.001382, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17033 s, avg_samples: 56.0, ips: 328.77863 samples/s, eta: 16:53:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:11:45] ppocr INFO: epoch: [385/500], global_step: 231000, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.943669, exp_rate<=1 : 0.978078, exp_rate<=2 : 0.995256, exp_rate<=3 : 0.998148, loss: 0.001733, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20176 s, avg_samples: 56.0, ips: 277.55064 samples/s, eta: 16:53:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:12:02] ppocr INFO: epoch: [385/500], global_step: 231100, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.944046, exp_rate<=1 : 0.978200, exp_rate<=2 : 0.995086, exp_rate<=3 : 0.998123, loss: 0.001906, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17372 s, avg_samples: 56.0, ips: 322.36017 samples/s, eta: 16:52:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:12:22] ppocr INFO: epoch: [385/500], global_step: 231200, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.941152, exp_rate<=1 : 0.976850, exp_rate<=2 : 0.994302, exp_rate<=3 : 0.997642, loss: 0.001494, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20031 s, avg_samples: 56.0, ips: 279.56570 samples/s, eta: 16:52:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:12:41] ppocr INFO: epoch: [385/500], global_step: 231300, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.941524, exp_rate<=1 : 0.976986, exp_rate<=2 : 0.994298, exp_rate<=3 : 0.997725, loss: 0.000937, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18767 s, avg_samples: 56.0, ips: 298.39147 samples/s, eta: 16:52:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:13:00] ppocr INFO: epoch: [385/500], global_step: 231400, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.941550, exp_rate<=1 : 0.976890, exp_rate<=2 : 0.994208, exp_rate<=3 : 0.997726, loss: 0.001316, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18818 s, avg_samples: 56.0, ips: 297.58594 samples/s, eta: 16:51:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:13:17] ppocr INFO: epoch: [385/500], global_step: 231500, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.940611, exp_rate<=1 : 0.976708, exp_rate<=2 : 0.994155, exp_rate<=3 : 0.997702, loss: 0.001871, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17267 s, avg_samples: 56.0, ips: 324.32126 samples/s, eta: 16:51:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:13:35] ppocr INFO: epoch: [385/500], global_step: 231600, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.940045, exp_rate<=1 : 0.976670, exp_rate<=2 : 0.994052, exp_rate<=3 : 0.997680, loss: 0.002420, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17911 s, avg_samples: 56.0, ips: 312.65055 samples/s, eta: 16:51:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:13:55] ppocr INFO: epoch: [385/500], global_step: 231700, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.938756, exp_rate<=1 : 0.976076, exp_rate<=2 : 0.993974, exp_rate<=3 : 0.997673, loss: 0.001747, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20400 s, avg_samples: 56.0, ips: 274.50617 samples/s, eta: 16:50:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:14:13] ppocr INFO: epoch: [385/500], global_step: 231800, lr: 0.000100, edit distance: 0.001386, exp_rate: 0.938727, exp_rate<=1 : 0.976189, exp_rate<=2 : 0.993991, exp_rate<=3 : 0.997707, loss: 0.001400, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17967 s, avg_samples: 56.0, ips: 311.67403 samples/s, eta: 16:50:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:14:32] ppocr INFO: epoch: [385/500], global_step: 231900, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.938858, exp_rate<=1 : 0.976349, exp_rate<=2 : 0.994103, exp_rate<=3 : 0.997773, loss: 0.001061, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18939 s, avg_samples: 56.0, ips: 295.68910 samples/s, eta: 16:50:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:14:51] ppocr INFO: epoch: [385/500], global_step: 232000, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.938942, exp_rate<=1 : 0.976576, exp_rate<=2 : 0.994193, exp_rate<=3 : 0.997782, loss: 0.001608, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18185 s, avg_samples: 56.0, ips: 307.94661 samples/s, eta: 16:49:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:15:11] ppocr INFO: epoch: [385/500], global_step: 232100, lr: 0.000100, edit distance: 0.001382, exp_rate: 0.938620, exp_rate<=1 : 0.976346, exp_rate<=2 : 0.994177, exp_rate<=3 : 0.997827, loss: 0.001200, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19992 s, avg_samples: 56.0, ips: 280.10595 samples/s, eta: 16:49:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:15:29] ppocr INFO: epoch: [385/500], global_step: 232200, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.938757, exp_rate<=1 : 0.976295, exp_rate<=2 : 0.994168, exp_rate<=3 : 0.997836, loss: 0.001360, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17909 s, avg_samples: 56.0, ips: 312.69457 samples/s, eta: 16:49:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:15:47] ppocr INFO: epoch: [385/500], global_step: 232300, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.938391, exp_rate<=1 : 0.976049, exp_rate<=2 : 0.994133, exp_rate<=3 : 0.997846, loss: 0.002392, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18453 s, avg_samples: 56.0, ips: 303.47906 samples/s, eta: 16:48:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:16:07] ppocr INFO: epoch: [385/500], global_step: 232400, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.937670, exp_rate<=1 : 0.975695, exp_rate<=2 : 0.994098, exp_rate<=3 : 0.997817, loss: 0.002400, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19712 s, avg_samples: 56.0, ips: 284.09673 samples/s, eta: 16:48:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:16:27] ppocr INFO: epoch: [385/500], global_step: 232500, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.937159, exp_rate<=1 : 0.975412, exp_rate<=2 : 0.994041, exp_rate<=3 : 0.997811, loss: 0.001882, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19771 s, avg_samples: 56.0, ips: 283.23642 samples/s, eta: 16:48:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:16:45] ppocr INFO: epoch: [385/500], global_step: 232600, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.937210, exp_rate<=1 : 0.975378, exp_rate<=2 : 0.994064, exp_rate<=3 : 0.997806, loss: 0.001877, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18280 s, avg_samples: 56.0, ips: 306.34715 samples/s, eta: 16:48:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:17:04] ppocr INFO: epoch: [385/500], global_step: 232700, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.936760, exp_rate<=1 : 0.975318, exp_rate<=2 : 0.994095, exp_rate<=3 : 0.997812, loss: 0.001775, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19594 s, avg_samples: 56.0, ips: 285.80245 samples/s, eta: 16:47:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:17:23] ppocr INFO: epoch: [385/500], global_step: 232800, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.936734, exp_rate<=1 : 0.975372, exp_rate<=2 : 0.994108, exp_rate<=3 : 0.997841, loss: 0.001959, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18448 s, avg_samples: 56.0, ips: 303.54957 samples/s, eta: 16:47:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:17:41] ppocr INFO: epoch: [385/500], global_step: 232900, lr: 0.000100, edit distance: 0.001446, exp_rate: 0.936709, exp_rate<=1 : 0.975345, exp_rate<=2 : 0.994098, exp_rate<=3 : 0.997849, loss: 0.002026, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18374 s, avg_samples: 56.0, ips: 304.78125 samples/s, eta: 16:47:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:17:43] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 15:17:44] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_385 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 15:18:03] ppocr INFO: epoch: [386/500], global_step: 233000, lr: 0.000100, edit distance: 0.001053, exp_rate: 0.943963, exp_rate<=1 : 0.979030, exp_rate<=2 : 0.995706, exp_rate<=3 : 0.998681, loss: 0.001148, avg_reader_cost: 0.03214 s, avg_batch_cost: 0.21753 s, avg_samples: 56.0, ips: 257.43446 samples/s, eta: 16:46:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:18:21] ppocr INFO: epoch: [386/500], global_step: 233100, lr: 0.000100, edit distance: 0.001129, exp_rate: 0.945309, exp_rate<=1 : 0.978440, exp_rate<=2 : 0.995313, exp_rate<=3 : 0.998011, loss: 0.000896, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18215 s, avg_samples: 56.0, ips: 307.44535 samples/s, eta: 16:46:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:18:39] ppocr INFO: epoch: [386/500], global_step: 233200, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.948822, exp_rate<=1 : 0.981214, exp_rate<=2 : 0.996038, exp_rate<=3 : 0.998279, loss: 0.001259, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17811 s, avg_samples: 56.0, ips: 314.42051 samples/s, eta: 16:46:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:18:57] ppocr INFO: epoch: [386/500], global_step: 233300, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.945904, exp_rate<=1 : 0.979144, exp_rate<=2 : 0.995433, exp_rate<=3 : 0.998125, loss: 0.001641, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18150 s, avg_samples: 56.0, ips: 308.53738 samples/s, eta: 16:45:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:19:16] ppocr INFO: epoch: [386/500], global_step: 233400, lr: 0.000100, edit distance: 0.001387, exp_rate: 0.943810, exp_rate<=1 : 0.978909, exp_rate<=2 : 0.995517, exp_rate<=3 : 0.998268, loss: 0.001256, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18486 s, avg_samples: 56.0, ips: 302.93071 samples/s, eta: 16:45:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:19:33] ppocr INFO: epoch: [386/500], global_step: 233500, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.943927, exp_rate<=1 : 0.978942, exp_rate<=2 : 0.995693, exp_rate<=3 : 0.998359, loss: 0.001735, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17414 s, avg_samples: 56.0, ips: 321.57729 samples/s, eta: 16:45:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:19:52] ppocr INFO: epoch: [386/500], global_step: 233600, lr: 0.000100, edit distance: 0.001416, exp_rate: 0.942566, exp_rate<=1 : 0.978635, exp_rate<=2 : 0.995701, exp_rate<=3 : 0.998392, loss: 0.001605, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18609 s, avg_samples: 56.0, ips: 300.92938 samples/s, eta: 16:44:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:20:11] ppocr INFO: epoch: [386/500], global_step: 233700, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.941863, exp_rate<=1 : 0.978186, exp_rate<=2 : 0.995699, exp_rate<=3 : 0.998372, loss: 0.002175, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19298 s, avg_samples: 56.0, ips: 290.18920 samples/s, eta: 16:44:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:20:29] ppocr INFO: epoch: [386/500], global_step: 233800, lr: 0.000100, edit distance: 0.001414, exp_rate: 0.941764, exp_rate<=1 : 0.978268, exp_rate<=2 : 0.995588, exp_rate<=3 : 0.998253, loss: 0.001599, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18185 s, avg_samples: 56.0, ips: 307.94033 samples/s, eta: 16:44:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:20:47] ppocr INFO: epoch: [386/500], global_step: 233900, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.942371, exp_rate<=1 : 0.978332, exp_rate<=2 : 0.995529, exp_rate<=3 : 0.998230, loss: 0.001284, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17734 s, avg_samples: 56.0, ips: 315.77177 samples/s, eta: 16:43:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:21:06] ppocr INFO: epoch: [386/500], global_step: 234000, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.941884, exp_rate<=1 : 0.978221, exp_rate<=2 : 0.995595, exp_rate<=3 : 0.998229, loss: 0.001728, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.25280 samples/s, eta: 16:43:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:21:24] ppocr INFO: epoch: [386/500], global_step: 234100, lr: 0.000100, edit distance: 0.001405, exp_rate: 0.941921, exp_rate<=1 : 0.978449, exp_rate<=2 : 0.995685, exp_rate<=3 : 0.998321, loss: 0.001453, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17631 s, avg_samples: 56.0, ips: 317.62084 samples/s, eta: 16:43:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:21:41] ppocr INFO: epoch: [386/500], global_step: 234200, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.941467, exp_rate<=1 : 0.978080, exp_rate<=2 : 0.995615, exp_rate<=3 : 0.998290, loss: 0.001277, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17199 s, avg_samples: 56.0, ips: 325.59986 samples/s, eta: 16:42:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:22:00] ppocr INFO: epoch: [386/500], global_step: 234300, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.941397, exp_rate<=1 : 0.978005, exp_rate<=2 : 0.995626, exp_rate<=3 : 0.998288, loss: 0.001583, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18838 s, avg_samples: 56.0, ips: 297.27271 samples/s, eta: 16:42:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:22:20] ppocr INFO: epoch: [386/500], global_step: 234400, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.939975, exp_rate<=1 : 0.977359, exp_rate<=2 : 0.995452, exp_rate<=3 : 0.998203, loss: 0.001801, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20646 s, avg_samples: 56.0, ips: 271.24189 samples/s, eta: 16:42:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:22:41] ppocr INFO: epoch: [386/500], global_step: 234500, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.939382, exp_rate<=1 : 0.976872, exp_rate<=2 : 0.995300, exp_rate<=3 : 0.998159, loss: 0.001380, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20856 s, avg_samples: 56.0, ips: 268.50423 samples/s, eta: 16:41:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:23:00] ppocr INFO: epoch: [386/500], global_step: 234600, lr: 0.000100, edit distance: 0.001422, exp_rate: 0.938762, exp_rate<=1 : 0.976775, exp_rate<=2 : 0.995229, exp_rate<=3 : 0.998187, loss: 0.001530, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18936 s, avg_samples: 56.0, ips: 295.73258 samples/s, eta: 16:41:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:23:20] ppocr INFO: epoch: [386/500], global_step: 234700, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.938295, exp_rate<=1 : 0.976541, exp_rate<=2 : 0.995060, exp_rate<=3 : 0.998153, loss: 0.001471, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19786 s, avg_samples: 56.0, ips: 283.03129 samples/s, eta: 16:41:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:23:39] ppocr INFO: epoch: [386/500], global_step: 234800, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.938316, exp_rate<=1 : 0.976531, exp_rate<=2 : 0.995058, exp_rate<=3 : 0.998179, loss: 0.001381, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19118 s, avg_samples: 56.0, ips: 292.91303 samples/s, eta: 16:40:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:23:58] ppocr INFO: epoch: [386/500], global_step: 234900, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.938009, exp_rate<=1 : 0.976513, exp_rate<=2 : 0.995069, exp_rate<=3 : 0.998173, loss: 0.001751, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18807 s, avg_samples: 56.0, ips: 297.76387 samples/s, eta: 16:40:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:24:16] ppocr INFO: epoch: [386/500], global_step: 235000, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.937959, exp_rate<=1 : 0.976490, exp_rate<=2 : 0.995011, exp_rate<=3 : 0.998172, loss: 0.001559, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17889 s, avg_samples: 56.0, ips: 313.04608 samples/s, eta: 16:40:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:24:34] ppocr INFO: epoch: [386/500], global_step: 235100, lr: 0.000100, edit distance: 0.001443, exp_rate: 0.937828, exp_rate<=1 : 0.976387, exp_rate<=2 : 0.994973, exp_rate<=3 : 0.998169, loss: 0.001733, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18462 s, avg_samples: 56.0, ips: 303.31919 samples/s, eta: 16:39:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:24:54] ppocr INFO: epoch: [386/500], global_step: 235200, lr: 0.000100, edit distance: 0.001446, exp_rate: 0.937438, exp_rate<=1 : 0.976085, exp_rate<=2 : 0.994837, exp_rate<=3 : 0.998094, loss: 0.001695, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19609 s, avg_samples: 56.0, ips: 285.57953 samples/s, eta: 16:39:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:25:13] ppocr INFO: epoch: [386/500], global_step: 235300, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.937567, exp_rate<=1 : 0.976118, exp_rate<=2 : 0.994863, exp_rate<=3 : 0.998145, loss: 0.001428, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18919 s, avg_samples: 56.0, ips: 296.00488 samples/s, eta: 16:39:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:25:33] ppocr INFO: epoch: [386/500], global_step: 235400, lr: 0.000100, edit distance: 0.001435, exp_rate: 0.937366, exp_rate<=1 : 0.976049, exp_rate<=2 : 0.994790, exp_rate<=3 : 0.998122, loss: 0.001638, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19946 s, avg_samples: 56.0, ips: 280.75669 samples/s, eta: 16:38:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:25:51] ppocr INFO: epoch: [386/500], global_step: 235500, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.937381, exp_rate<=1 : 0.976049, exp_rate<=2 : 0.994844, exp_rate<=3 : 0.998153, loss: 0.001637, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18400 s, avg_samples: 56.0, ips: 304.34209 samples/s, eta: 16:38:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:26:11] ppocr INFO: epoch: [386/500], global_step: 235600, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.936796, exp_rate<=1 : 0.975789, exp_rate<=2 : 0.994867, exp_rate<=3 : 0.998166, loss: 0.001664, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19960 s, avg_samples: 56.0, ips: 280.56354 samples/s, eta: 16:38:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:26:19] ppocr INFO: epoch: [386/500], global_step: 235640, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.936691, exp_rate<=1 : 0.975725, exp_rate<=2 : 0.994870, exp_rate<=3 : 0.998159, loss: 0.001384, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07371 s, avg_samples: 22.4, ips: 303.88720 samples/s, eta: 16:38:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:26:21] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 15:26:32] ppocr INFO: epoch: [387/500], global_step: 235700, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.947103, exp_rate<=1 : 0.978842, exp_rate<=2 : 0.994520, exp_rate<=3 : 0.998557, loss: 0.001407, avg_reader_cost: 0.02894 s, avg_batch_cost: 0.13642 s, avg_samples: 33.6, ips: 246.29858 samples/s, eta: 16:37:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:26:53] ppocr INFO: epoch: [387/500], global_step: 235800, lr: 0.000100, edit distance: 0.001060, exp_rate: 0.942762, exp_rate<=1 : 0.978266, exp_rate<=2 : 0.995108, exp_rate<=3 : 0.998220, loss: 0.001278, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20219 s, avg_samples: 56.0, ips: 276.97102 samples/s, eta: 16:37:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:27:11] ppocr INFO: epoch: [387/500], global_step: 235900, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.941965, exp_rate<=1 : 0.977273, exp_rate<=2 : 0.994792, exp_rate<=3 : 0.998075, loss: 0.001281, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18042 s, avg_samples: 56.0, ips: 310.39217 samples/s, eta: 16:37:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:27:30] ppocr INFO: epoch: [387/500], global_step: 236000, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.939092, exp_rate<=1 : 0.975654, exp_rate<=2 : 0.994448, exp_rate<=3 : 0.998115, loss: 0.001731, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19107 s, avg_samples: 56.0, ips: 293.07992 samples/s, eta: 16:36:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:27:49] ppocr INFO: epoch: [387/500], global_step: 236100, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.939325, exp_rate<=1 : 0.976509, exp_rate<=2 : 0.994649, exp_rate<=3 : 0.998097, loss: 0.000877, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19077 s, avg_samples: 56.0, ips: 293.54300 samples/s, eta: 16:36:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:28:07] ppocr INFO: epoch: [387/500], global_step: 236200, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.939340, exp_rate<=1 : 0.976570, exp_rate<=2 : 0.994788, exp_rate<=3 : 0.998156, loss: 0.001355, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18285 s, avg_samples: 56.0, ips: 306.26305 samples/s, eta: 16:36:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:28:26] ppocr INFO: epoch: [387/500], global_step: 236300, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.938748, exp_rate<=1 : 0.975926, exp_rate<=2 : 0.994647, exp_rate<=3 : 0.998078, loss: 0.001125, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19010 s, avg_samples: 56.0, ips: 294.57515 samples/s, eta: 16:35:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:28:45] ppocr INFO: epoch: [387/500], global_step: 236400, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.939612, exp_rate<=1 : 0.976456, exp_rate<=2 : 0.994815, exp_rate<=3 : 0.998130, loss: 0.001990, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18594 s, avg_samples: 56.0, ips: 301.16711 samples/s, eta: 16:35:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:29:03] ppocr INFO: epoch: [387/500], global_step: 236500, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.940563, exp_rate<=1 : 0.977020, exp_rate<=2 : 0.995092, exp_rate<=3 : 0.998256, loss: 0.001706, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17819 s, avg_samples: 56.0, ips: 314.26416 samples/s, eta: 16:35:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:29:21] ppocr INFO: epoch: [387/500], global_step: 236600, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.940329, exp_rate<=1 : 0.976794, exp_rate<=2 : 0.995026, exp_rate<=3 : 0.998253, loss: 0.001602, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18729 s, avg_samples: 56.0, ips: 299.00191 samples/s, eta: 16:34:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:29:39] ppocr INFO: epoch: [387/500], global_step: 236700, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.940135, exp_rate<=1 : 0.976651, exp_rate<=2 : 0.994985, exp_rate<=3 : 0.998290, loss: 0.001373, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18074 s, avg_samples: 56.0, ips: 309.82945 samples/s, eta: 16:34:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:29:58] ppocr INFO: epoch: [387/500], global_step: 236800, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.940274, exp_rate<=1 : 0.976501, exp_rate<=2 : 0.994945, exp_rate<=3 : 0.998260, loss: 0.001495, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18394 s, avg_samples: 56.0, ips: 304.45510 samples/s, eta: 16:34:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:30:18] ppocr INFO: epoch: [387/500], global_step: 236900, lr: 0.000100, edit distance: 0.001270, exp_rate: 0.940067, exp_rate<=1 : 0.976429, exp_rate<=2 : 0.994972, exp_rate<=3 : 0.998315, loss: 0.001907, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19872 s, avg_samples: 56.0, ips: 281.80870 samples/s, eta: 16:33:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:30:37] ppocr INFO: epoch: [387/500], global_step: 237000, lr: 0.000100, edit distance: 0.001277, exp_rate: 0.939651, exp_rate<=1 : 0.976261, exp_rate<=2 : 0.994997, exp_rate<=3 : 0.998353, loss: 0.001311, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19499 s, avg_samples: 56.0, ips: 287.19280 samples/s, eta: 16:33:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:30:56] ppocr INFO: epoch: [387/500], global_step: 237100, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.940002, exp_rate<=1 : 0.976459, exp_rate<=2 : 0.994978, exp_rate<=3 : 0.998337, loss: 0.001086, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18803 s, avg_samples: 56.0, ips: 297.81932 samples/s, eta: 16:33:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:31:14] ppocr INFO: epoch: [387/500], global_step: 237200, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.939610, exp_rate<=1 : 0.976302, exp_rate<=2 : 0.994893, exp_rate<=3 : 0.998368, loss: 0.001479, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17774 s, avg_samples: 56.0, ips: 315.06609 samples/s, eta: 16:32:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:31:34] ppocr INFO: epoch: [387/500], global_step: 237300, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.938853, exp_rate<=1 : 0.976026, exp_rate<=2 : 0.994769, exp_rate<=3 : 0.998338, loss: 0.002396, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20163 s, avg_samples: 56.0, ips: 277.73030 samples/s, eta: 16:32:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:31:52] ppocr INFO: epoch: [387/500], global_step: 237400, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.938813, exp_rate<=1 : 0.976046, exp_rate<=2 : 0.994787, exp_rate<=3 : 0.998358, loss: 0.001586, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17835 s, avg_samples: 56.0, ips: 313.98285 samples/s, eta: 16:32:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:32:12] ppocr INFO: epoch: [387/500], global_step: 237500, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.938456, exp_rate<=1 : 0.975898, exp_rate<=2 : 0.994767, exp_rate<=3 : 0.998357, loss: 0.001444, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19731 s, avg_samples: 56.0, ips: 283.81307 samples/s, eta: 16:32:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:32:30] ppocr INFO: epoch: [387/500], global_step: 237600, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.938556, exp_rate<=1 : 0.975922, exp_rate<=2 : 0.994787, exp_rate<=3 : 0.998339, loss: 0.001919, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18035 s, avg_samples: 56.0, ips: 310.50835 samples/s, eta: 16:31:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:32:49] ppocr INFO: epoch: [387/500], global_step: 237700, lr: 0.000100, edit distance: 0.001343, exp_rate: 0.938049, exp_rate<=1 : 0.975880, exp_rate<=2 : 0.994784, exp_rate<=3 : 0.998348, loss: 0.001798, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19149 s, avg_samples: 56.0, ips: 292.44622 samples/s, eta: 16:31:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:33:08] ppocr INFO: epoch: [387/500], global_step: 237800, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.937063, exp_rate<=1 : 0.975496, exp_rate<=2 : 0.994669, exp_rate<=3 : 0.998330, loss: 0.002124, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19331 s, avg_samples: 56.0, ips: 289.68587 samples/s, eta: 16:31:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:33:26] ppocr INFO: epoch: [387/500], global_step: 237900, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.936850, exp_rate<=1 : 0.975521, exp_rate<=2 : 0.994608, exp_rate<=3 : 0.998329, loss: 0.001611, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18022 s, avg_samples: 56.0, ips: 310.73263 samples/s, eta: 16:30:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:33:45] ppocr INFO: epoch: [387/500], global_step: 238000, lr: 0.000100, edit distance: 0.001405, exp_rate: 0.936248, exp_rate<=1 : 0.975245, exp_rate<=2 : 0.994490, exp_rate<=3 : 0.998275, loss: 0.002595, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18191 s, avg_samples: 56.0, ips: 307.84298 samples/s, eta: 16:30:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:34:03] ppocr INFO: epoch: [387/500], global_step: 238100, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.936008, exp_rate<=1 : 0.975179, exp_rate<=2 : 0.994437, exp_rate<=3 : 0.998207, loss: 0.001587, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18108 s, avg_samples: 56.0, ips: 309.25953 samples/s, eta: 16:30:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:34:23] ppocr INFO: epoch: [387/500], global_step: 238200, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.935386, exp_rate<=1 : 0.974955, exp_rate<=2 : 0.994394, exp_rate<=3 : 0.998166, loss: 0.001687, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19903 s, avg_samples: 56.0, ips: 281.37119 samples/s, eta: 16:29:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:34:41] ppocr INFO: epoch: [387/500], global_step: 238300, lr: 0.000100, edit distance: 0.001426, exp_rate: 0.935553, exp_rate<=1 : 0.975132, exp_rate<=2 : 0.994427, exp_rate<=3 : 0.998162, loss: 0.001529, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18360 s, avg_samples: 56.0, ips: 305.01896 samples/s, eta: 16:29:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:34:55] ppocr INFO: epoch: [387/500], global_step: 238380, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.935741, exp_rate<=1 : 0.975220, exp_rate<=2 : 0.994444, exp_rate<=3 : 0.998187, loss: 0.001713, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14334 s, avg_samples: 44.8, ips: 312.54173 samples/s, eta: 16:29:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:34:57] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 15:35:01] ppocr INFO: epoch: [388/500], global_step: 238400, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.956976, exp_rate<=1 : 0.985254, exp_rate<=2 : 0.996207, exp_rate<=3 : 0.997534, loss: 0.000702, avg_reader_cost: 0.02456 s, avg_batch_cost: 0.05868 s, avg_samples: 11.2, ips: 190.86887 samples/s, eta: 16:29:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:35:21] ppocr INFO: epoch: [388/500], global_step: 238500, lr: 0.000100, edit distance: 0.001027, exp_rate: 0.950913, exp_rate<=1 : 0.979768, exp_rate<=2 : 0.995495, exp_rate<=3 : 0.998061, loss: 0.000950, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19910 s, avg_samples: 56.0, ips: 281.26726 samples/s, eta: 16:28:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:35:39] ppocr INFO: epoch: [388/500], global_step: 238600, lr: 0.000100, edit distance: 0.001037, exp_rate: 0.950738, exp_rate<=1 : 0.981627, exp_rate<=2 : 0.996371, exp_rate<=3 : 0.998433, loss: 0.000850, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17934 s, avg_samples: 56.0, ips: 312.25824 samples/s, eta: 16:28:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:35:57] ppocr INFO: epoch: [388/500], global_step: 238700, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.949707, exp_rate<=1 : 0.981629, exp_rate<=2 : 0.996319, exp_rate<=3 : 0.998343, loss: 0.001656, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17693 s, avg_samples: 56.0, ips: 316.51085 samples/s, eta: 16:28:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:36:15] ppocr INFO: epoch: [388/500], global_step: 238800, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.949869, exp_rate<=1 : 0.981399, exp_rate<=2 : 0.995933, exp_rate<=3 : 0.998303, loss: 0.001268, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17843 s, avg_samples: 56.0, ips: 313.83992 samples/s, eta: 16:27:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:36:34] ppocr INFO: epoch: [388/500], global_step: 238900, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.947790, exp_rate<=1 : 0.979944, exp_rate<=2 : 0.995514, exp_rate<=3 : 0.998088, loss: 0.001636, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18894 s, avg_samples: 56.0, ips: 296.39648 samples/s, eta: 16:27:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:36:53] ppocr INFO: epoch: [388/500], global_step: 239000, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.946065, exp_rate<=1 : 0.979123, exp_rate<=2 : 0.995378, exp_rate<=3 : 0.997923, loss: 0.001567, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18962 s, avg_samples: 56.0, ips: 295.33426 samples/s, eta: 16:27:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:37:11] ppocr INFO: epoch: [388/500], global_step: 239100, lr: 0.000100, edit distance: 0.001177, exp_rate: 0.945333, exp_rate<=1 : 0.978698, exp_rate<=2 : 0.995226, exp_rate<=3 : 0.997920, loss: 0.001703, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18664 s, avg_samples: 56.0, ips: 300.04178 samples/s, eta: 16:26:46, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:37:30] ppocr INFO: epoch: [388/500], global_step: 239200, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.944975, exp_rate<=1 : 0.978721, exp_rate<=2 : 0.995206, exp_rate<=3 : 0.997957, loss: 0.001849, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18478 s, avg_samples: 56.0, ips: 303.05961 samples/s, eta: 16:26:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:37:49] ppocr INFO: epoch: [388/500], global_step: 239300, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.943573, exp_rate<=1 : 0.978017, exp_rate<=2 : 0.994993, exp_rate<=3 : 0.997889, loss: 0.001790, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19691 s, avg_samples: 56.0, ips: 284.38688 samples/s, eta: 16:26:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:38:08] ppocr INFO: epoch: [388/500], global_step: 239400, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942156, exp_rate<=1 : 0.977441, exp_rate<=2 : 0.994937, exp_rate<=3 : 0.997915, loss: 0.001991, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18365 s, avg_samples: 56.0, ips: 304.92475 samples/s, eta: 16:25:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:38:28] ppocr INFO: epoch: [388/500], global_step: 239500, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.941063, exp_rate<=1 : 0.976941, exp_rate<=2 : 0.994745, exp_rate<=3 : 0.997872, loss: 0.001489, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19851 s, avg_samples: 56.0, ips: 282.09809 samples/s, eta: 16:25:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:38:47] ppocr INFO: epoch: [388/500], global_step: 239600, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.940442, exp_rate<=1 : 0.976815, exp_rate<=2 : 0.994616, exp_rate<=3 : 0.997868, loss: 0.001439, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19433 s, avg_samples: 56.0, ips: 288.17176 samples/s, eta: 16:25:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:39:04] ppocr INFO: epoch: [388/500], global_step: 239700, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.940474, exp_rate<=1 : 0.976759, exp_rate<=2 : 0.994695, exp_rate<=3 : 0.997924, loss: 0.001295, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17234 s, avg_samples: 56.0, ips: 324.93476 samples/s, eta: 16:24:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:39:23] ppocr INFO: epoch: [388/500], global_step: 239800, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.940952, exp_rate<=1 : 0.977063, exp_rate<=2 : 0.994716, exp_rate<=3 : 0.997904, loss: 0.001341, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18795 s, avg_samples: 56.0, ips: 297.94550 samples/s, eta: 16:24:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:39:43] ppocr INFO: epoch: [388/500], global_step: 239900, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.940602, exp_rate<=1 : 0.976904, exp_rate<=2 : 0.994743, exp_rate<=3 : 0.997941, loss: 0.001521, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20058 s, avg_samples: 56.0, ips: 279.18624 samples/s, eta: 16:24:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:40:01] ppocr INFO: epoch: [388/500], global_step: 240000, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.940511, exp_rate<=1 : 0.977090, exp_rate<=2 : 0.994794, exp_rate<=3 : 0.997972, loss: 0.001521, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17949 s, avg_samples: 56.0, ips: 311.98777 samples/s, eta: 16:23:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:40:20] ppocr INFO: epoch: [388/500], global_step: 240100, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.940495, exp_rate<=1 : 0.977022, exp_rate<=2 : 0.994761, exp_rate<=3 : 0.998001, loss: 0.002132, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18665 s, avg_samples: 56.0, ips: 300.03051 samples/s, eta: 16:23:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:40:39] ppocr INFO: epoch: [388/500], global_step: 240200, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.939715, exp_rate<=1 : 0.976635, exp_rate<=2 : 0.994679, exp_rate<=3 : 0.998037, loss: 0.001612, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19382 s, avg_samples: 56.0, ips: 288.92064 samples/s, eta: 16:23:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:40:58] ppocr INFO: epoch: [388/500], global_step: 240300, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.939245, exp_rate<=1 : 0.976462, exp_rate<=2 : 0.994695, exp_rate<=3 : 0.998051, loss: 0.001560, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18975 s, avg_samples: 56.0, ips: 295.12150 samples/s, eta: 16:22:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:41:18] ppocr INFO: epoch: [388/500], global_step: 240400, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.939108, exp_rate<=1 : 0.976444, exp_rate<=2 : 0.994620, exp_rate<=3 : 0.998037, loss: 0.001427, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19244 s, avg_samples: 56.0, ips: 290.99239 samples/s, eta: 16:22:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:41:37] ppocr INFO: epoch: [388/500], global_step: 240500, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.938969, exp_rate<=1 : 0.976337, exp_rate<=2 : 0.994565, exp_rate<=3 : 0.997963, loss: 0.001461, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19039 s, avg_samples: 56.0, ips: 294.13628 samples/s, eta: 16:22:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:41:55] ppocr INFO: epoch: [388/500], global_step: 240600, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.939284, exp_rate<=1 : 0.976458, exp_rate<=2 : 0.994610, exp_rate<=3 : 0.997988, loss: 0.001804, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18168 s, avg_samples: 56.0, ips: 308.23431 samples/s, eta: 16:21:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:42:13] ppocr INFO: epoch: [388/500], global_step: 240700, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.939089, exp_rate<=1 : 0.976535, exp_rate<=2 : 0.994702, exp_rate<=3 : 0.998006, loss: 0.001330, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18082 s, avg_samples: 56.0, ips: 309.69297 samples/s, eta: 16:21:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:42:31] ppocr INFO: epoch: [388/500], global_step: 240800, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.939113, exp_rate<=1 : 0.976638, exp_rate<=2 : 0.994739, exp_rate<=3 : 0.998036, loss: 0.001121, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17952 s, avg_samples: 56.0, ips: 311.93536 samples/s, eta: 16:21:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:42:50] ppocr INFO: epoch: [388/500], global_step: 240900, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.938790, exp_rate<=1 : 0.976407, exp_rate<=2 : 0.994703, exp_rate<=3 : 0.998003, loss: 0.001647, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18585 s, avg_samples: 56.0, ips: 301.32143 samples/s, eta: 16:20:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:43:08] ppocr INFO: epoch: [388/500], global_step: 241000, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.938392, exp_rate<=1 : 0.976326, exp_rate<=2 : 0.994768, exp_rate<=3 : 0.998021, loss: 0.002158, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18518 s, avg_samples: 56.0, ips: 302.41167 samples/s, eta: 16:20:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:43:27] ppocr INFO: epoch: [388/500], global_step: 241100, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.938061, exp_rate<=1 : 0.976117, exp_rate<=2 : 0.994720, exp_rate<=3 : 0.997999, loss: 0.001846, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18855 s, avg_samples: 56.0, ips: 296.99572 samples/s, eta: 16:20:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:43:32] ppocr INFO: epoch: [388/500], global_step: 241120, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.937958, exp_rate<=1 : 0.976089, exp_rate<=2 : 0.994713, exp_rate<=3 : 0.998005, loss: 0.002160, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.04575 s, avg_samples: 11.2, ips: 244.79170 samples/s, eta: 16:20:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:43:34] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 15:43:49] ppocr INFO: epoch: [389/500], global_step: 241200, lr: 0.000100, edit distance: 0.001626, exp_rate: 0.942091, exp_rate<=1 : 0.979187, exp_rate<=2 : 0.995599, exp_rate<=3 : 0.998848, loss: 0.001658, avg_reader_cost: 0.02709 s, avg_batch_cost: 0.17339 s, avg_samples: 44.8, ips: 258.37754 samples/s, eta: 16:19:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:44:08] ppocr INFO: epoch: [389/500], global_step: 241300, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.943435, exp_rate<=1 : 0.979086, exp_rate<=2 : 0.995344, exp_rate<=3 : 0.998904, loss: 0.002419, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18873 s, avg_samples: 56.0, ips: 296.72283 samples/s, eta: 16:19:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:44:28] ppocr INFO: epoch: [389/500], global_step: 241400, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.941825, exp_rate<=1 : 0.978178, exp_rate<=2 : 0.995395, exp_rate<=3 : 0.998898, loss: 0.001038, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20135 s, avg_samples: 56.0, ips: 278.12044 samples/s, eta: 16:19:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:44:47] ppocr INFO: epoch: [389/500], global_step: 241500, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.942098, exp_rate<=1 : 0.978282, exp_rate<=2 : 0.995163, exp_rate<=3 : 0.998492, loss: 0.001800, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19319 s, avg_samples: 56.0, ips: 289.87299 samples/s, eta: 16:19:00, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:45:06] ppocr INFO: epoch: [389/500], global_step: 241600, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.942280, exp_rate<=1 : 0.977953, exp_rate<=2 : 0.995345, exp_rate<=3 : 0.998510, loss: 0.001546, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19181 s, avg_samples: 56.0, ips: 291.96254 samples/s, eta: 16:18:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:45:25] ppocr INFO: epoch: [389/500], global_step: 241700, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.942460, exp_rate<=1 : 0.978480, exp_rate<=2 : 0.995445, exp_rate<=3 : 0.998571, loss: 0.002074, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18492 s, avg_samples: 56.0, ips: 302.82596 samples/s, eta: 16:18:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:45:45] ppocr INFO: epoch: [389/500], global_step: 241800, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.941192, exp_rate<=1 : 0.977712, exp_rate<=2 : 0.995019, exp_rate<=3 : 0.998400, loss: 0.001991, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20237 s, avg_samples: 56.0, ips: 276.72768 samples/s, eta: 16:18:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:46:04] ppocr INFO: epoch: [389/500], global_step: 241900, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.939321, exp_rate<=1 : 0.977130, exp_rate<=2 : 0.994543, exp_rate<=3 : 0.998266, loss: 0.001916, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18662 s, avg_samples: 56.0, ips: 300.07521 samples/s, eta: 16:17:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:46:24] ppocr INFO: epoch: [389/500], global_step: 242000, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.938428, exp_rate<=1 : 0.976792, exp_rate<=2 : 0.994307, exp_rate<=3 : 0.998200, loss: 0.001370, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19622 s, avg_samples: 56.0, ips: 285.39841 samples/s, eta: 16:17:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:46:42] ppocr INFO: epoch: [389/500], global_step: 242100, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.938692, exp_rate<=1 : 0.976593, exp_rate<=2 : 0.994331, exp_rate<=3 : 0.998120, loss: 0.001577, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18035 s, avg_samples: 56.0, ips: 310.50952 samples/s, eta: 16:17:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:47:01] ppocr INFO: epoch: [389/500], global_step: 242200, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.939242, exp_rate<=1 : 0.976652, exp_rate<=2 : 0.994491, exp_rate<=3 : 0.998175, loss: 0.002130, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18931 s, avg_samples: 56.0, ips: 295.80549 samples/s, eta: 16:16:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:47:19] ppocr INFO: epoch: [389/500], global_step: 242300, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.938228, exp_rate<=1 : 0.976368, exp_rate<=2 : 0.994502, exp_rate<=3 : 0.998143, loss: 0.001303, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18079 s, avg_samples: 56.0, ips: 309.74774 samples/s, eta: 16:16:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:47:38] ppocr INFO: epoch: [389/500], global_step: 242400, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.937010, exp_rate<=1 : 0.975908, exp_rate<=2 : 0.994422, exp_rate<=3 : 0.998066, loss: 0.002945, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19356 s, avg_samples: 56.0, ips: 289.32215 samples/s, eta: 16:16:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:47:55] ppocr INFO: epoch: [389/500], global_step: 242500, lr: 0.000100, edit distance: 0.001367, exp_rate: 0.937075, exp_rate<=1 : 0.976030, exp_rate<=2 : 0.994550, exp_rate<=3 : 0.998134, loss: 0.001539, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17045 s, avg_samples: 56.0, ips: 328.55056 samples/s, eta: 16:15:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:48:12] ppocr INFO: epoch: [389/500], global_step: 242600, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.936815, exp_rate<=1 : 0.975903, exp_rate<=2 : 0.994609, exp_rate<=3 : 0.998183, loss: 0.001962, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16935 s, avg_samples: 56.0, ips: 330.67707 samples/s, eta: 16:15:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:48:31] ppocr INFO: epoch: [389/500], global_step: 242700, lr: 0.000100, edit distance: 0.001402, exp_rate: 0.936394, exp_rate<=1 : 0.975846, exp_rate<=2 : 0.994641, exp_rate<=3 : 0.998213, loss: 0.001778, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19276 s, avg_samples: 56.0, ips: 290.51011 samples/s, eta: 16:15:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:48:49] ppocr INFO: epoch: [389/500], global_step: 242800, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.936143, exp_rate<=1 : 0.975691, exp_rate<=2 : 0.994572, exp_rate<=3 : 0.998183, loss: 0.001032, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18098 s, avg_samples: 56.0, ips: 309.43131 samples/s, eta: 16:14:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:49:07] ppocr INFO: epoch: [389/500], global_step: 242900, lr: 0.000100, edit distance: 0.001400, exp_rate: 0.936294, exp_rate<=1 : 0.975756, exp_rate<=2 : 0.994557, exp_rate<=3 : 0.998173, loss: 0.001840, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17894 s, avg_samples: 56.0, ips: 312.94750 samples/s, eta: 16:14:21, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:49:27] ppocr INFO: epoch: [389/500], global_step: 243000, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.935096, exp_rate<=1 : 0.975262, exp_rate<=2 : 0.994369, exp_rate<=3 : 0.998165, loss: 0.001837, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19578 s, avg_samples: 56.0, ips: 286.03447 samples/s, eta: 16:14:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:49:46] ppocr INFO: epoch: [389/500], global_step: 243100, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.934595, exp_rate<=1 : 0.974950, exp_rate<=2 : 0.994226, exp_rate<=3 : 0.998085, loss: 0.001549, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19115 s, avg_samples: 56.0, ips: 292.96816 samples/s, eta: 16:13:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:50:04] ppocr INFO: epoch: [389/500], global_step: 243200, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.934416, exp_rate<=1 : 0.974703, exp_rate<=2 : 0.994177, exp_rate<=3 : 0.998070, loss: 0.001539, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18222 s, avg_samples: 56.0, ips: 307.32861 samples/s, eta: 16:13:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:50:23] ppocr INFO: epoch: [389/500], global_step: 243300, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.934470, exp_rate<=1 : 0.974688, exp_rate<=2 : 0.994241, exp_rate<=3 : 0.998085, loss: 0.001235, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18506 s, avg_samples: 56.0, ips: 302.61197 samples/s, eta: 16:13:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:50:42] ppocr INFO: epoch: [389/500], global_step: 243400, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.934632, exp_rate<=1 : 0.974694, exp_rate<=2 : 0.994244, exp_rate<=3 : 0.998151, loss: 0.001554, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18815 s, avg_samples: 56.0, ips: 297.63303 samples/s, eta: 16:12:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:51:01] ppocr INFO: epoch: [389/500], global_step: 243500, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.934369, exp_rate<=1 : 0.974604, exp_rate<=2 : 0.994202, exp_rate<=3 : 0.998140, loss: 0.001161, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19560 s, avg_samples: 56.0, ips: 286.30068 samples/s, eta: 16:12:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:51:21] ppocr INFO: epoch: [389/500], global_step: 243600, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.934283, exp_rate<=1 : 0.974544, exp_rate<=2 : 0.994128, exp_rate<=3 : 0.998129, loss: 0.001648, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19256 s, avg_samples: 56.0, ips: 290.81963 samples/s, eta: 16:12:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:51:38] ppocr INFO: epoch: [389/500], global_step: 243700, lr: 0.000100, edit distance: 0.001428, exp_rate: 0.934681, exp_rate<=1 : 0.974699, exp_rate<=2 : 0.994176, exp_rate<=3 : 0.998144, loss: 0.001711, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17676 s, avg_samples: 56.0, ips: 316.82092 samples/s, eta: 16:11:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:51:56] ppocr INFO: epoch: [389/500], global_step: 243800, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.934808, exp_rate<=1 : 0.974775, exp_rate<=2 : 0.994177, exp_rate<=3 : 0.998111, loss: 0.001330, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18050 s, avg_samples: 56.0, ips: 310.25259 samples/s, eta: 16:11:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:52:07] ppocr INFO: epoch: [389/500], global_step: 243860, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.935143, exp_rate<=1 : 0.974906, exp_rate<=2 : 0.994207, exp_rate<=3 : 0.998140, loss: 0.001521, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10947 s, avg_samples: 33.6, ips: 306.93252 samples/s, eta: 16:11:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:52:09] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 15:52:17] ppocr INFO: epoch: [390/500], global_step: 243900, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.942025, exp_rate<=1 : 0.980737, exp_rate<=2 : 0.995287, exp_rate<=3 : 0.999096, loss: 0.001801, avg_reader_cost: 0.02528 s, avg_batch_cost: 0.10130 s, avg_samples: 22.4, ips: 221.11697 samples/s, eta: 16:11:07, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:52:37] ppocr INFO: epoch: [390/500], global_step: 244000, lr: 0.000100, edit distance: 0.001114, exp_rate: 0.944279, exp_rate<=1 : 0.978087, exp_rate<=2 : 0.995216, exp_rate<=3 : 0.998450, loss: 0.001239, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19538 s, avg_samples: 56.0, ips: 286.61773 samples/s, eta: 16:10:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:52:56] ppocr INFO: epoch: [390/500], global_step: 244100, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.944332, exp_rate<=1 : 0.977592, exp_rate<=2 : 0.995307, exp_rate<=3 : 0.998642, loss: 0.001812, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18659 s, avg_samples: 56.0, ips: 300.12511 samples/s, eta: 16:10:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:53:13] ppocr INFO: epoch: [390/500], global_step: 244200, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.944540, exp_rate<=1 : 0.978316, exp_rate<=2 : 0.995475, exp_rate<=3 : 0.998624, loss: 0.000999, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17528 s, avg_samples: 56.0, ips: 319.48636 samples/s, eta: 16:10:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:53:33] ppocr INFO: epoch: [390/500], global_step: 244300, lr: 0.000100, edit distance: 0.001167, exp_rate: 0.943601, exp_rate<=1 : 0.977720, exp_rate<=2 : 0.994961, exp_rate<=3 : 0.998396, loss: 0.001511, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19405 s, avg_samples: 56.0, ips: 288.58372 samples/s, eta: 16:09:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:53:51] ppocr INFO: epoch: [390/500], global_step: 244400, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.943693, exp_rate<=1 : 0.978234, exp_rate<=2 : 0.995015, exp_rate<=3 : 0.998372, loss: 0.001442, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18398 s, avg_samples: 56.0, ips: 304.38732 samples/s, eta: 16:09:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:54:10] ppocr INFO: epoch: [390/500], global_step: 244500, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.942860, exp_rate<=1 : 0.978095, exp_rate<=2 : 0.994977, exp_rate<=3 : 0.998305, loss: 0.001422, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18761 s, avg_samples: 56.0, ips: 298.48898 samples/s, eta: 16:09:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:54:29] ppocr INFO: epoch: [390/500], global_step: 244600, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.941803, exp_rate<=1 : 0.977462, exp_rate<=2 : 0.994793, exp_rate<=3 : 0.998078, loss: 0.001000, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18908 s, avg_samples: 56.0, ips: 296.17051 samples/s, eta: 16:08:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:54:47] ppocr INFO: epoch: [390/500], global_step: 244700, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.942960, exp_rate<=1 : 0.977929, exp_rate<=2 : 0.995003, exp_rate<=3 : 0.998141, loss: 0.001439, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18154 s, avg_samples: 56.0, ips: 308.46946 samples/s, eta: 16:08:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:55:05] ppocr INFO: epoch: [390/500], global_step: 244800, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.942554, exp_rate<=1 : 0.978171, exp_rate<=2 : 0.995127, exp_rate<=3 : 0.998119, loss: 0.001505, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18405 s, avg_samples: 56.0, ips: 304.26679 samples/s, eta: 16:08:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:55:24] ppocr INFO: epoch: [390/500], global_step: 244900, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.942368, exp_rate<=1 : 0.978056, exp_rate<=2 : 0.994965, exp_rate<=3 : 0.998161, loss: 0.001547, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18571 s, avg_samples: 56.0, ips: 301.54716 samples/s, eta: 16:07:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:55:43] ppocr INFO: epoch: [390/500], global_step: 245000, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.942143, exp_rate<=1 : 0.978177, exp_rate<=2 : 0.994882, exp_rate<=3 : 0.998136, loss: 0.001235, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19298 s, avg_samples: 56.0, ips: 290.18692 samples/s, eta: 16:07:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:56:04] ppocr INFO: epoch: [390/500], global_step: 245100, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.941081, exp_rate<=1 : 0.977613, exp_rate<=2 : 0.994649, exp_rate<=3 : 0.998036, loss: 0.001572, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20237 s, avg_samples: 56.0, ips: 276.71689 samples/s, eta: 16:07:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:56:25] ppocr INFO: epoch: [390/500], global_step: 245200, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.940081, exp_rate<=1 : 0.977039, exp_rate<=2 : 0.994313, exp_rate<=3 : 0.997957, loss: 0.001761, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20940 s, avg_samples: 56.0, ips: 267.42844 samples/s, eta: 16:06:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:56:43] ppocr INFO: epoch: [390/500], global_step: 245300, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.940387, exp_rate<=1 : 0.977190, exp_rate<=2 : 0.994493, exp_rate<=3 : 0.998065, loss: 0.000941, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18483 s, avg_samples: 56.0, ips: 302.97514 samples/s, eta: 16:06:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:57:03] ppocr INFO: epoch: [390/500], global_step: 245400, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.939956, exp_rate<=1 : 0.976928, exp_rate<=2 : 0.994454, exp_rate<=3 : 0.998070, loss: 0.001781, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19776 s, avg_samples: 56.0, ips: 283.16911 samples/s, eta: 16:06:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:57:21] ppocr INFO: epoch: [390/500], global_step: 245500, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.939768, exp_rate<=1 : 0.976785, exp_rate<=2 : 0.994410, exp_rate<=3 : 0.998085, loss: 0.001816, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18445 s, avg_samples: 56.0, ips: 303.59946 samples/s, eta: 16:05:57, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:57:40] ppocr INFO: epoch: [390/500], global_step: 245600, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.939383, exp_rate<=1 : 0.976602, exp_rate<=2 : 0.994389, exp_rate<=3 : 0.998070, loss: 0.001580, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18432 s, avg_samples: 56.0, ips: 303.82523 samples/s, eta: 16:05:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:57:58] ppocr INFO: epoch: [390/500], global_step: 245700, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.939101, exp_rate<=1 : 0.976469, exp_rate<=2 : 0.994449, exp_rate<=3 : 0.998098, loss: 0.001149, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17793 s, avg_samples: 56.0, ips: 314.72481 samples/s, eta: 16:05:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:58:18] ppocr INFO: epoch: [390/500], global_step: 245800, lr: 0.000100, edit distance: 0.001341, exp_rate: 0.938893, exp_rate<=1 : 0.976377, exp_rate<=2 : 0.994416, exp_rate<=3 : 0.998068, loss: 0.001769, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20134 s, avg_samples: 56.0, ips: 278.13377 samples/s, eta: 16:04:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:58:36] ppocr INFO: epoch: [390/500], global_step: 245900, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.938992, exp_rate<=1 : 0.976467, exp_rate<=2 : 0.994464, exp_rate<=3 : 0.998037, loss: 0.001708, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17925 s, avg_samples: 56.0, ips: 312.41818 samples/s, eta: 16:04:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:58:54] ppocr INFO: epoch: [390/500], global_step: 246000, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.939224, exp_rate<=1 : 0.976666, exp_rate<=2 : 0.994556, exp_rate<=3 : 0.998084, loss: 0.002179, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18345 s, avg_samples: 56.0, ips: 305.25715 samples/s, eta: 16:04:17, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:59:13] ppocr INFO: epoch: [390/500], global_step: 246100, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.939034, exp_rate<=1 : 0.976614, exp_rate<=2 : 0.994530, exp_rate<=3 : 0.998077, loss: 0.001281, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18905 s, avg_samples: 56.0, ips: 296.21368 samples/s, eta: 16:03:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:59:31] ppocr INFO: epoch: [390/500], global_step: 246200, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.939033, exp_rate<=1 : 0.976637, exp_rate<=2 : 0.994527, exp_rate<=3 : 0.998058, loss: 0.001823, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18127 s, avg_samples: 56.0, ips: 308.92352 samples/s, eta: 16:03:37, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 15:59:50] ppocr INFO: epoch: [390/500], global_step: 246300, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.938392, exp_rate<=1 : 0.976376, exp_rate<=2 : 0.994509, exp_rate<=3 : 0.998075, loss: 0.001940, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19348 s, avg_samples: 56.0, ips: 289.43488 samples/s, eta: 16:03:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:00:09] ppocr INFO: epoch: [390/500], global_step: 246400, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.938265, exp_rate<=1 : 0.976313, exp_rate<=2 : 0.994537, exp_rate<=3 : 0.998066, loss: 0.001758, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18519 s, avg_samples: 56.0, ips: 302.38816 samples/s, eta: 16:02:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:00:27] ppocr INFO: epoch: [390/500], global_step: 246500, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.938206, exp_rate<=1 : 0.976310, exp_rate<=2 : 0.994533, exp_rate<=3 : 0.998065, loss: 0.002160, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17916 s, avg_samples: 56.0, ips: 312.57835 samples/s, eta: 16:02:38, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:00:44] ppocr INFO: epoch: [390/500], global_step: 246600, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.937771, exp_rate<=1 : 0.976259, exp_rate<=2 : 0.994559, exp_rate<=3 : 0.998090, loss: 0.001936, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17179 s, avg_samples: 56.0, ips: 325.97327 samples/s, eta: 16:02:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:00:46] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 16:00:47] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_390 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 16:01:06] ppocr INFO: epoch: [391/500], global_step: 246700, lr: 0.000100, edit distance: 0.001367, exp_rate: 0.934391, exp_rate<=1 : 0.972474, exp_rate<=2 : 0.994870, exp_rate<=3 : 0.998421, loss: 0.001450, avg_reader_cost: 0.03155 s, avg_batch_cost: 0.22362 s, avg_samples: 56.0, ips: 250.42537 samples/s, eta: 16:02:01, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:01:27] ppocr INFO: epoch: [391/500], global_step: 246800, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.932191, exp_rate<=1 : 0.972249, exp_rate<=2 : 0.993658, exp_rate<=3 : 0.997751, loss: 0.001530, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20554 s, avg_samples: 56.0, ips: 272.45514 samples/s, eta: 16:01:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:01:46] ppocr INFO: epoch: [391/500], global_step: 246900, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.934320, exp_rate<=1 : 0.973245, exp_rate<=2 : 0.993585, exp_rate<=3 : 0.997578, loss: 0.001090, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18840 s, avg_samples: 56.0, ips: 297.23345 samples/s, eta: 16:01:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:02:04] ppocr INFO: epoch: [391/500], global_step: 247000, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.938016, exp_rate<=1 : 0.974761, exp_rate<=2 : 0.993797, exp_rate<=3 : 0.997759, loss: 0.001325, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18305 s, avg_samples: 56.0, ips: 305.93266 samples/s, eta: 16:01:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:02:24] ppocr INFO: epoch: [391/500], global_step: 247100, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.937353, exp_rate<=1 : 0.974843, exp_rate<=2 : 0.993665, exp_rate<=3 : 0.997640, loss: 0.001287, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19710 s, avg_samples: 56.0, ips: 284.11999 samples/s, eta: 16:00:45, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:02:42] ppocr INFO: epoch: [391/500], global_step: 247200, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.940104, exp_rate<=1 : 0.976367, exp_rate<=2 : 0.994043, exp_rate<=3 : 0.997883, loss: 0.000694, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18398 s, avg_samples: 56.0, ips: 304.37759 samples/s, eta: 16:00:25, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:03:00] ppocr INFO: epoch: [391/500], global_step: 247300, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.940883, exp_rate<=1 : 0.976918, exp_rate<=2 : 0.994198, exp_rate<=3 : 0.997993, loss: 0.001429, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18011 s, avg_samples: 56.0, ips: 310.92204 samples/s, eta: 16:00:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:03:18] ppocr INFO: epoch: [391/500], global_step: 247400, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.940960, exp_rate<=1 : 0.977106, exp_rate<=2 : 0.994255, exp_rate<=3 : 0.998035, loss: 0.001095, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17737 s, avg_samples: 56.0, ips: 315.73082 samples/s, eta: 15:59:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:03:37] ppocr INFO: epoch: [391/500], global_step: 247500, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.941666, exp_rate<=1 : 0.977341, exp_rate<=2 : 0.994433, exp_rate<=3 : 0.998033, loss: 0.001719, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18698 s, avg_samples: 56.0, ips: 299.50347 samples/s, eta: 15:59:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:03:55] ppocr INFO: epoch: [391/500], global_step: 247600, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.941379, exp_rate<=1 : 0.977494, exp_rate<=2 : 0.994521, exp_rate<=3 : 0.998033, loss: 0.001549, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18101 s, avg_samples: 56.0, ips: 309.38329 samples/s, eta: 15:59:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:04:13] ppocr INFO: epoch: [391/500], global_step: 247700, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.941032, exp_rate<=1 : 0.977461, exp_rate<=2 : 0.994537, exp_rate<=3 : 0.998067, loss: 0.001566, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18335 s, avg_samples: 56.0, ips: 305.42126 samples/s, eta: 15:58:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:04:32] ppocr INFO: epoch: [391/500], global_step: 247800, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.941134, exp_rate<=1 : 0.977520, exp_rate<=2 : 0.994673, exp_rate<=3 : 0.998140, loss: 0.001748, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18424 s, avg_samples: 56.0, ips: 303.95289 samples/s, eta: 15:58:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:04:51] ppocr INFO: epoch: [391/500], global_step: 247900, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.940455, exp_rate<=1 : 0.977373, exp_rate<=2 : 0.994587, exp_rate<=3 : 0.998110, loss: 0.001728, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18970 s, avg_samples: 56.0, ips: 295.20332 samples/s, eta: 15:58:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:05:09] ppocr INFO: epoch: [391/500], global_step: 248000, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.940107, exp_rate<=1 : 0.977319, exp_rate<=2 : 0.994614, exp_rate<=3 : 0.998156, loss: 0.001319, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18293 s, avg_samples: 56.0, ips: 306.12630 samples/s, eta: 15:57:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:05:27] ppocr INFO: epoch: [391/500], global_step: 248100, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.939616, exp_rate<=1 : 0.977210, exp_rate<=2 : 0.994771, exp_rate<=3 : 0.998191, loss: 0.001583, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18239 s, avg_samples: 56.0, ips: 307.03154 samples/s, eta: 15:57:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:05:45] ppocr INFO: epoch: [391/500], global_step: 248200, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.939600, exp_rate<=1 : 0.977215, exp_rate<=2 : 0.994890, exp_rate<=3 : 0.998246, loss: 0.001287, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17902 s, avg_samples: 56.0, ips: 312.80842 samples/s, eta: 15:57:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:06:03] ppocr INFO: epoch: [391/500], global_step: 248300, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.939640, exp_rate<=1 : 0.977247, exp_rate<=2 : 0.994921, exp_rate<=3 : 0.998253, loss: 0.001615, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18135 s, avg_samples: 56.0, ips: 308.79765 samples/s, eta: 15:56:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:06:22] ppocr INFO: epoch: [391/500], global_step: 248400, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.939196, exp_rate<=1 : 0.977209, exp_rate<=2 : 0.994943, exp_rate<=3 : 0.998258, loss: 0.001315, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18228 s, avg_samples: 56.0, ips: 307.21618 samples/s, eta: 15:56:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:06:40] ppocr INFO: epoch: [391/500], global_step: 248500, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.939288, exp_rate<=1 : 0.977141, exp_rate<=2 : 0.995005, exp_rate<=3 : 0.998279, loss: 0.001744, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18744 s, avg_samples: 56.0, ips: 298.75728 samples/s, eta: 15:56:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:07:00] ppocr INFO: epoch: [391/500], global_step: 248600, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.938507, exp_rate<=1 : 0.976758, exp_rate<=2 : 0.994911, exp_rate<=3 : 0.998269, loss: 0.002053, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19794 s, avg_samples: 56.0, ips: 282.91930 samples/s, eta: 15:55:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:07:20] ppocr INFO: epoch: [391/500], global_step: 248700, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.939038, exp_rate<=1 : 0.976837, exp_rate<=2 : 0.994939, exp_rate<=3 : 0.998256, loss: 0.001428, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19822 s, avg_samples: 56.0, ips: 282.51531 samples/s, eta: 15:55:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:07:38] ppocr INFO: epoch: [391/500], global_step: 248800, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.939137, exp_rate<=1 : 0.976822, exp_rate<=2 : 0.994954, exp_rate<=3 : 0.998264, loss: 0.001598, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18222 s, avg_samples: 56.0, ips: 307.32114 samples/s, eta: 15:55:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:07:58] ppocr INFO: epoch: [391/500], global_step: 248900, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.938474, exp_rate<=1 : 0.976588, exp_rate<=2 : 0.994851, exp_rate<=3 : 0.998205, loss: 0.001390, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19847 s, avg_samples: 56.0, ips: 282.16025 samples/s, eta: 15:54:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:08:17] ppocr INFO: epoch: [391/500], global_step: 249000, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.938042, exp_rate<=1 : 0.976444, exp_rate<=2 : 0.994763, exp_rate<=3 : 0.998153, loss: 0.001810, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18713 s, avg_samples: 56.0, ips: 299.24972 samples/s, eta: 15:54:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:08:36] ppocr INFO: epoch: [391/500], global_step: 249100, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.937582, exp_rate<=1 : 0.976231, exp_rate<=2 : 0.994646, exp_rate<=3 : 0.998125, loss: 0.001658, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18770 s, avg_samples: 56.0, ips: 298.35083 samples/s, eta: 15:54:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:08:55] ppocr INFO: epoch: [391/500], global_step: 249200, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.937471, exp_rate<=1 : 0.976200, exp_rate<=2 : 0.994678, exp_rate<=3 : 0.998135, loss: 0.001767, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18733 s, avg_samples: 56.0, ips: 298.94459 samples/s, eta: 15:53:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:09:13] ppocr INFO: epoch: [391/500], global_step: 249300, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.937149, exp_rate<=1 : 0.976046, exp_rate<=2 : 0.994567, exp_rate<=3 : 0.998102, loss: 0.001510, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18135 s, avg_samples: 56.0, ips: 308.79475 samples/s, eta: 15:53:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:09:20] ppocr INFO: epoch: [391/500], global_step: 249340, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.937151, exp_rate<=1 : 0.976054, exp_rate<=2 : 0.994586, exp_rate<=3 : 0.998120, loss: 0.001525, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07678 s, avg_samples: 22.4, ips: 291.73804 samples/s, eta: 15:53:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:09:23] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 16:09:34] ppocr INFO: epoch: [392/500], global_step: 249400, lr: 0.000100, edit distance: 0.001613, exp_rate: 0.944280, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.996187, exp_rate<=3 : 0.999646, loss: 0.001516, avg_reader_cost: 0.03038 s, avg_batch_cost: 0.13893 s, avg_samples: 33.6, ips: 241.85415 samples/s, eta: 15:53:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:09:53] ppocr INFO: epoch: [392/500], global_step: 249500, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.945717, exp_rate<=1 : 0.980493, exp_rate<=2 : 0.996136, exp_rate<=3 : 0.999175, loss: 0.001092, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18320 s, avg_samples: 56.0, ips: 305.66933 samples/s, eta: 15:52:52, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:10:11] ppocr INFO: epoch: [392/500], global_step: 249600, lr: 0.000100, edit distance: 0.001407, exp_rate: 0.944139, exp_rate<=1 : 0.979325, exp_rate<=2 : 0.995555, exp_rate<=3 : 0.998755, loss: 0.002106, avg_reader_cost: 0.00018 s, avg_batch_cost: 0.18067 s, avg_samples: 56.0, ips: 309.95690 samples/s, eta: 15:52:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:10:29] ppocr INFO: epoch: [392/500], global_step: 249700, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.942821, exp_rate<=1 : 0.979103, exp_rate<=2 : 0.995299, exp_rate<=3 : 0.998624, loss: 0.001721, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17884 s, avg_samples: 56.0, ips: 313.12040 samples/s, eta: 15:52:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:10:47] ppocr INFO: epoch: [392/500], global_step: 249800, lr: 0.000100, edit distance: 0.001397, exp_rate: 0.941483, exp_rate<=1 : 0.978428, exp_rate<=2 : 0.994621, exp_rate<=3 : 0.998446, loss: 0.001588, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18258 s, avg_samples: 56.0, ips: 306.71857 samples/s, eta: 15:51:50, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:11:07] ppocr INFO: epoch: [392/500], global_step: 249900, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.940007, exp_rate<=1 : 0.978160, exp_rate<=2 : 0.994752, exp_rate<=3 : 0.998463, loss: 0.001328, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20134 s, avg_samples: 56.0, ips: 278.13699 samples/s, eta: 15:51:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:11:27] ppocr INFO: epoch: [392/500], global_step: 250000, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.939775, exp_rate<=1 : 0.977716, exp_rate<=2 : 0.994680, exp_rate<=3 : 0.998410, loss: 0.001836, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19490 s, avg_samples: 56.0, ips: 287.32857 samples/s, eta: 15:51:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:11:46] ppocr INFO: epoch: [392/500], global_step: 250100, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.939598, exp_rate<=1 : 0.977346, exp_rate<=2 : 0.994648, exp_rate<=3 : 0.998307, loss: 0.001452, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19345 s, avg_samples: 56.0, ips: 289.48655 samples/s, eta: 15:50:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:12:06] ppocr INFO: epoch: [392/500], global_step: 250200, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.939542, exp_rate<=1 : 0.976879, exp_rate<=2 : 0.994408, exp_rate<=3 : 0.998284, loss: 0.001236, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19863 s, avg_samples: 56.0, ips: 281.93301 samples/s, eta: 15:50:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:12:26] ppocr INFO: epoch: [392/500], global_step: 250300, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.939405, exp_rate<=1 : 0.976940, exp_rate<=2 : 0.994386, exp_rate<=3 : 0.998304, loss: 0.001268, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19894 s, avg_samples: 56.0, ips: 281.49344 samples/s, eta: 15:50:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:12:44] ppocr INFO: epoch: [392/500], global_step: 250400, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939716, exp_rate<=1 : 0.976922, exp_rate<=2 : 0.994393, exp_rate<=3 : 0.998348, loss: 0.001359, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18108 s, avg_samples: 56.0, ips: 309.26395 samples/s, eta: 15:49:58, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:13:03] ppocr INFO: epoch: [392/500], global_step: 250500, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.939967, exp_rate<=1 : 0.976972, exp_rate<=2 : 0.994551, exp_rate<=3 : 0.998351, loss: 0.001733, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19376 s, avg_samples: 56.0, ips: 289.02053 samples/s, eta: 15:49:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:13:21] ppocr INFO: epoch: [392/500], global_step: 250600, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.940410, exp_rate<=1 : 0.977170, exp_rate<=2 : 0.994602, exp_rate<=3 : 0.998315, loss: 0.001565, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17709 s, avg_samples: 56.0, ips: 316.23145 samples/s, eta: 15:49:18, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:13:40] ppocr INFO: epoch: [392/500], global_step: 250700, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.940602, exp_rate<=1 : 0.977146, exp_rate<=2 : 0.994580, exp_rate<=3 : 0.998227, loss: 0.001517, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19486 s, avg_samples: 56.0, ips: 287.38567 samples/s, eta: 15:48:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:13:59] ppocr INFO: epoch: [392/500], global_step: 250800, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.940306, exp_rate<=1 : 0.976752, exp_rate<=2 : 0.994482, exp_rate<=3 : 0.998180, loss: 0.001329, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18590 s, avg_samples: 56.0, ips: 301.24409 samples/s, eta: 15:48:39, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:14:17] ppocr INFO: epoch: [392/500], global_step: 250900, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.940434, exp_rate<=1 : 0.976680, exp_rate<=2 : 0.994563, exp_rate<=3 : 0.998215, loss: 0.001707, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18408 s, avg_samples: 56.0, ips: 304.22236 samples/s, eta: 15:48:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:14:36] ppocr INFO: epoch: [392/500], global_step: 251000, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.940337, exp_rate<=1 : 0.976780, exp_rate<=2 : 0.994568, exp_rate<=3 : 0.998218, loss: 0.002140, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18643 s, avg_samples: 56.0, ips: 300.38730 samples/s, eta: 15:47:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:14:56] ppocr INFO: epoch: [392/500], global_step: 251100, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.939734, exp_rate<=1 : 0.976541, exp_rate<=2 : 0.994491, exp_rate<=3 : 0.998101, loss: 0.001268, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20325 s, avg_samples: 56.0, ips: 275.52460 samples/s, eta: 15:47:42, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:15:16] ppocr INFO: epoch: [392/500], global_step: 251200, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.939703, exp_rate<=1 : 0.976500, exp_rate<=2 : 0.994596, exp_rate<=3 : 0.998157, loss: 0.001658, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19089 s, avg_samples: 56.0, ips: 293.36877 samples/s, eta: 15:47:22, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:15:34] ppocr INFO: epoch: [392/500], global_step: 251300, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.939616, exp_rate<=1 : 0.976512, exp_rate<=2 : 0.994583, exp_rate<=3 : 0.998199, loss: 0.001949, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18347 s, avg_samples: 56.0, ips: 305.23532 samples/s, eta: 15:47:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:15:51] ppocr INFO: epoch: [392/500], global_step: 251400, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.939531, exp_rate<=1 : 0.976585, exp_rate<=2 : 0.994645, exp_rate<=3 : 0.998212, loss: 0.001640, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17201 s, avg_samples: 56.0, ips: 325.55913 samples/s, eta: 15:46:41, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:16:11] ppocr INFO: epoch: [392/500], global_step: 251500, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.939190, exp_rate<=1 : 0.976235, exp_rate<=2 : 0.994479, exp_rate<=3 : 0.998155, loss: 0.001278, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20122 s, avg_samples: 56.0, ips: 278.30487 samples/s, eta: 15:46:23, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:16:29] ppocr INFO: epoch: [392/500], global_step: 251600, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.939389, exp_rate<=1 : 0.976449, exp_rate<=2 : 0.994536, exp_rate<=3 : 0.998199, loss: 0.001873, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17918 s, avg_samples: 56.0, ips: 312.54057 samples/s, eta: 15:46:02, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:16:49] ppocr INFO: epoch: [392/500], global_step: 251700, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.938826, exp_rate<=1 : 0.976180, exp_rate<=2 : 0.994456, exp_rate<=3 : 0.998163, loss: 0.001563, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19216 s, avg_samples: 56.0, ips: 291.43133 samples/s, eta: 15:45:43, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:17:07] ppocr INFO: epoch: [392/500], global_step: 251800, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.938573, exp_rate<=1 : 0.976138, exp_rate<=2 : 0.994397, exp_rate<=3 : 0.998120, loss: 0.001649, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18870 s, avg_samples: 56.0, ips: 296.76888 samples/s, eta: 15:45:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:17:26] ppocr INFO: epoch: [392/500], global_step: 251900, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.938486, exp_rate<=1 : 0.976039, exp_rate<=2 : 0.994359, exp_rate<=3 : 0.998129, loss: 0.001831, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18320 s, avg_samples: 56.0, ips: 305.68005 samples/s, eta: 15:45:04, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:17:45] ppocr INFO: epoch: [392/500], global_step: 252000, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.938357, exp_rate<=1 : 0.976030, exp_rate<=2 : 0.994325, exp_rate<=3 : 0.998079, loss: 0.001610, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18751 s, avg_samples: 56.0, ips: 298.64968 samples/s, eta: 15:44:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:17:58] ppocr INFO: epoch: [392/500], global_step: 252080, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.938065, exp_rate<=1 : 0.975851, exp_rate<=2 : 0.994269, exp_rate<=3 : 0.998045, loss: 0.001485, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.13790 s, avg_samples: 44.8, ips: 324.88165 samples/s, eta: 15:44:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:18:01] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 16:18:05] ppocr INFO: epoch: [393/500], global_step: 252100, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.941815, exp_rate<=1 : 0.980035, exp_rate<=2 : 0.996103, exp_rate<=3 : 1.000000, loss: 0.001415, avg_reader_cost: 0.02851 s, avg_batch_cost: 0.06481 s, avg_samples: 11.2, ips: 172.81987 samples/s, eta: 15:44:26, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:18:24] ppocr INFO: epoch: [393/500], global_step: 252200, lr: 0.000100, edit distance: 0.001308, exp_rate: 0.942954, exp_rate<=1 : 0.979001, exp_rate<=2 : 0.995101, exp_rate<=3 : 0.998869, loss: 0.000815, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18714 s, avg_samples: 56.0, ips: 299.23940 samples/s, eta: 15:44:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:18:43] ppocr INFO: epoch: [393/500], global_step: 252300, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.943146, exp_rate<=1 : 0.977668, exp_rate<=2 : 0.994958, exp_rate<=3 : 0.998721, loss: 0.001610, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19412 s, avg_samples: 56.0, ips: 288.47888 samples/s, eta: 15:43:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:19:03] ppocr INFO: epoch: [393/500], global_step: 252400, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.944129, exp_rate<=1 : 0.977480, exp_rate<=2 : 0.994952, exp_rate<=3 : 0.998853, loss: 0.001841, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19638 s, avg_samples: 56.0, ips: 285.16413 samples/s, eta: 15:43:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:19:20] ppocr INFO: epoch: [393/500], global_step: 252500, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.944817, exp_rate<=1 : 0.978059, exp_rate<=2 : 0.995102, exp_rate<=3 : 0.998745, loss: 0.001434, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17562 s, avg_samples: 56.0, ips: 318.87656 samples/s, eta: 15:43:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:19:39] ppocr INFO: epoch: [393/500], global_step: 252600, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.944051, exp_rate<=1 : 0.977885, exp_rate<=2 : 0.995135, exp_rate<=3 : 0.998644, loss: 0.002086, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18363 s, avg_samples: 56.0, ips: 304.96727 samples/s, eta: 15:42:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:19:58] ppocr INFO: epoch: [393/500], global_step: 252700, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.942933, exp_rate<=1 : 0.977409, exp_rate<=2 : 0.994795, exp_rate<=3 : 0.998366, loss: 0.000756, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19552 s, avg_samples: 56.0, ips: 286.41914 samples/s, eta: 15:42:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:20:18] ppocr INFO: epoch: [393/500], global_step: 252800, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.943379, exp_rate<=1 : 0.977622, exp_rate<=2 : 0.994902, exp_rate<=3 : 0.998362, loss: 0.001118, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19798 s, avg_samples: 56.0, ips: 282.85322 samples/s, eta: 15:42:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:20:37] ppocr INFO: epoch: [393/500], global_step: 252900, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.941995, exp_rate<=1 : 0.977044, exp_rate<=2 : 0.994711, exp_rate<=3 : 0.998263, loss: 0.002590, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18599 s, avg_samples: 56.0, ips: 301.09779 samples/s, eta: 15:41:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:20:55] ppocr INFO: epoch: [393/500], global_step: 253000, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.940729, exp_rate<=1 : 0.976868, exp_rate<=2 : 0.994765, exp_rate<=3 : 0.998313, loss: 0.001698, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18130 s, avg_samples: 56.0, ips: 308.88765 samples/s, eta: 15:41:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:21:14] ppocr INFO: epoch: [393/500], global_step: 253100, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.940080, exp_rate<=1 : 0.976712, exp_rate<=2 : 0.994739, exp_rate<=3 : 0.998357, loss: 0.001236, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19490 s, avg_samples: 56.0, ips: 287.33022 samples/s, eta: 15:41:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:21:33] ppocr INFO: epoch: [393/500], global_step: 253200, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.940052, exp_rate<=1 : 0.976883, exp_rate<=2 : 0.994697, exp_rate<=3 : 0.998329, loss: 0.001477, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19104 s, avg_samples: 56.0, ips: 293.13903 samples/s, eta: 15:40:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:21:53] ppocr INFO: epoch: [393/500], global_step: 253300, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.939361, exp_rate<=1 : 0.976490, exp_rate<=2 : 0.994751, exp_rate<=3 : 0.998376, loss: 0.001402, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19425 s, avg_samples: 56.0, ips: 288.28900 samples/s, eta: 15:40:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:22:10] ppocr INFO: epoch: [393/500], global_step: 253400, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.939182, exp_rate<=1 : 0.976127, exp_rate<=2 : 0.994672, exp_rate<=3 : 0.998351, loss: 0.000872, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16910 s, avg_samples: 56.0, ips: 331.15658 samples/s, eta: 15:40:12, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:22:29] ppocr INFO: epoch: [393/500], global_step: 253500, lr: 0.000100, edit distance: 0.001308, exp_rate: 0.939064, exp_rate<=1 : 0.976119, exp_rate<=2 : 0.994738, exp_rate<=3 : 0.998390, loss: 0.001303, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19286 s, avg_samples: 56.0, ips: 290.37143 samples/s, eta: 15:39:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:22:47] ppocr INFO: epoch: [393/500], global_step: 253600, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.939340, exp_rate<=1 : 0.976189, exp_rate<=2 : 0.994726, exp_rate<=3 : 0.998400, loss: 0.001244, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17961 s, avg_samples: 56.0, ips: 311.79284 samples/s, eta: 15:39:33, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:23:06] ppocr INFO: epoch: [393/500], global_step: 253700, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.939026, exp_rate<=1 : 0.976092, exp_rate<=2 : 0.994696, exp_rate<=3 : 0.998370, loss: 0.001738, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18924 s, avg_samples: 56.0, ips: 295.91789 samples/s, eta: 15:39:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:23:24] ppocr INFO: epoch: [393/500], global_step: 253800, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.938813, exp_rate<=1 : 0.975968, exp_rate<=2 : 0.994654, exp_rate<=3 : 0.998365, loss: 0.001432, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18322 s, avg_samples: 56.0, ips: 305.64435 samples/s, eta: 15:38:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:23:43] ppocr INFO: epoch: [393/500], global_step: 253900, lr: 0.000100, edit distance: 0.001343, exp_rate: 0.938527, exp_rate<=1 : 0.975893, exp_rate<=2 : 0.994624, exp_rate<=3 : 0.998358, loss: 0.001527, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18823 s, avg_samples: 56.0, ips: 297.51338 samples/s, eta: 15:38:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:24:02] ppocr INFO: epoch: [393/500], global_step: 254000, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.937995, exp_rate<=1 : 0.975648, exp_rate<=2 : 0.994614, exp_rate<=3 : 0.998360, loss: 0.002380, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18655 s, avg_samples: 56.0, ips: 300.19441 samples/s, eta: 15:38:14, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:24:20] ppocr INFO: epoch: [393/500], global_step: 254100, lr: 0.000100, edit distance: 0.001372, exp_rate: 0.937716, exp_rate<=1 : 0.975458, exp_rate<=2 : 0.994518, exp_rate<=3 : 0.998321, loss: 0.001475, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18412 s, avg_samples: 56.0, ips: 304.14525 samples/s, eta: 15:37:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:24:40] ppocr INFO: epoch: [393/500], global_step: 254200, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.937434, exp_rate<=1 : 0.975342, exp_rate<=2 : 0.994520, exp_rate<=3 : 0.998313, loss: 0.002037, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20081 s, avg_samples: 56.0, ips: 278.87012 samples/s, eta: 15:37:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:24:58] ppocr INFO: epoch: [393/500], global_step: 254300, lr: 0.000100, edit distance: 0.001387, exp_rate: 0.937174, exp_rate<=1 : 0.975209, exp_rate<=2 : 0.994497, exp_rate<=3 : 0.998318, loss: 0.002109, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17902 s, avg_samples: 56.0, ips: 312.81668 samples/s, eta: 15:37:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:25:17] ppocr INFO: epoch: [393/500], global_step: 254400, lr: 0.000100, edit distance: 0.001386, exp_rate: 0.936953, exp_rate<=1 : 0.975149, exp_rate<=2 : 0.994455, exp_rate<=3 : 0.998293, loss: 0.001524, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18363 s, avg_samples: 56.0, ips: 304.96044 samples/s, eta: 15:36:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:25:35] ppocr INFO: epoch: [393/500], global_step: 254500, lr: 0.000100, edit distance: 0.001380, exp_rate: 0.937467, exp_rate<=1 : 0.975356, exp_rate<=2 : 0.994541, exp_rate<=3 : 0.998298, loss: 0.001884, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18057 s, avg_samples: 56.0, ips: 310.12447 samples/s, eta: 15:36:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:25:53] ppocr INFO: epoch: [393/500], global_step: 254600, lr: 0.000100, edit distance: 0.001384, exp_rate: 0.937267, exp_rate<=1 : 0.975330, exp_rate<=2 : 0.994572, exp_rate<=3 : 0.998302, loss: 0.001863, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18672 s, avg_samples: 56.0, ips: 299.90733 samples/s, eta: 15:36:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:26:13] ppocr INFO: epoch: [393/500], global_step: 254700, lr: 0.000100, edit distance: 0.001386, exp_rate: 0.937135, exp_rate<=1 : 0.975207, exp_rate<=2 : 0.994530, exp_rate<=3 : 0.998304, loss: 0.002063, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19080 s, avg_samples: 56.0, ips: 293.50736 samples/s, eta: 15:35:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:26:31] ppocr INFO: epoch: [393/500], global_step: 254800, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.937298, exp_rate<=1 : 0.975272, exp_rate<=2 : 0.994531, exp_rate<=3 : 0.998293, loss: 0.001495, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18075 s, avg_samples: 56.0, ips: 309.81566 samples/s, eta: 15:35:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:26:34] ppocr INFO: epoch: [393/500], global_step: 254820, lr: 0.000100, edit distance: 0.001379, exp_rate: 0.937366, exp_rate<=1 : 0.975297, exp_rate<=2 : 0.994515, exp_rate<=3 : 0.998279, loss: 0.001850, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03409 s, avg_samples: 11.2, ips: 328.51871 samples/s, eta: 15:35:31, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:26:37] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 16:26:52] ppocr INFO: epoch: [394/500], global_step: 254900, lr: 0.000100, edit distance: 0.001077, exp_rate: 0.953185, exp_rate<=1 : 0.983536, exp_rate<=2 : 0.995625, exp_rate<=3 : 0.999747, loss: 0.001539, avg_reader_cost: 0.03094 s, avg_batch_cost: 0.17949 s, avg_samples: 44.8, ips: 249.59388 samples/s, eta: 15:35:19, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:27:11] ppocr INFO: epoch: [394/500], global_step: 255000, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.945796, exp_rate<=1 : 0.980960, exp_rate<=2 : 0.994928, exp_rate<=3 : 0.999057, loss: 0.001046, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18668 s, avg_samples: 56.0, ips: 299.98299 samples/s, eta: 15:34:59, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:27:30] ppocr INFO: epoch: [394/500], global_step: 255100, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.945245, exp_rate<=1 : 0.979960, exp_rate<=2 : 0.995127, exp_rate<=3 : 0.998657, loss: 0.001291, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19344 s, avg_samples: 56.0, ips: 289.50069 samples/s, eta: 15:34:40, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:27:49] ppocr INFO: epoch: [394/500], global_step: 255200, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.945570, exp_rate<=1 : 0.979231, exp_rate<=2 : 0.995041, exp_rate<=3 : 0.998590, loss: 0.001448, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18409 s, avg_samples: 56.0, ips: 304.20027 samples/s, eta: 15:34:20, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:28:09] ppocr INFO: epoch: [394/500], global_step: 255300, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.943921, exp_rate<=1 : 0.978556, exp_rate<=2 : 0.994783, exp_rate<=3 : 0.998468, loss: 0.001288, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20477 s, avg_samples: 56.0, ips: 273.47927 samples/s, eta: 15:34:03, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:28:28] ppocr INFO: epoch: [394/500], global_step: 255400, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.944490, exp_rate<=1 : 0.978079, exp_rate<=2 : 0.994649, exp_rate<=3 : 0.998451, loss: 0.002104, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19296 s, avg_samples: 56.0, ips: 290.20882 samples/s, eta: 15:33:44, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:28:47] ppocr INFO: epoch: [394/500], global_step: 255500, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.943177, exp_rate<=1 : 0.977947, exp_rate<=2 : 0.994837, exp_rate<=3 : 0.998525, loss: 0.000897, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18811 s, avg_samples: 56.0, ips: 297.69446 samples/s, eta: 15:33:24, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:29:06] ppocr INFO: epoch: [394/500], global_step: 255600, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.942684, exp_rate<=1 : 0.977641, exp_rate<=2 : 0.994814, exp_rate<=3 : 0.998420, loss: 0.001969, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18665 s, avg_samples: 56.0, ips: 300.03181 samples/s, eta: 15:33:05, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:29:26] ppocr INFO: epoch: [394/500], global_step: 255700, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.940425, exp_rate<=1 : 0.976791, exp_rate<=2 : 0.994660, exp_rate<=3 : 0.998381, loss: 0.001349, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20631 s, avg_samples: 56.0, ips: 271.43908 samples/s, eta: 15:32:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:29:45] ppocr INFO: epoch: [394/500], global_step: 255800, lr: 0.000100, edit distance: 0.001341, exp_rate: 0.939823, exp_rate<=1 : 0.976172, exp_rate<=2 : 0.994482, exp_rate<=3 : 0.998259, loss: 0.001334, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18801 s, avg_samples: 56.0, ips: 297.85774 samples/s, eta: 15:32:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:30:05] ppocr INFO: epoch: [394/500], global_step: 255900, lr: 0.000100, edit distance: 0.001415, exp_rate: 0.936682, exp_rate<=1 : 0.974862, exp_rate<=2 : 0.993911, exp_rate<=3 : 0.998014, loss: 0.001584, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19570 s, avg_samples: 56.0, ips: 286.15254 samples/s, eta: 15:32:09, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:30:23] ppocr INFO: epoch: [394/500], global_step: 256000, lr: 0.000100, edit distance: 0.001463, exp_rate: 0.935678, exp_rate<=1 : 0.974645, exp_rate<=2 : 0.993898, exp_rate<=3 : 0.997952, loss: 0.001682, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18174 s, avg_samples: 56.0, ips: 308.13888 samples/s, eta: 15:31:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:30:42] ppocr INFO: epoch: [394/500], global_step: 256100, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.936059, exp_rate<=1 : 0.974683, exp_rate<=2 : 0.993802, exp_rate<=3 : 0.997864, loss: 0.001072, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19171 s, avg_samples: 56.0, ips: 292.11243 samples/s, eta: 15:31:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:31:01] ppocr INFO: epoch: [394/500], global_step: 256200, lr: 0.000100, edit distance: 0.001436, exp_rate: 0.935972, exp_rate<=1 : 0.974639, exp_rate<=2 : 0.993774, exp_rate<=3 : 0.997842, loss: 0.001379, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18998 s, avg_samples: 56.0, ips: 294.76787 samples/s, eta: 15:31:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:31:19] ppocr INFO: epoch: [394/500], global_step: 256300, lr: 0.000100, edit distance: 0.001432, exp_rate: 0.936542, exp_rate<=1 : 0.974923, exp_rate<=2 : 0.993835, exp_rate<=3 : 0.997872, loss: 0.001394, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17367 s, avg_samples: 56.0, ips: 322.45878 samples/s, eta: 15:30:49, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:31:38] ppocr INFO: epoch: [394/500], global_step: 256400, lr: 0.000100, edit distance: 0.001438, exp_rate: 0.936571, exp_rate<=1 : 0.974952, exp_rate<=2 : 0.993800, exp_rate<=3 : 0.997840, loss: 0.001654, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19181 s, avg_samples: 56.0, ips: 291.95273 samples/s, eta: 15:30:30, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:31:58] ppocr INFO: epoch: [394/500], global_step: 256500, lr: 0.000100, edit distance: 0.001437, exp_rate: 0.936210, exp_rate<=1 : 0.974765, exp_rate<=2 : 0.993692, exp_rate<=3 : 0.997777, loss: 0.001935, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19677 s, avg_samples: 56.0, ips: 284.59095 samples/s, eta: 15:30:11, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:32:15] ppocr INFO: epoch: [394/500], global_step: 256600, lr: 0.000100, edit distance: 0.001420, exp_rate: 0.936491, exp_rate<=1 : 0.975046, exp_rate<=2 : 0.993799, exp_rate<=3 : 0.997842, loss: 0.001284, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17718 s, avg_samples: 56.0, ips: 316.07081 samples/s, eta: 15:29:51, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:32:33] ppocr INFO: epoch: [394/500], global_step: 256700, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.937082, exp_rate<=1 : 0.975264, exp_rate<=2 : 0.993919, exp_rate<=3 : 0.997890, loss: 0.000861, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17298 s, avg_samples: 56.0, ips: 323.74194 samples/s, eta: 15:29:29, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:32:51] ppocr INFO: epoch: [394/500], global_step: 256800, lr: 0.000100, edit distance: 0.001406, exp_rate: 0.936876, exp_rate<=1 : 0.975006, exp_rate<=2 : 0.993792, exp_rate<=3 : 0.997807, loss: 0.001501, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18633 s, avg_samples: 56.0, ips: 300.55010 samples/s, eta: 15:29:10, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:33:08] ppocr INFO: epoch: [394/500], global_step: 256900, lr: 0.000100, edit distance: 0.001407, exp_rate: 0.936948, exp_rate<=1 : 0.974898, exp_rate<=2 : 0.993589, exp_rate<=3 : 0.997753, loss: 0.002468, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17071 s, avg_samples: 56.0, ips: 328.04266 samples/s, eta: 15:28:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:33:27] ppocr INFO: epoch: [394/500], global_step: 257000, lr: 0.000100, edit distance: 0.001404, exp_rate: 0.937013, exp_rate<=1 : 0.974909, exp_rate<=2 : 0.993655, exp_rate<=3 : 0.997803, loss: 0.001351, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18240 s, avg_samples: 56.0, ips: 307.02456 samples/s, eta: 15:28:28, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:33:44] ppocr INFO: epoch: [394/500], global_step: 257100, lr: 0.000100, edit distance: 0.001406, exp_rate: 0.936940, exp_rate<=1 : 0.974885, exp_rate<=2 : 0.993700, exp_rate<=3 : 0.997852, loss: 0.001958, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16903 s, avg_samples: 56.0, ips: 331.29817 samples/s, eta: 15:28:06, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:34:03] ppocr INFO: epoch: [394/500], global_step: 257200, lr: 0.000100, edit distance: 0.001412, exp_rate: 0.936482, exp_rate<=1 : 0.974638, exp_rate<=2 : 0.993605, exp_rate<=3 : 0.997836, loss: 0.000990, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19211 s, avg_samples: 56.0, ips: 291.50552 samples/s, eta: 15:27:47, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:34:21] ppocr INFO: epoch: [394/500], global_step: 257300, lr: 0.000100, edit distance: 0.001417, exp_rate: 0.936588, exp_rate<=1 : 0.974778, exp_rate<=2 : 0.993709, exp_rate<=3 : 0.997854, loss: 0.001296, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17979 s, avg_samples: 56.0, ips: 311.46944 samples/s, eta: 15:27:27, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:34:40] ppocr INFO: epoch: [394/500], global_step: 257400, lr: 0.000100, edit distance: 0.001421, exp_rate: 0.936306, exp_rate<=1 : 0.974607, exp_rate<=2 : 0.993697, exp_rate<=3 : 0.997839, loss: 0.001274, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19126 s, avg_samples: 56.0, ips: 292.78931 samples/s, eta: 15:27:08, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:34:59] ppocr INFO: epoch: [394/500], global_step: 257500, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.936194, exp_rate<=1 : 0.974568, exp_rate<=2 : 0.993732, exp_rate<=3 : 0.997860, loss: 0.001365, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18645 s, avg_samples: 56.0, ips: 300.35597 samples/s, eta: 15:26:48, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:35:10] ppocr INFO: epoch: [394/500], global_step: 257560, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.936244, exp_rate<=1 : 0.974619, exp_rate<=2 : 0.993769, exp_rate<=3 : 0.997879, loss: 0.001493, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11488 s, avg_samples: 33.6, ips: 292.47118 samples/s, eta: 15:26:36, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:35:12] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 16:35:21] ppocr INFO: epoch: [395/500], global_step: 257600, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.937832, exp_rate<=1 : 0.976526, exp_rate<=2 : 0.992801, exp_rate<=3 : 0.996949, loss: 0.001422, avg_reader_cost: 0.02631 s, avg_batch_cost: 0.10947 s, avg_samples: 22.4, ips: 204.61463 samples/s, eta: 15:26:32, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:35:40] ppocr INFO: epoch: [395/500], global_step: 257700, lr: 0.000100, edit distance: 0.001096, exp_rate: 0.948867, exp_rate<=1 : 0.979552, exp_rate<=2 : 0.994946, exp_rate<=3 : 0.998024, loss: 0.001193, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18972 s, avg_samples: 56.0, ips: 295.17122 samples/s, eta: 15:26:13, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:35:59] ppocr INFO: epoch: [395/500], global_step: 257800, lr: 0.000100, edit distance: 0.001065, exp_rate: 0.948038, exp_rate<=1 : 0.981484, exp_rate<=2 : 0.995672, exp_rate<=3 : 0.998122, loss: 0.001183, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18463 s, avg_samples: 56.0, ips: 303.30879 samples/s, eta: 15:25:53, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:36:18] ppocr INFO: epoch: [395/500], global_step: 257900, lr: 0.000100, edit distance: 0.001052, exp_rate: 0.947770, exp_rate<=1 : 0.981164, exp_rate<=2 : 0.995407, exp_rate<=3 : 0.997824, loss: 0.001131, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19761 s, avg_samples: 56.0, ips: 283.38236 samples/s, eta: 15:25:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:36:37] ppocr INFO: epoch: [395/500], global_step: 258000, lr: 0.000100, edit distance: 0.001058, exp_rate: 0.948825, exp_rate<=1 : 0.981806, exp_rate<=2 : 0.995583, exp_rate<=3 : 0.997981, loss: 0.001896, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18397 s, avg_samples: 56.0, ips: 304.39560 samples/s, eta: 15:25:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:36:54] ppocr INFO: epoch: [395/500], global_step: 258100, lr: 0.000100, edit distance: 0.001118, exp_rate: 0.947524, exp_rate<=1 : 0.981258, exp_rate<=2 : 0.995502, exp_rate<=3 : 0.997950, loss: 0.001426, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17293 s, avg_samples: 56.0, ips: 323.82276 samples/s, eta: 15:24:54, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:37:13] ppocr INFO: epoch: [395/500], global_step: 258200, lr: 0.000100, edit distance: 0.001111, exp_rate: 0.946513, exp_rate<=1 : 0.980908, exp_rate<=2 : 0.995168, exp_rate<=3 : 0.997820, loss: 0.001824, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18827 s, avg_samples: 56.0, ips: 297.45179 samples/s, eta: 15:24:34, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:37:33] ppocr INFO: epoch: [395/500], global_step: 258300, lr: 0.000100, edit distance: 0.001122, exp_rate: 0.944816, exp_rate<=1 : 0.980037, exp_rate<=2 : 0.994921, exp_rate<=3 : 0.997825, loss: 0.001670, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19698 s, avg_samples: 56.0, ips: 284.29394 samples/s, eta: 15:24:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:37:51] ppocr INFO: epoch: [395/500], global_step: 258400, lr: 0.000100, edit distance: 0.001119, exp_rate: 0.945154, exp_rate<=1 : 0.980169, exp_rate<=2 : 0.995102, exp_rate<=3 : 0.997944, loss: 0.001518, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18213 s, avg_samples: 56.0, ips: 307.46887 samples/s, eta: 15:23:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:38:09] ppocr INFO: epoch: [395/500], global_step: 258500, lr: 0.000100, edit distance: 0.001132, exp_rate: 0.944991, exp_rate<=1 : 0.979778, exp_rate<=2 : 0.995102, exp_rate<=3 : 0.998041, loss: 0.001468, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18045 s, avg_samples: 56.0, ips: 310.34198 samples/s, eta: 15:23:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:38:27] ppocr INFO: epoch: [395/500], global_step: 258600, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.944243, exp_rate<=1 : 0.979500, exp_rate<=2 : 0.994994, exp_rate<=3 : 0.997976, loss: 0.001585, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18502 s, avg_samples: 56.0, ips: 302.66852 samples/s, eta: 15:23:15, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:38:45] ppocr INFO: epoch: [395/500], global_step: 258700, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.944429, exp_rate<=1 : 0.979618, exp_rate<=2 : 0.995067, exp_rate<=3 : 0.997981, loss: 0.001291, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18035 s, avg_samples: 56.0, ips: 310.50123 samples/s, eta: 15:22:55, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:39:04] ppocr INFO: epoch: [395/500], global_step: 258800, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.943718, exp_rate<=1 : 0.979161, exp_rate<=2 : 0.995037, exp_rate<=3 : 0.997983, loss: 0.001258, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18661 s, avg_samples: 56.0, ips: 300.08743 samples/s, eta: 15:22:35, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:39:23] ppocr INFO: epoch: [395/500], global_step: 258900, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.943085, exp_rate<=1 : 0.978996, exp_rate<=2 : 0.994951, exp_rate<=3 : 0.997976, loss: 0.001707, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19016 s, avg_samples: 56.0, ips: 294.49393 samples/s, eta: 15:22:16, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:39:42] ppocr INFO: epoch: [395/500], global_step: 259000, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.942385, exp_rate<=1 : 0.978691, exp_rate<=2 : 0.994763, exp_rate<=3 : 0.997938, loss: 0.002004, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18515 s, avg_samples: 56.0, ips: 302.45643 samples/s, eta: 15:21:56, max_mem_reserved: 30359 MB, max_mem_allocated: 24983 MB [2024/07/10 16:40:01] ppocr INFO: epoch: [395/500], global_step: 259100, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.941807, exp_rate<=1 : 0.978581, exp_rate<=2 : 0.994754, exp_rate<=3 : 0.997985, loss: 0.001934, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19318 s, avg_samples: 56.0, ips: 289.88595 samples/s, eta: 15:21:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:40:18] ppocr INFO: epoch: [395/500], global_step: 259200, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.941561, exp_rate<=1 : 0.978392, exp_rate<=2 : 0.994727, exp_rate<=3 : 0.998051, loss: 0.001401, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17266 s, avg_samples: 56.0, ips: 324.34592 samples/s, eta: 15:21:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:40:37] ppocr INFO: epoch: [395/500], global_step: 259300, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.941756, exp_rate<=1 : 0.978374, exp_rate<=2 : 0.994753, exp_rate<=3 : 0.998054, loss: 0.001053, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18284 s, avg_samples: 56.0, ips: 306.27435 samples/s, eta: 15:20:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:40:55] ppocr INFO: epoch: [395/500], global_step: 259400, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.941775, exp_rate<=1 : 0.978328, exp_rate<=2 : 0.994758, exp_rate<=3 : 0.998024, loss: 0.001546, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18177 s, avg_samples: 56.0, ips: 308.07379 samples/s, eta: 15:20:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:41:13] ppocr INFO: epoch: [395/500], global_step: 259500, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.941066, exp_rate<=1 : 0.977994, exp_rate<=2 : 0.994739, exp_rate<=3 : 0.998037, loss: 0.001760, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18477 s, avg_samples: 56.0, ips: 303.08205 samples/s, eta: 15:20:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:41:31] ppocr INFO: epoch: [395/500], global_step: 259600, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.940807, exp_rate<=1 : 0.977724, exp_rate<=2 : 0.994703, exp_rate<=3 : 0.998099, loss: 0.001336, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17997 s, avg_samples: 56.0, ips: 311.16981 samples/s, eta: 15:19:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:41:51] ppocr INFO: epoch: [395/500], global_step: 259700, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.940374, exp_rate<=1 : 0.977387, exp_rate<=2 : 0.994654, exp_rate<=3 : 0.998047, loss: 0.001607, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19534 s, avg_samples: 56.0, ips: 286.67353 samples/s, eta: 15:19:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:42:08] ppocr INFO: epoch: [395/500], global_step: 259800, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.940565, exp_rate<=1 : 0.977535, exp_rate<=2 : 0.994684, exp_rate<=3 : 0.998068, loss: 0.001947, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17281 s, avg_samples: 56.0, ips: 324.06432 samples/s, eta: 15:19:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:42:28] ppocr INFO: epoch: [395/500], global_step: 259900, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.939865, exp_rate<=1 : 0.977237, exp_rate<=2 : 0.994674, exp_rate<=3 : 0.998084, loss: 0.002260, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20162 s, avg_samples: 56.0, ips: 277.74883 samples/s, eta: 15:18:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:42:50] ppocr INFO: epoch: [395/500], global_step: 260000, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.938416, exp_rate<=1 : 0.976492, exp_rate<=2 : 0.994440, exp_rate<=3 : 0.997970, loss: 0.001813, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21414 s, avg_samples: 56.0, ips: 261.51408 samples/s, eta: 15:18:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:43:09] ppocr INFO: epoch: [395/500], global_step: 260100, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.937754, exp_rate<=1 : 0.976268, exp_rate<=2 : 0.994375, exp_rate<=3 : 0.997968, loss: 0.001658, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18822 s, avg_samples: 56.0, ips: 297.52536 samples/s, eta: 15:18:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:43:28] ppocr INFO: epoch: [395/500], global_step: 260200, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.937486, exp_rate<=1 : 0.976094, exp_rate<=2 : 0.994368, exp_rate<=3 : 0.997979, loss: 0.001704, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18997 s, avg_samples: 56.0, ips: 294.78142 samples/s, eta: 15:18:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:43:47] ppocr INFO: epoch: [395/500], global_step: 260300, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.937356, exp_rate<=1 : 0.976028, exp_rate<=2 : 0.994302, exp_rate<=3 : 0.997966, loss: 0.001416, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19309 s, avg_samples: 56.0, ips: 290.01555 samples/s, eta: 15:17:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:43:49] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 16:43:50] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_395 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 16:44:08] ppocr INFO: epoch: [396/500], global_step: 260400, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.947940, exp_rate<=1 : 0.978607, exp_rate<=2 : 0.996152, exp_rate<=3 : 0.998329, loss: 0.001418, avg_reader_cost: 0.03388 s, avg_batch_cost: 0.21417 s, avg_samples: 56.0, ips: 261.47288 samples/s, eta: 15:17:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:44:27] ppocr INFO: epoch: [396/500], global_step: 260500, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.947129, exp_rate<=1 : 0.979424, exp_rate<=2 : 0.996120, exp_rate<=3 : 0.998738, loss: 0.001164, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18117 s, avg_samples: 56.0, ips: 309.09643 samples/s, eta: 15:17:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:44:47] ppocr INFO: epoch: [396/500], global_step: 260600, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.947539, exp_rate<=1 : 0.980342, exp_rate<=2 : 0.996114, exp_rate<=3 : 0.998722, loss: 0.001408, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20144 s, avg_samples: 56.0, ips: 278.00235 samples/s, eta: 15:16:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:45:06] ppocr INFO: epoch: [396/500], global_step: 260700, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.944764, exp_rate<=1 : 0.979676, exp_rate<=2 : 0.995836, exp_rate<=3 : 0.998733, loss: 0.001788, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19617 s, avg_samples: 56.0, ips: 285.46622 samples/s, eta: 15:16:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:45:26] ppocr INFO: epoch: [396/500], global_step: 260800, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.944554, exp_rate<=1 : 0.979493, exp_rate<=2 : 0.995848, exp_rate<=3 : 0.998728, loss: 0.001541, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19827 s, avg_samples: 56.0, ips: 282.43923 samples/s, eta: 15:16:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:45:46] ppocr INFO: epoch: [396/500], global_step: 260900, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.943696, exp_rate<=1 : 0.979339, exp_rate<=2 : 0.995763, exp_rate<=3 : 0.998630, loss: 0.001039, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20074 s, avg_samples: 56.0, ips: 278.96267 samples/s, eta: 15:15:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:46:04] ppocr INFO: epoch: [396/500], global_step: 261000, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.945374, exp_rate<=1 : 0.980074, exp_rate<=2 : 0.995769, exp_rate<=3 : 0.998677, loss: 0.001269, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17929 s, avg_samples: 56.0, ips: 312.34203 samples/s, eta: 15:15:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:46:23] ppocr INFO: epoch: [396/500], global_step: 261100, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.943991, exp_rate<=1 : 0.979800, exp_rate<=2 : 0.995819, exp_rate<=3 : 0.998652, loss: 0.001458, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18268 s, avg_samples: 56.0, ips: 306.54571 samples/s, eta: 15:15:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:46:42] ppocr INFO: epoch: [396/500], global_step: 261200, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.942873, exp_rate<=1 : 0.979058, exp_rate<=2 : 0.995428, exp_rate<=3 : 0.998398, loss: 0.001274, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19372 s, avg_samples: 56.0, ips: 289.07747 samples/s, eta: 15:14:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:47:02] ppocr INFO: epoch: [396/500], global_step: 261300, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.942315, exp_rate<=1 : 0.978750, exp_rate<=2 : 0.995249, exp_rate<=3 : 0.998376, loss: 0.001905, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19646 s, avg_samples: 56.0, ips: 285.03840 samples/s, eta: 15:14:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:47:19] ppocr INFO: epoch: [396/500], global_step: 261400, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.941517, exp_rate<=1 : 0.978421, exp_rate<=2 : 0.995274, exp_rate<=3 : 0.998319, loss: 0.002953, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17731 s, avg_samples: 56.0, ips: 315.83220 samples/s, eta: 15:14:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:47:38] ppocr INFO: epoch: [396/500], global_step: 261500, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.940820, exp_rate<=1 : 0.978311, exp_rate<=2 : 0.995330, exp_rate<=3 : 0.998317, loss: 0.001996, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19044 s, avg_samples: 56.0, ips: 294.05493 samples/s, eta: 15:13:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:47:59] ppocr INFO: epoch: [396/500], global_step: 261600, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.940077, exp_rate<=1 : 0.977806, exp_rate<=2 : 0.995128, exp_rate<=3 : 0.998300, loss: 0.000945, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20199 s, avg_samples: 56.0, ips: 277.23729 samples/s, eta: 15:13:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:48:18] ppocr INFO: epoch: [396/500], global_step: 261700, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.940541, exp_rate<=1 : 0.977970, exp_rate<=2 : 0.995254, exp_rate<=3 : 0.998331, loss: 0.001553, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19078 s, avg_samples: 56.0, ips: 293.53917 samples/s, eta: 15:13:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:48:35] ppocr INFO: epoch: [396/500], global_step: 261800, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.940965, exp_rate<=1 : 0.978028, exp_rate<=2 : 0.995244, exp_rate<=3 : 0.998371, loss: 0.001474, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16984 s, avg_samples: 56.0, ips: 329.72507 samples/s, eta: 15:12:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:48:55] ppocr INFO: epoch: [396/500], global_step: 261900, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.939664, exp_rate<=1 : 0.977464, exp_rate<=2 : 0.995093, exp_rate<=3 : 0.998312, loss: 0.001858, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20280 s, avg_samples: 56.0, ips: 276.13385 samples/s, eta: 15:12:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:49:14] ppocr INFO: epoch: [396/500], global_step: 262000, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.939195, exp_rate<=1 : 0.977075, exp_rate<=2 : 0.994993, exp_rate<=3 : 0.998266, loss: 0.001387, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19082 s, avg_samples: 56.0, ips: 293.47185 samples/s, eta: 15:12:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:49:32] ppocr INFO: epoch: [396/500], global_step: 262100, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.939482, exp_rate<=1 : 0.977063, exp_rate<=2 : 0.995088, exp_rate<=3 : 0.998315, loss: 0.001448, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17874 s, avg_samples: 56.0, ips: 313.31029 samples/s, eta: 15:11:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:49:50] ppocr INFO: epoch: [396/500], global_step: 262200, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.939895, exp_rate<=1 : 0.977241, exp_rate<=2 : 0.995121, exp_rate<=3 : 0.998356, loss: 0.001834, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17678 s, avg_samples: 56.0, ips: 316.77731 samples/s, eta: 15:11:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:50:09] ppocr INFO: epoch: [396/500], global_step: 262300, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.939243, exp_rate<=1 : 0.976986, exp_rate<=2 : 0.995000, exp_rate<=3 : 0.998351, loss: 0.001977, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18762 s, avg_samples: 56.0, ips: 298.47179 samples/s, eta: 15:11:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:50:27] ppocr INFO: epoch: [396/500], global_step: 262400, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.939081, exp_rate<=1 : 0.976911, exp_rate<=2 : 0.994936, exp_rate<=3 : 0.998306, loss: 0.001495, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18512 s, avg_samples: 56.0, ips: 302.50592 samples/s, eta: 15:10:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:50:44] ppocr INFO: epoch: [396/500], global_step: 262500, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.939351, exp_rate<=1 : 0.977020, exp_rate<=2 : 0.994996, exp_rate<=3 : 0.998339, loss: 0.001950, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16903 s, avg_samples: 56.0, ips: 331.30836 samples/s, eta: 15:10:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:51:02] ppocr INFO: epoch: [396/500], global_step: 262600, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.939085, exp_rate<=1 : 0.976720, exp_rate<=2 : 0.994909, exp_rate<=3 : 0.998300, loss: 0.001575, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18311 s, avg_samples: 56.0, ips: 305.81966 samples/s, eta: 15:10:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:51:22] ppocr INFO: epoch: [396/500], global_step: 262700, lr: 0.000100, edit distance: 0.001354, exp_rate: 0.938757, exp_rate<=1 : 0.976494, exp_rate<=2 : 0.994840, exp_rate<=3 : 0.998306, loss: 0.001708, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20077 s, avg_samples: 56.0, ips: 278.93209 samples/s, eta: 15:09:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:51:42] ppocr INFO: epoch: [396/500], global_step: 262800, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.938503, exp_rate<=1 : 0.976475, exp_rate<=2 : 0.994857, exp_rate<=3 : 0.998310, loss: 0.001547, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19808 s, avg_samples: 56.0, ips: 282.71606 samples/s, eta: 15:09:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:51:59] ppocr INFO: epoch: [396/500], global_step: 262900, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.938614, exp_rate<=1 : 0.976490, exp_rate<=2 : 0.994884, exp_rate<=3 : 0.998318, loss: 0.001824, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17191 s, avg_samples: 56.0, ips: 325.76127 samples/s, eta: 15:09:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:52:17] ppocr INFO: epoch: [396/500], global_step: 263000, lr: 0.000100, edit distance: 0.001354, exp_rate: 0.938766, exp_rate<=1 : 0.976629, exp_rate<=2 : 0.994923, exp_rate<=3 : 0.998321, loss: 0.001382, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17079 s, avg_samples: 56.0, ips: 327.88472 samples/s, eta: 15:08:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:52:24] ppocr INFO: epoch: [396/500], global_step: 263040, lr: 0.000100, edit distance: 0.001359, exp_rate: 0.938515, exp_rate<=1 : 0.976525, exp_rate<=2 : 0.994878, exp_rate<=3 : 0.998308, loss: 0.002086, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.07499 s, avg_samples: 22.4, ips: 298.72504 samples/s, eta: 15:08:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:52:26] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 16:52:37] ppocr INFO: epoch: [397/500], global_step: 263100, lr: 0.000100, edit distance: 0.001073, exp_rate: 0.945753, exp_rate<=1 : 0.979953, exp_rate<=2 : 0.996860, exp_rate<=3 : 0.999293, loss: 0.000943, avg_reader_cost: 0.02569 s, avg_batch_cost: 0.13286 s, avg_samples: 33.6, ips: 252.90457 samples/s, eta: 15:08:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:52:57] ppocr INFO: epoch: [397/500], global_step: 263200, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.935797, exp_rate<=1 : 0.976497, exp_rate<=2 : 0.994898, exp_rate<=3 : 0.998458, loss: 0.001643, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19222 s, avg_samples: 56.0, ips: 291.33191 samples/s, eta: 15:08:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:53:17] ppocr INFO: epoch: [397/500], global_step: 263300, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.935476, exp_rate<=1 : 0.974198, exp_rate<=2 : 0.993929, exp_rate<=3 : 0.998036, loss: 0.001149, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20097 s, avg_samples: 56.0, ips: 278.64919 samples/s, eta: 15:08:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:53:35] ppocr INFO: epoch: [397/500], global_step: 263400, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.938611, exp_rate<=1 : 0.976071, exp_rate<=2 : 0.994602, exp_rate<=3 : 0.998181, loss: 0.001025, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18589 s, avg_samples: 56.0, ips: 301.25411 samples/s, eta: 15:07:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:53:53] ppocr INFO: epoch: [397/500], global_step: 263500, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.941207, exp_rate<=1 : 0.977405, exp_rate<=2 : 0.994952, exp_rate<=3 : 0.998402, loss: 0.001418, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18067 s, avg_samples: 56.0, ips: 309.95470 samples/s, eta: 15:07:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:54:12] ppocr INFO: epoch: [397/500], global_step: 263600, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.942366, exp_rate<=1 : 0.977870, exp_rate<=2 : 0.995179, exp_rate<=3 : 0.998540, loss: 0.001800, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18167 s, avg_samples: 56.0, ips: 308.25188 samples/s, eta: 15:07:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:54:31] ppocr INFO: epoch: [397/500], global_step: 263700, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.942474, exp_rate<=1 : 0.978094, exp_rate<=2 : 0.995086, exp_rate<=3 : 0.998587, loss: 0.001506, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19011 s, avg_samples: 56.0, ips: 294.56087 samples/s, eta: 15:06:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:54:49] ppocr INFO: epoch: [397/500], global_step: 263800, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.942855, exp_rate<=1 : 0.978249, exp_rate<=2 : 0.995304, exp_rate<=3 : 0.998658, loss: 0.001061, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18475 s, avg_samples: 56.0, ips: 303.11810 samples/s, eta: 15:06:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:55:08] ppocr INFO: epoch: [397/500], global_step: 263900, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.942179, exp_rate<=1 : 0.977837, exp_rate<=2 : 0.995234, exp_rate<=3 : 0.998677, loss: 0.001339, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18569 s, avg_samples: 56.0, ips: 301.58595 samples/s, eta: 15:06:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:55:26] ppocr INFO: epoch: [397/500], global_step: 264000, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.941262, exp_rate<=1 : 0.977475, exp_rate<=2 : 0.995228, exp_rate<=3 : 0.998674, loss: 0.001249, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18678 s, avg_samples: 56.0, ips: 299.82447 samples/s, eta: 15:05:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:55:45] ppocr INFO: epoch: [397/500], global_step: 264100, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.941850, exp_rate<=1 : 0.977485, exp_rate<=2 : 0.995319, exp_rate<=3 : 0.998700, loss: 0.000886, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18314 s, avg_samples: 56.0, ips: 305.76928 samples/s, eta: 15:05:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:56:03] ppocr INFO: epoch: [397/500], global_step: 264200, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.942361, exp_rate<=1 : 0.977810, exp_rate<=2 : 0.995358, exp_rate<=3 : 0.998681, loss: 0.001152, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18225 s, avg_samples: 56.0, ips: 307.26729 samples/s, eta: 15:05:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:56:21] ppocr INFO: epoch: [397/500], global_step: 264300, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.942291, exp_rate<=1 : 0.977865, exp_rate<=2 : 0.995356, exp_rate<=3 : 0.998698, loss: 0.001730, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18143 s, avg_samples: 56.0, ips: 308.65092 samples/s, eta: 15:04:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:56:41] ppocr INFO: epoch: [397/500], global_step: 264400, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.941467, exp_rate<=1 : 0.977469, exp_rate<=2 : 0.995217, exp_rate<=3 : 0.998617, loss: 0.001742, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19563 s, avg_samples: 56.0, ips: 286.24986 samples/s, eta: 15:04:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:57:00] ppocr INFO: epoch: [397/500], global_step: 264500, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.940567, exp_rate<=1 : 0.977243, exp_rate<=2 : 0.995070, exp_rate<=3 : 0.998545, loss: 0.001539, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19345 s, avg_samples: 56.0, ips: 289.48529 samples/s, eta: 15:04:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:57:19] ppocr INFO: epoch: [397/500], global_step: 264600, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.939695, exp_rate<=1 : 0.976881, exp_rate<=2 : 0.994905, exp_rate<=3 : 0.998511, loss: 0.001470, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19315 s, avg_samples: 56.0, ips: 289.92469 samples/s, eta: 15:03:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:57:38] ppocr INFO: epoch: [397/500], global_step: 264700, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.939125, exp_rate<=1 : 0.976919, exp_rate<=2 : 0.994945, exp_rate<=3 : 0.998549, loss: 0.002285, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18132 s, avg_samples: 56.0, ips: 308.84123 samples/s, eta: 15:03:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:57:57] ppocr INFO: epoch: [397/500], global_step: 264800, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.938385, exp_rate<=1 : 0.976660, exp_rate<=2 : 0.994974, exp_rate<=3 : 0.998574, loss: 0.001545, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19684 s, avg_samples: 56.0, ips: 284.48964 samples/s, eta: 15:03:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:58:17] ppocr INFO: epoch: [397/500], global_step: 264900, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.937924, exp_rate<=1 : 0.976329, exp_rate<=2 : 0.994813, exp_rate<=3 : 0.998491, loss: 0.001683, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19820 s, avg_samples: 56.0, ips: 282.54909 samples/s, eta: 15:02:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:58:36] ppocr INFO: epoch: [397/500], global_step: 265000, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.937991, exp_rate<=1 : 0.976285, exp_rate<=2 : 0.994649, exp_rate<=3 : 0.998379, loss: 0.001659, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18564 s, avg_samples: 56.0, ips: 301.65119 samples/s, eta: 15:02:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:58:55] ppocr INFO: epoch: [397/500], global_step: 265100, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.937675, exp_rate<=1 : 0.976257, exp_rate<=2 : 0.994653, exp_rate<=3 : 0.998368, loss: 0.001778, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19509 s, avg_samples: 56.0, ips: 287.04063 samples/s, eta: 15:02:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:59:12] ppocr INFO: epoch: [397/500], global_step: 265200, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.937943, exp_rate<=1 : 0.976390, exp_rate<=2 : 0.994709, exp_rate<=3 : 0.998401, loss: 0.001349, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16847 s, avg_samples: 56.0, ips: 332.41266 samples/s, eta: 15:01:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:59:30] ppocr INFO: epoch: [397/500], global_step: 265300, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.937760, exp_rate<=1 : 0.976415, exp_rate<=2 : 0.994693, exp_rate<=3 : 0.998386, loss: 0.002022, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18035 s, avg_samples: 56.0, ips: 310.50472 samples/s, eta: 15:01:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 16:59:49] ppocr INFO: epoch: [397/500], global_step: 265400, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.937454, exp_rate<=1 : 0.976315, exp_rate<=2 : 0.994651, exp_rate<=3 : 0.998363, loss: 0.001916, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18460 s, avg_samples: 56.0, ips: 303.35368 samples/s, eta: 15:01:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:00:07] ppocr INFO: epoch: [397/500], global_step: 265500, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.937334, exp_rate<=1 : 0.976269, exp_rate<=2 : 0.994666, exp_rate<=3 : 0.998379, loss: 0.001467, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18295 s, avg_samples: 56.0, ips: 306.10171 samples/s, eta: 15:00:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:00:26] ppocr INFO: epoch: [397/500], global_step: 265600, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.937339, exp_rate<=1 : 0.976297, exp_rate<=2 : 0.994637, exp_rate<=3 : 0.998370, loss: 0.001077, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18582 s, avg_samples: 56.0, ips: 301.36270 samples/s, eta: 15:00:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:00:45] ppocr INFO: epoch: [397/500], global_step: 265700, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.937197, exp_rate<=1 : 0.976246, exp_rate<=2 : 0.994597, exp_rate<=3 : 0.998329, loss: 0.001646, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18944 s, avg_samples: 56.0, ips: 295.60936 samples/s, eta: 15:00:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:01:00] ppocr INFO: epoch: [397/500], global_step: 265780, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.936847, exp_rate<=1 : 0.975988, exp_rate<=2 : 0.994476, exp_rate<=3 : 0.998242, loss: 0.001735, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15420 s, avg_samples: 44.8, ips: 290.52636 samples/s, eta: 14:59:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:01:02] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 17:01:06] ppocr INFO: epoch: [398/500], global_step: 265800, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.912946, exp_rate<=1 : 0.971046, exp_rate<=2 : 0.985989, exp_rate<=3 : 0.995387, loss: 0.001450, avg_reader_cost: 0.02624 s, avg_batch_cost: 0.06511 s, avg_samples: 11.2, ips: 172.00445 samples/s, eta: 14:59:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:01:25] ppocr INFO: epoch: [398/500], global_step: 265900, lr: 0.000100, edit distance: 0.001177, exp_rate: 0.936349, exp_rate<=1 : 0.976244, exp_rate<=2 : 0.993293, exp_rate<=3 : 0.997980, loss: 0.001404, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18994 s, avg_samples: 56.0, ips: 294.83656 samples/s, eta: 14:59:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:01:45] ppocr INFO: epoch: [398/500], global_step: 266000, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.936610, exp_rate<=1 : 0.974724, exp_rate<=2 : 0.994103, exp_rate<=3 : 0.997705, loss: 0.001583, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19823 s, avg_samples: 56.0, ips: 282.50218 samples/s, eta: 14:59:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:02:03] ppocr INFO: epoch: [398/500], global_step: 266100, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.939501, exp_rate<=1 : 0.976267, exp_rate<=2 : 0.994574, exp_rate<=3 : 0.997884, loss: 0.000707, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18029 s, avg_samples: 56.0, ips: 310.60473 samples/s, eta: 14:58:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:02:23] ppocr INFO: epoch: [398/500], global_step: 266200, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.941594, exp_rate<=1 : 0.977519, exp_rate<=2 : 0.994551, exp_rate<=3 : 0.997975, loss: 0.001449, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19193 s, avg_samples: 56.0, ips: 291.77207 samples/s, eta: 14:58:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:02:42] ppocr INFO: epoch: [398/500], global_step: 266300, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.940895, exp_rate<=1 : 0.977228, exp_rate<=2 : 0.994508, exp_rate<=3 : 0.998041, loss: 0.000930, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19272 s, avg_samples: 56.0, ips: 290.58240 samples/s, eta: 14:58:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:03:00] ppocr INFO: epoch: [398/500], global_step: 266400, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.941201, exp_rate<=1 : 0.976615, exp_rate<=2 : 0.994229, exp_rate<=3 : 0.997955, loss: 0.001437, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18589 s, avg_samples: 56.0, ips: 301.24913 samples/s, eta: 14:57:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:03:19] ppocr INFO: epoch: [398/500], global_step: 266500, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.941736, exp_rate<=1 : 0.977100, exp_rate<=2 : 0.994495, exp_rate<=3 : 0.998089, loss: 0.001665, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18951 s, avg_samples: 56.0, ips: 295.49554 samples/s, eta: 14:57:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:03:38] ppocr INFO: epoch: [398/500], global_step: 266600, lr: 0.000100, edit distance: 0.001270, exp_rate: 0.941074, exp_rate<=1 : 0.977047, exp_rate<=2 : 0.994489, exp_rate<=3 : 0.998105, loss: 0.001799, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18509 s, avg_samples: 56.0, ips: 302.55934 samples/s, eta: 14:57:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:03:56] ppocr INFO: epoch: [398/500], global_step: 266700, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.941810, exp_rate<=1 : 0.977519, exp_rate<=2 : 0.994763, exp_rate<=3 : 0.998221, loss: 0.001366, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17574 s, avg_samples: 56.0, ips: 318.65455 samples/s, eta: 14:56:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:04:13] ppocr INFO: epoch: [398/500], global_step: 266800, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.942453, exp_rate<=1 : 0.977623, exp_rate<=2 : 0.994879, exp_rate<=3 : 0.998203, loss: 0.001108, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17314 s, avg_samples: 56.0, ips: 323.43924 samples/s, eta: 14:56:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:04:32] ppocr INFO: epoch: [398/500], global_step: 266900, lr: 0.000100, edit distance: 0.001270, exp_rate: 0.941844, exp_rate<=1 : 0.977391, exp_rate<=2 : 0.995002, exp_rate<=3 : 0.998264, loss: 0.001452, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19150 s, avg_samples: 56.0, ips: 292.42815 samples/s, eta: 14:56:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:04:51] ppocr INFO: epoch: [398/500], global_step: 267000, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.940238, exp_rate<=1 : 0.976725, exp_rate<=2 : 0.994770, exp_rate<=3 : 0.998166, loss: 0.001720, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19210 s, avg_samples: 56.0, ips: 291.51744 samples/s, eta: 14:56:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:05:10] ppocr INFO: epoch: [398/500], global_step: 267100, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.940035, exp_rate<=1 : 0.976616, exp_rate<=2 : 0.994709, exp_rate<=3 : 0.998111, loss: 0.001438, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18497 s, avg_samples: 56.0, ips: 302.75137 samples/s, eta: 14:55:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:05:29] ppocr INFO: epoch: [398/500], global_step: 267200, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.939917, exp_rate<=1 : 0.976545, exp_rate<=2 : 0.994604, exp_rate<=3 : 0.998041, loss: 0.001499, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19177 s, avg_samples: 56.0, ips: 292.00990 samples/s, eta: 14:55:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:05:47] ppocr INFO: epoch: [398/500], global_step: 267300, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.940241, exp_rate<=1 : 0.976681, exp_rate<=2 : 0.994634, exp_rate<=3 : 0.998048, loss: 0.001586, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18293 s, avg_samples: 56.0, ips: 306.12651 samples/s, eta: 14:55:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:06:07] ppocr INFO: epoch: [398/500], global_step: 267400, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.939853, exp_rate<=1 : 0.976463, exp_rate<=2 : 0.994655, exp_rate<=3 : 0.998027, loss: 0.001452, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19155 s, avg_samples: 56.0, ips: 292.35506 samples/s, eta: 14:54:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:06:26] ppocr INFO: epoch: [398/500], global_step: 267500, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.939301, exp_rate<=1 : 0.976190, exp_rate<=2 : 0.994660, exp_rate<=3 : 0.998032, loss: 0.001718, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19393 s, avg_samples: 56.0, ips: 288.76811 samples/s, eta: 14:54:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:06:46] ppocr INFO: epoch: [398/500], global_step: 267600, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.938876, exp_rate<=1 : 0.976072, exp_rate<=2 : 0.994589, exp_rate<=3 : 0.998005, loss: 0.001426, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19662 s, avg_samples: 56.0, ips: 284.80857 samples/s, eta: 14:54:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:07:04] ppocr INFO: epoch: [398/500], global_step: 267700, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.938291, exp_rate<=1 : 0.975734, exp_rate<=2 : 0.994564, exp_rate<=3 : 0.998018, loss: 0.001560, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18380 s, avg_samples: 56.0, ips: 304.68441 samples/s, eta: 14:53:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:07:23] ppocr INFO: epoch: [398/500], global_step: 267800, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.937789, exp_rate<=1 : 0.975684, exp_rate<=2 : 0.994457, exp_rate<=3 : 0.997984, loss: 0.001479, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19350 s, avg_samples: 56.0, ips: 289.40651 samples/s, eta: 14:53:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:07:42] ppocr INFO: epoch: [398/500], global_step: 267900, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.937683, exp_rate<=1 : 0.975627, exp_rate<=2 : 0.994428, exp_rate<=3 : 0.998010, loss: 0.001678, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18362 s, avg_samples: 56.0, ips: 304.98515 samples/s, eta: 14:53:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:08:01] ppocr INFO: epoch: [398/500], global_step: 268000, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.937520, exp_rate<=1 : 0.975485, exp_rate<=2 : 0.994494, exp_rate<=3 : 0.998053, loss: 0.001825, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18911 s, avg_samples: 56.0, ips: 296.11933 samples/s, eta: 14:52:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:08:18] ppocr INFO: epoch: [398/500], global_step: 268100, lr: 0.000100, edit distance: 0.001374, exp_rate: 0.937227, exp_rate<=1 : 0.975393, exp_rate<=2 : 0.994533, exp_rate<=3 : 0.998056, loss: 0.001516, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17276 s, avg_samples: 56.0, ips: 324.15226 samples/s, eta: 14:52:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:08:36] ppocr INFO: epoch: [398/500], global_step: 268200, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.937472, exp_rate<=1 : 0.975482, exp_rate<=2 : 0.994577, exp_rate<=3 : 0.998089, loss: 0.001481, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.17974 s, avg_samples: 56.0, ips: 311.56317 samples/s, eta: 14:52:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:08:55] ppocr INFO: epoch: [398/500], global_step: 268300, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.937262, exp_rate<=1 : 0.975445, exp_rate<=2 : 0.994605, exp_rate<=3 : 0.998083, loss: 0.001828, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18839 s, avg_samples: 56.0, ips: 297.25785 samples/s, eta: 14:51:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:09:15] ppocr INFO: epoch: [398/500], global_step: 268400, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.936808, exp_rate<=1 : 0.975280, exp_rate<=2 : 0.994599, exp_rate<=3 : 0.998069, loss: 0.001737, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20135 s, avg_samples: 56.0, ips: 278.11663 samples/s, eta: 14:51:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:09:34] ppocr INFO: epoch: [398/500], global_step: 268500, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.936466, exp_rate<=1 : 0.975227, exp_rate<=2 : 0.994620, exp_rate<=3 : 0.998076, loss: 0.002145, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18959 s, avg_samples: 56.0, ips: 295.37375 samples/s, eta: 14:51:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:09:37] ppocr INFO: epoch: [398/500], global_step: 268520, lr: 0.000100, edit distance: 0.001391, exp_rate: 0.936469, exp_rate<=1 : 0.975270, exp_rate<=2 : 0.994636, exp_rate<=3 : 0.998085, loss: 0.001033, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03298 s, avg_samples: 11.2, ips: 339.61192 samples/s, eta: 14:51:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:09:39] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 17:09:56] ppocr INFO: epoch: [399/500], global_step: 268600, lr: 0.000100, edit distance: 0.001586, exp_rate: 0.929590, exp_rate<=1 : 0.972498, exp_rate<=2 : 0.992347, exp_rate<=3 : 0.996982, loss: 0.002077, avg_reader_cost: 0.02507 s, avg_batch_cost: 0.18258 s, avg_samples: 44.8, ips: 245.37206 samples/s, eta: 14:50:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:10:14] ppocr INFO: epoch: [399/500], global_step: 268700, lr: 0.000100, edit distance: 0.001622, exp_rate: 0.937079, exp_rate<=1 : 0.976980, exp_rate<=2 : 0.995207, exp_rate<=3 : 0.998162, loss: 0.002123, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.18109 s, avg_samples: 56.0, ips: 309.24164 samples/s, eta: 14:50:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:10:32] ppocr INFO: epoch: [399/500], global_step: 268800, lr: 0.000100, edit distance: 0.001629, exp_rate: 0.935708, exp_rate<=1 : 0.975890, exp_rate<=2 : 0.994810, exp_rate<=3 : 0.997870, loss: 0.001818, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17904 s, avg_samples: 56.0, ips: 312.78439 samples/s, eta: 14:50:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:10:51] ppocr INFO: epoch: [399/500], global_step: 268900, lr: 0.000100, edit distance: 0.001556, exp_rate: 0.936401, exp_rate<=1 : 0.976062, exp_rate<=2 : 0.994457, exp_rate<=3 : 0.997611, loss: 0.001956, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19191 s, avg_samples: 56.0, ips: 291.79894 samples/s, eta: 14:49:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:11:10] ppocr INFO: epoch: [399/500], global_step: 269000, lr: 0.000100, edit distance: 0.001464, exp_rate: 0.938413, exp_rate<=1 : 0.976523, exp_rate<=2 : 0.994706, exp_rate<=3 : 0.997730, loss: 0.001981, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19589 s, avg_samples: 56.0, ips: 285.87871 samples/s, eta: 14:49:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:11:29] ppocr INFO: epoch: [399/500], global_step: 269100, lr: 0.000100, edit distance: 0.001389, exp_rate: 0.940168, exp_rate<=1 : 0.976916, exp_rate<=2 : 0.994272, exp_rate<=3 : 0.997684, loss: 0.001216, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18128 s, avg_samples: 56.0, ips: 308.91691 samples/s, eta: 14:49:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:11:46] ppocr INFO: epoch: [399/500], global_step: 269200, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.941136, exp_rate<=1 : 0.977534, exp_rate<=2 : 0.994620, exp_rate<=3 : 0.997869, loss: 0.001532, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17770 s, avg_samples: 56.0, ips: 315.14351 samples/s, eta: 14:48:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:12:05] ppocr INFO: epoch: [399/500], global_step: 269300, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.940968, exp_rate<=1 : 0.977397, exp_rate<=2 : 0.994739, exp_rate<=3 : 0.997937, loss: 0.001107, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18683 s, avg_samples: 56.0, ips: 299.74531 samples/s, eta: 14:48:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:12:25] ppocr INFO: epoch: [399/500], global_step: 269400, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.940734, exp_rate<=1 : 0.977093, exp_rate<=2 : 0.994649, exp_rate<=3 : 0.997746, loss: 0.001104, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19681 s, avg_samples: 56.0, ips: 284.53553 samples/s, eta: 14:48:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:12:44] ppocr INFO: epoch: [399/500], global_step: 269500, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.940338, exp_rate<=1 : 0.976595, exp_rate<=2 : 0.994393, exp_rate<=3 : 0.997597, loss: 0.001168, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18976 s, avg_samples: 56.0, ips: 295.10817 samples/s, eta: 14:47:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:13:04] ppocr INFO: epoch: [399/500], global_step: 269600, lr: 0.000100, edit distance: 0.001331, exp_rate: 0.939805, exp_rate<=1 : 0.976210, exp_rate<=2 : 0.994315, exp_rate<=3 : 0.997552, loss: 0.001560, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20550 s, avg_samples: 56.0, ips: 272.50193 samples/s, eta: 14:47:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:13:22] ppocr INFO: epoch: [399/500], global_step: 269700, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.939832, exp_rate<=1 : 0.976345, exp_rate<=2 : 0.994412, exp_rate<=3 : 0.997607, loss: 0.001182, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17541 s, avg_samples: 56.0, ips: 319.24384 samples/s, eta: 14:47:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:13:41] ppocr INFO: epoch: [399/500], global_step: 269800, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.939819, exp_rate<=1 : 0.976340, exp_rate<=2 : 0.994359, exp_rate<=3 : 0.997656, loss: 0.001478, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18669 s, avg_samples: 56.0, ips: 299.95842 samples/s, eta: 14:46:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:14:00] ppocr INFO: epoch: [399/500], global_step: 269900, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.940454, exp_rate<=1 : 0.976669, exp_rate<=2 : 0.994395, exp_rate<=3 : 0.997675, loss: 0.001382, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19041 s, avg_samples: 56.0, ips: 294.10025 samples/s, eta: 14:46:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:14:19] ppocr INFO: epoch: [399/500], global_step: 270000, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.940466, exp_rate<=1 : 0.976847, exp_rate<=2 : 0.994419, exp_rate<=3 : 0.997679, loss: 0.002033, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19646 s, avg_samples: 56.0, ips: 285.05104 samples/s, eta: 14:46:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:14:37] ppocr INFO: epoch: [399/500], global_step: 270100, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.940075, exp_rate<=1 : 0.976865, exp_rate<=2 : 0.994497, exp_rate<=3 : 0.997771, loss: 0.001908, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17911 s, avg_samples: 56.0, ips: 312.66407 samples/s, eta: 14:45:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:14:57] ppocr INFO: epoch: [399/500], global_step: 270200, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.939345, exp_rate<=1 : 0.976563, exp_rate<=2 : 0.994531, exp_rate<=3 : 0.997769, loss: 0.001811, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19614 s, avg_samples: 56.0, ips: 285.51495 samples/s, eta: 14:45:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:15:15] ppocr INFO: epoch: [399/500], global_step: 270300, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.939392, exp_rate<=1 : 0.976656, exp_rate<=2 : 0.994550, exp_rate<=3 : 0.997816, loss: 0.001542, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18402 s, avg_samples: 56.0, ips: 304.31564 samples/s, eta: 14:45:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:15:33] ppocr INFO: epoch: [399/500], global_step: 270400, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.939553, exp_rate<=1 : 0.976753, exp_rate<=2 : 0.994545, exp_rate<=3 : 0.997833, loss: 0.002019, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18145 s, avg_samples: 56.0, ips: 308.62825 samples/s, eta: 14:45:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:15:53] ppocr INFO: epoch: [399/500], global_step: 270500, lr: 0.000100, edit distance: 0.001359, exp_rate: 0.938706, exp_rate<=1 : 0.976607, exp_rate<=2 : 0.994565, exp_rate<=3 : 0.997846, loss: 0.001770, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19079 s, avg_samples: 56.0, ips: 293.51631 samples/s, eta: 14:44:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:16:11] ppocr INFO: epoch: [399/500], global_step: 270600, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.938845, exp_rate<=1 : 0.976761, exp_rate<=2 : 0.994627, exp_rate<=3 : 0.997863, loss: 0.001289, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18212 s, avg_samples: 56.0, ips: 307.49012 samples/s, eta: 14:44:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:16:29] ppocr INFO: epoch: [399/500], global_step: 270700, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.938717, exp_rate<=1 : 0.976701, exp_rate<=2 : 0.994602, exp_rate<=3 : 0.997858, loss: 0.001802, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18191 s, avg_samples: 56.0, ips: 307.84267 samples/s, eta: 14:44:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:16:46] ppocr INFO: epoch: [399/500], global_step: 270800, lr: 0.000100, edit distance: 0.001364, exp_rate: 0.938725, exp_rate<=1 : 0.976729, exp_rate<=2 : 0.994668, exp_rate<=3 : 0.997924, loss: 0.001495, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17342 s, avg_samples: 56.0, ips: 322.90787 samples/s, eta: 14:43:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:17:06] ppocr INFO: epoch: [399/500], global_step: 270900, lr: 0.000100, edit distance: 0.001384, exp_rate: 0.938212, exp_rate<=1 : 0.976477, exp_rate<=2 : 0.994576, exp_rate<=3 : 0.997876, loss: 0.001794, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19616 s, avg_samples: 56.0, ips: 285.47884 samples/s, eta: 14:43:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:17:25] ppocr INFO: epoch: [399/500], global_step: 271000, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.937930, exp_rate<=1 : 0.976319, exp_rate<=2 : 0.994465, exp_rate<=3 : 0.997853, loss: 0.000838, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18805 s, avg_samples: 56.0, ips: 297.78902 samples/s, eta: 14:43:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:17:44] ppocr INFO: epoch: [399/500], global_step: 271100, lr: 0.000100, edit distance: 0.001409, exp_rate: 0.937602, exp_rate<=1 : 0.976179, exp_rate<=2 : 0.994380, exp_rate<=3 : 0.997796, loss: 0.001940, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19004 s, avg_samples: 56.0, ips: 294.67914 samples/s, eta: 14:42:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:18:03] ppocr INFO: epoch: [399/500], global_step: 271200, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.937206, exp_rate<=1 : 0.975995, exp_rate<=2 : 0.994301, exp_rate<=3 : 0.997818, loss: 0.002008, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19018 s, avg_samples: 56.0, ips: 294.46385 samples/s, eta: 14:42:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:18:14] ppocr INFO: epoch: [399/500], global_step: 271260, lr: 0.000100, edit distance: 0.001408, exp_rate: 0.937172, exp_rate<=1 : 0.975947, exp_rate<=2 : 0.994289, exp_rate<=3 : 0.997787, loss: 0.001161, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11072 s, avg_samples: 33.6, ips: 303.47548 samples/s, eta: 14:42:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:18:16] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 17:18:25] ppocr INFO: epoch: [400/500], global_step: 271300, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.941149, exp_rate<=1 : 0.976872, exp_rate<=2 : 0.994667, exp_rate<=3 : 0.998161, loss: 0.002243, avg_reader_cost: 0.02781 s, avg_batch_cost: 0.10725 s, avg_samples: 22.4, ips: 208.85265 samples/s, eta: 14:42:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:18:44] ppocr INFO: epoch: [400/500], global_step: 271400, lr: 0.000100, edit distance: 0.001133, exp_rate: 0.945320, exp_rate<=1 : 0.978392, exp_rate<=2 : 0.994441, exp_rate<=3 : 0.997969, loss: 0.001746, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19131 s, avg_samples: 56.0, ips: 292.71464 samples/s, eta: 14:41:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:19:02] ppocr INFO: epoch: [400/500], global_step: 271500, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.944608, exp_rate<=1 : 0.978213, exp_rate<=2 : 0.994723, exp_rate<=3 : 0.998098, loss: 0.001086, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18254 s, avg_samples: 56.0, ips: 306.77519 samples/s, eta: 14:41:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:19:21] ppocr INFO: epoch: [400/500], global_step: 271600, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.943862, exp_rate<=1 : 0.977999, exp_rate<=2 : 0.994505, exp_rate<=3 : 0.998097, loss: 0.001518, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19290 s, avg_samples: 56.0, ips: 290.30234 samples/s, eta: 14:41:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:19:39] ppocr INFO: epoch: [400/500], global_step: 271700, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.943350, exp_rate<=1 : 0.978488, exp_rate<=2 : 0.994919, exp_rate<=3 : 0.998334, loss: 0.001841, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17484 s, avg_samples: 56.0, ips: 320.29582 samples/s, eta: 14:40:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:19:57] ppocr INFO: epoch: [400/500], global_step: 271800, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.941580, exp_rate<=1 : 0.978292, exp_rate<=2 : 0.994751, exp_rate<=3 : 0.998326, loss: 0.001526, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17773 s, avg_samples: 56.0, ips: 315.09154 samples/s, eta: 14:40:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:20:15] ppocr INFO: epoch: [400/500], global_step: 271900, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.940528, exp_rate<=1 : 0.977661, exp_rate<=2 : 0.994566, exp_rate<=3 : 0.998192, loss: 0.001219, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18688 s, avg_samples: 56.0, ips: 299.65659 samples/s, eta: 14:40:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:20:36] ppocr INFO: epoch: [400/500], global_step: 272000, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.939534, exp_rate<=1 : 0.977097, exp_rate<=2 : 0.994229, exp_rate<=3 : 0.997884, loss: 0.001604, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20694 s, avg_samples: 56.0, ips: 270.60496 samples/s, eta: 14:39:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:20:54] ppocr INFO: epoch: [400/500], global_step: 272100, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.940255, exp_rate<=1 : 0.977312, exp_rate<=2 : 0.994412, exp_rate<=3 : 0.997991, loss: 0.001417, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18203 s, avg_samples: 56.0, ips: 307.64571 samples/s, eta: 14:39:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:21:13] ppocr INFO: epoch: [400/500], global_step: 272200, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.941404, exp_rate<=1 : 0.977645, exp_rate<=2 : 0.994532, exp_rate<=3 : 0.998047, loss: 0.001718, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18103 s, avg_samples: 56.0, ips: 309.34460 samples/s, eta: 14:39:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:21:30] ppocr INFO: epoch: [400/500], global_step: 272300, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.941798, exp_rate<=1 : 0.977750, exp_rate<=2 : 0.994589, exp_rate<=3 : 0.998055, loss: 0.001541, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17579 s, avg_samples: 56.0, ips: 318.57015 samples/s, eta: 14:38:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:21:49] ppocr INFO: epoch: [400/500], global_step: 272400, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.941310, exp_rate<=1 : 0.977740, exp_rate<=2 : 0.994551, exp_rate<=3 : 0.998132, loss: 0.002177, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19150 s, avg_samples: 56.0, ips: 292.42940 samples/s, eta: 14:38:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:22:08] ppocr INFO: epoch: [400/500], global_step: 272500, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.941317, exp_rate<=1 : 0.977784, exp_rate<=2 : 0.994573, exp_rate<=3 : 0.998051, loss: 0.001131, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18393 s, avg_samples: 56.0, ips: 304.47085 samples/s, eta: 14:38:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:22:25] ppocr INFO: epoch: [400/500], global_step: 272600, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.941778, exp_rate<=1 : 0.978014, exp_rate<=2 : 0.994564, exp_rate<=3 : 0.997973, loss: 0.001209, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17619 s, avg_samples: 56.0, ips: 317.83560 samples/s, eta: 14:37:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:22:43] ppocr INFO: epoch: [400/500], global_step: 272700, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.942521, exp_rate<=1 : 0.978397, exp_rate<=2 : 0.994767, exp_rate<=3 : 0.998048, loss: 0.001792, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17808 s, avg_samples: 56.0, ips: 314.46164 samples/s, eta: 14:37:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:23:03] ppocr INFO: epoch: [400/500], global_step: 272800, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.941808, exp_rate<=1 : 0.978159, exp_rate<=2 : 0.994776, exp_rate<=3 : 0.998031, loss: 0.001474, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19968 s, avg_samples: 56.0, ips: 280.45145 samples/s, eta: 14:37:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:23:23] ppocr INFO: epoch: [400/500], global_step: 272900, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.941276, exp_rate<=1 : 0.977945, exp_rate<=2 : 0.994655, exp_rate<=3 : 0.997988, loss: 0.001361, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19895 s, avg_samples: 56.0, ips: 281.47338 samples/s, eta: 14:36:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:23:41] ppocr INFO: epoch: [400/500], global_step: 273000, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.941187, exp_rate<=1 : 0.978075, exp_rate<=2 : 0.994781, exp_rate<=3 : 0.998029, loss: 0.001589, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17743 s, avg_samples: 56.0, ips: 315.61447 samples/s, eta: 14:36:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:24:00] ppocr INFO: epoch: [400/500], global_step: 273100, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.940766, exp_rate<=1 : 0.977977, exp_rate<=2 : 0.994785, exp_rate<=3 : 0.997982, loss: 0.001552, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18820 s, avg_samples: 56.0, ips: 297.55850 samples/s, eta: 14:36:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:24:18] ppocr INFO: epoch: [400/500], global_step: 273200, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.940609, exp_rate<=1 : 0.977872, exp_rate<=2 : 0.994764, exp_rate<=3 : 0.997948, loss: 0.001486, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18236 s, avg_samples: 56.0, ips: 307.08099 samples/s, eta: 14:35:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:24:38] ppocr INFO: epoch: [400/500], global_step: 273300, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.940067, exp_rate<=1 : 0.977600, exp_rate<=2 : 0.994671, exp_rate<=3 : 0.997915, loss: 0.002197, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20092 s, avg_samples: 56.0, ips: 278.71565 samples/s, eta: 14:35:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:24:57] ppocr INFO: epoch: [400/500], global_step: 273400, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.939527, exp_rate<=1 : 0.977243, exp_rate<=2 : 0.994439, exp_rate<=3 : 0.997815, loss: 0.001547, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18492 s, avg_samples: 56.0, ips: 302.83394 samples/s, eta: 14:35:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:25:16] ppocr INFO: epoch: [400/500], global_step: 273500, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.938969, exp_rate<=1 : 0.977055, exp_rate<=2 : 0.994411, exp_rate<=3 : 0.997804, loss: 0.001312, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18983 s, avg_samples: 56.0, ips: 294.99963 samples/s, eta: 14:34:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:25:35] ppocr INFO: epoch: [400/500], global_step: 273600, lr: 0.000100, edit distance: 0.001308, exp_rate: 0.938920, exp_rate<=1 : 0.976983, exp_rate<=2 : 0.994408, exp_rate<=3 : 0.997810, loss: 0.001838, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19129 s, avg_samples: 56.0, ips: 292.75616 samples/s, eta: 14:34:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:25:54] ppocr INFO: epoch: [400/500], global_step: 273700, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.938345, exp_rate<=1 : 0.976888, exp_rate<=2 : 0.994329, exp_rate<=3 : 0.997803, loss: 0.001630, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19709 s, avg_samples: 56.0, ips: 284.12890 samples/s, eta: 14:34:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:26:14] ppocr INFO: epoch: [400/500], global_step: 273800, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.938331, exp_rate<=1 : 0.976820, exp_rate<=2 : 0.994368, exp_rate<=3 : 0.997824, loss: 0.002138, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19165 s, avg_samples: 56.0, ips: 292.20254 samples/s, eta: 14:33:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:26:34] ppocr INFO: epoch: [400/500], global_step: 273900, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.937793, exp_rate<=1 : 0.976640, exp_rate<=2 : 0.994338, exp_rate<=3 : 0.997830, loss: 0.001595, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19947 s, avg_samples: 56.0, ips: 280.74278 samples/s, eta: 14:33:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:26:52] ppocr INFO: epoch: [400/500], global_step: 274000, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.937945, exp_rate<=1 : 0.976708, exp_rate<=2 : 0.994374, exp_rate<=3 : 0.997835, loss: 0.001336, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18154 s, avg_samples: 56.0, ips: 308.47565 samples/s, eta: 14:33:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:26:54] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 17:26:55] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_400 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 17:27:14] ppocr INFO: epoch: [401/500], global_step: 274100, lr: 0.000100, edit distance: 0.001022, exp_rate: 0.949683, exp_rate<=1 : 0.980347, exp_rate<=2 : 0.995264, exp_rate<=3 : 0.997435, loss: 0.001013, avg_reader_cost: 0.03326 s, avg_batch_cost: 0.22029 s, avg_samples: 56.0, ips: 254.20911 samples/s, eta: 14:33:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:27:33] ppocr INFO: epoch: [401/500], global_step: 274200, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.946195, exp_rate<=1 : 0.978383, exp_rate<=2 : 0.995040, exp_rate<=3 : 0.997571, loss: 0.001642, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19083 s, avg_samples: 56.0, ips: 293.45830 samples/s, eta: 14:32:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:27:52] ppocr INFO: epoch: [401/500], global_step: 274300, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.944481, exp_rate<=1 : 0.977488, exp_rate<=2 : 0.994294, exp_rate<=3 : 0.997418, loss: 0.001052, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19397 s, avg_samples: 56.0, ips: 288.70739 samples/s, eta: 14:32:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:28:11] ppocr INFO: epoch: [401/500], global_step: 274400, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.945637, exp_rate<=1 : 0.978472, exp_rate<=2 : 0.994055, exp_rate<=3 : 0.997668, loss: 0.001420, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18779 s, avg_samples: 56.0, ips: 298.19972 samples/s, eta: 14:32:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:28:29] ppocr INFO: epoch: [401/500], global_step: 274500, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.945975, exp_rate<=1 : 0.979209, exp_rate<=2 : 0.994357, exp_rate<=3 : 0.997926, loss: 0.001543, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18330 s, avg_samples: 56.0, ips: 305.50389 samples/s, eta: 14:31:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:28:47] ppocr INFO: epoch: [401/500], global_step: 274600, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.946323, exp_rate<=1 : 0.979674, exp_rate<=2 : 0.994587, exp_rate<=3 : 0.998031, loss: 0.001482, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17020 s, avg_samples: 56.0, ips: 329.02797 samples/s, eta: 14:31:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:29:07] ppocr INFO: epoch: [401/500], global_step: 274700, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.945735, exp_rate<=1 : 0.979618, exp_rate<=2 : 0.994523, exp_rate<=3 : 0.997968, loss: 0.001231, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20082 s, avg_samples: 56.0, ips: 278.86183 samples/s, eta: 14:31:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:29:25] ppocr INFO: epoch: [401/500], global_step: 274800, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.944115, exp_rate<=1 : 0.979433, exp_rate<=2 : 0.994343, exp_rate<=3 : 0.998070, loss: 0.001940, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18852 s, avg_samples: 56.0, ips: 297.04532 samples/s, eta: 14:30:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:29:45] ppocr INFO: epoch: [401/500], global_step: 274900, lr: 0.000100, edit distance: 0.001161, exp_rate: 0.944310, exp_rate<=1 : 0.979561, exp_rate<=2 : 0.994546, exp_rate<=3 : 0.998191, loss: 0.001486, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19072 s, avg_samples: 56.0, ips: 293.63065 samples/s, eta: 14:30:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:30:02] ppocr INFO: epoch: [401/500], global_step: 275000, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.943798, exp_rate<=1 : 0.979229, exp_rate<=2 : 0.994566, exp_rate<=3 : 0.998215, loss: 0.001419, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17773 s, avg_samples: 56.0, ips: 315.08860 samples/s, eta: 14:30:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:30:22] ppocr INFO: epoch: [401/500], global_step: 275100, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.943585, exp_rate<=1 : 0.979152, exp_rate<=2 : 0.994671, exp_rate<=3 : 0.998259, loss: 0.001536, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19259 s, avg_samples: 56.0, ips: 290.77510 samples/s, eta: 14:29:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:30:41] ppocr INFO: epoch: [401/500], global_step: 275200, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.942604, exp_rate<=1 : 0.978747, exp_rate<=2 : 0.994540, exp_rate<=3 : 0.998194, loss: 0.001955, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19657 s, avg_samples: 56.0, ips: 284.88981 samples/s, eta: 14:29:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:31:01] ppocr INFO: epoch: [401/500], global_step: 275300, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.942459, exp_rate<=1 : 0.978711, exp_rate<=2 : 0.994578, exp_rate<=3 : 0.998221, loss: 0.001292, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19673 s, avg_samples: 56.0, ips: 284.65380 samples/s, eta: 14:29:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:31:19] ppocr INFO: epoch: [401/500], global_step: 275400, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942122, exp_rate<=1 : 0.978395, exp_rate<=2 : 0.994558, exp_rate<=3 : 0.998197, loss: 0.001397, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18149 s, avg_samples: 56.0, ips: 308.55420 samples/s, eta: 14:28:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:31:37] ppocr INFO: epoch: [401/500], global_step: 275500, lr: 0.000100, edit distance: 0.001214, exp_rate: 0.942235, exp_rate<=1 : 0.978450, exp_rate<=2 : 0.994655, exp_rate<=3 : 0.998174, loss: 0.001405, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18150 s, avg_samples: 56.0, ips: 308.53666 samples/s, eta: 14:28:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:31:56] ppocr INFO: epoch: [401/500], global_step: 275600, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.942173, exp_rate<=1 : 0.978311, exp_rate<=2 : 0.994646, exp_rate<=3 : 0.998171, loss: 0.001705, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18142 s, avg_samples: 56.0, ips: 308.67712 samples/s, eta: 14:28:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:32:15] ppocr INFO: epoch: [401/500], global_step: 275700, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.941679, exp_rate<=1 : 0.978121, exp_rate<=2 : 0.994592, exp_rate<=3 : 0.998151, loss: 0.001346, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19124 s, avg_samples: 56.0, ips: 292.81884 samples/s, eta: 14:27:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:32:35] ppocr INFO: epoch: [401/500], global_step: 275800, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.940813, exp_rate<=1 : 0.977455, exp_rate<=2 : 0.994356, exp_rate<=3 : 0.998074, loss: 0.001534, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20091 s, avg_samples: 56.0, ips: 278.73470 samples/s, eta: 14:27:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:32:53] ppocr INFO: epoch: [401/500], global_step: 275900, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.940733, exp_rate<=1 : 0.977438, exp_rate<=2 : 0.994388, exp_rate<=3 : 0.998039, loss: 0.001646, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18445 s, avg_samples: 56.0, ips: 303.60878 samples/s, eta: 14:27:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:33:11] ppocr INFO: epoch: [401/500], global_step: 276000, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.941028, exp_rate<=1 : 0.977620, exp_rate<=2 : 0.994456, exp_rate<=3 : 0.998088, loss: 0.001210, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17410 s, avg_samples: 56.0, ips: 321.66057 samples/s, eta: 14:26:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:33:30] ppocr INFO: epoch: [401/500], global_step: 276100, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.941060, exp_rate<=1 : 0.977645, exp_rate<=2 : 0.994516, exp_rate<=3 : 0.998085, loss: 0.001070, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18816 s, avg_samples: 56.0, ips: 297.61755 samples/s, eta: 14:26:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:33:49] ppocr INFO: epoch: [401/500], global_step: 276200, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.940546, exp_rate<=1 : 0.977413, exp_rate<=2 : 0.994450, exp_rate<=3 : 0.998060, loss: 0.001863, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19946 s, avg_samples: 56.0, ips: 280.76490 samples/s, eta: 14:26:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:34:08] ppocr INFO: epoch: [401/500], global_step: 276300, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.940558, exp_rate<=1 : 0.977217, exp_rate<=2 : 0.994365, exp_rate<=3 : 0.998029, loss: 0.001624, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18913 s, avg_samples: 56.0, ips: 296.09315 samples/s, eta: 14:25:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:34:27] ppocr INFO: epoch: [401/500], global_step: 276400, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.940303, exp_rate<=1 : 0.977178, exp_rate<=2 : 0.994379, exp_rate<=3 : 0.998036, loss: 0.001648, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18706 s, avg_samples: 56.0, ips: 299.36986 samples/s, eta: 14:25:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:34:47] ppocr INFO: epoch: [401/500], global_step: 276500, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.939969, exp_rate<=1 : 0.977081, exp_rate<=2 : 0.994375, exp_rate<=3 : 0.998041, loss: 0.001123, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19503 s, avg_samples: 56.0, ips: 287.14022 samples/s, eta: 14:25:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:35:05] ppocr INFO: epoch: [401/500], global_step: 276600, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.940025, exp_rate<=1 : 0.977076, exp_rate<=2 : 0.994373, exp_rate<=3 : 0.998041, loss: 0.001820, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18426 s, avg_samples: 56.0, ips: 303.91145 samples/s, eta: 14:24:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:35:22] ppocr INFO: epoch: [401/500], global_step: 276700, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.939962, exp_rate<=1 : 0.977002, exp_rate<=2 : 0.994392, exp_rate<=3 : 0.998040, loss: 0.001719, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16854 s, avg_samples: 56.0, ips: 332.26542 samples/s, eta: 14:24:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:35:30] ppocr INFO: epoch: [401/500], global_step: 276740, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.939825, exp_rate<=1 : 0.977003, exp_rate<=2 : 0.994408, exp_rate<=3 : 0.998043, loss: 0.001575, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.08222 s, avg_samples: 22.4, ips: 272.44757 samples/s, eta: 14:24:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:35:32] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 17:35:43] ppocr INFO: epoch: [402/500], global_step: 276800, lr: 0.000100, edit distance: 0.001096, exp_rate: 0.954113, exp_rate<=1 : 0.984807, exp_rate<=2 : 0.996606, exp_rate<=3 : 0.999646, loss: 0.001397, avg_reader_cost: 0.02496 s, avg_batch_cost: 0.12583 s, avg_samples: 33.6, ips: 267.02695 samples/s, eta: 14:24:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:36:00] ppocr INFO: epoch: [402/500], global_step: 276900, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.950107, exp_rate<=1 : 0.983151, exp_rate<=2 : 0.996331, exp_rate<=3 : 0.998932, loss: 0.001264, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17661 s, avg_samples: 56.0, ips: 317.07891 samples/s, eta: 14:24:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:36:18] ppocr INFO: epoch: [402/500], global_step: 277000, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.946465, exp_rate<=1 : 0.981775, exp_rate<=2 : 0.995771, exp_rate<=3 : 0.998737, loss: 0.001671, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17476 s, avg_samples: 56.0, ips: 320.43285 samples/s, eta: 14:23:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:36:36] ppocr INFO: epoch: [402/500], global_step: 277100, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.944959, exp_rate<=1 : 0.980258, exp_rate<=2 : 0.995776, exp_rate<=3 : 0.998750, loss: 0.001321, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18311 s, avg_samples: 56.0, ips: 305.83188 samples/s, eta: 14:23:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:36:56] ppocr INFO: epoch: [402/500], global_step: 277200, lr: 0.000100, edit distance: 0.001385, exp_rate: 0.943001, exp_rate<=1 : 0.978911, exp_rate<=2 : 0.994966, exp_rate<=3 : 0.998062, loss: 0.001831, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19731 s, avg_samples: 56.0, ips: 283.81494 samples/s, eta: 14:23:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:37:16] ppocr INFO: epoch: [402/500], global_step: 277300, lr: 0.000100, edit distance: 0.001394, exp_rate: 0.942199, exp_rate<=1 : 0.978865, exp_rate<=2 : 0.994833, exp_rate<=3 : 0.997892, loss: 0.001144, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20284 s, avg_samples: 56.0, ips: 276.08482 samples/s, eta: 14:22:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:37:36] ppocr INFO: epoch: [402/500], global_step: 277400, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.941858, exp_rate<=1 : 0.978298, exp_rate<=2 : 0.994732, exp_rate<=3 : 0.997771, loss: 0.001264, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19636 s, avg_samples: 56.0, ips: 285.19350 samples/s, eta: 14:22:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:37:54] ppocr INFO: epoch: [402/500], global_step: 277500, lr: 0.000100, edit distance: 0.001387, exp_rate: 0.941848, exp_rate<=1 : 0.978681, exp_rate<=2 : 0.994930, exp_rate<=3 : 0.997906, loss: 0.001651, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18246 s, avg_samples: 56.0, ips: 306.91398 samples/s, eta: 14:22:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:38:15] ppocr INFO: epoch: [402/500], global_step: 277600, lr: 0.000100, edit distance: 0.001358, exp_rate: 0.941370, exp_rate<=1 : 0.978362, exp_rate<=2 : 0.994844, exp_rate<=3 : 0.997941, loss: 0.001213, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20458 s, avg_samples: 56.0, ips: 273.73388 samples/s, eta: 14:21:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:38:33] ppocr INFO: epoch: [402/500], global_step: 277700, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.941589, exp_rate<=1 : 0.978493, exp_rate<=2 : 0.994918, exp_rate<=3 : 0.998000, loss: 0.001407, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18350 s, avg_samples: 56.0, ips: 305.17936 samples/s, eta: 14:21:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:38:52] ppocr INFO: epoch: [402/500], global_step: 277800, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.940344, exp_rate<=1 : 0.978016, exp_rate<=2 : 0.994910, exp_rate<=3 : 0.997948, loss: 0.000978, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18638 s, avg_samples: 56.0, ips: 300.46816 samples/s, eta: 14:21:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:39:10] ppocr INFO: epoch: [402/500], global_step: 277900, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.940754, exp_rate<=1 : 0.978075, exp_rate<=2 : 0.994950, exp_rate<=3 : 0.997951, loss: 0.001578, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17959 s, avg_samples: 56.0, ips: 311.81792 samples/s, eta: 14:20:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:39:31] ppocr INFO: epoch: [402/500], global_step: 278000, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.938550, exp_rate<=1 : 0.976960, exp_rate<=2 : 0.994634, exp_rate<=3 : 0.997813, loss: 0.001510, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21236 s, avg_samples: 56.0, ips: 263.69724 samples/s, eta: 14:20:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:39:50] ppocr INFO: epoch: [402/500], global_step: 278100, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.938582, exp_rate<=1 : 0.976774, exp_rate<=2 : 0.994618, exp_rate<=3 : 0.997805, loss: 0.001245, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19083 s, avg_samples: 56.0, ips: 293.45516 samples/s, eta: 14:20:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:40:09] ppocr INFO: epoch: [402/500], global_step: 278200, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.938049, exp_rate<=1 : 0.976760, exp_rate<=2 : 0.994634, exp_rate<=3 : 0.997782, loss: 0.001833, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18570 s, avg_samples: 56.0, ips: 301.55818 samples/s, eta: 14:19:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:40:28] ppocr INFO: epoch: [402/500], global_step: 278300, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.937322, exp_rate<=1 : 0.976482, exp_rate<=2 : 0.994544, exp_rate<=3 : 0.997810, loss: 0.002145, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.74967 samples/s, eta: 14:19:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:40:47] ppocr INFO: epoch: [402/500], global_step: 278400, lr: 0.000100, edit distance: 0.001383, exp_rate: 0.936595, exp_rate<=1 : 0.976317, exp_rate<=2 : 0.994450, exp_rate<=3 : 0.997844, loss: 0.002121, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18531 s, avg_samples: 56.0, ips: 302.20406 samples/s, eta: 14:19:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:41:05] ppocr INFO: epoch: [402/500], global_step: 278500, lr: 0.000100, edit distance: 0.001405, exp_rate: 0.936203, exp_rate<=1 : 0.976087, exp_rate<=2 : 0.994458, exp_rate<=3 : 0.997854, loss: 0.002077, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18791 s, avg_samples: 56.0, ips: 298.01442 samples/s, eta: 14:18:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:41:23] ppocr INFO: epoch: [402/500], global_step: 278600, lr: 0.000100, edit distance: 0.001425, exp_rate: 0.936411, exp_rate<=1 : 0.976194, exp_rate<=2 : 0.994552, exp_rate<=3 : 0.997917, loss: 0.001463, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17396 s, avg_samples: 56.0, ips: 321.92113 samples/s, eta: 14:18:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:41:40] ppocr INFO: epoch: [402/500], global_step: 278700, lr: 0.000100, edit distance: 0.001448, exp_rate: 0.936383, exp_rate<=1 : 0.976239, exp_rate<=2 : 0.994577, exp_rate<=3 : 0.997925, loss: 0.001816, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17606 s, avg_samples: 56.0, ips: 318.07888 samples/s, eta: 14:18:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:41:59] ppocr INFO: epoch: [402/500], global_step: 278800, lr: 0.000100, edit distance: 0.001442, exp_rate: 0.936797, exp_rate<=1 : 0.976517, exp_rate<=2 : 0.994631, exp_rate<=3 : 0.997952, loss: 0.001367, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18968 s, avg_samples: 56.0, ips: 295.23298 samples/s, eta: 14:17:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:42:18] ppocr INFO: epoch: [402/500], global_step: 278900, lr: 0.000100, edit distance: 0.001447, exp_rate: 0.936726, exp_rate<=1 : 0.976567, exp_rate<=2 : 0.994670, exp_rate<=3 : 0.997966, loss: 0.001727, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18717 s, avg_samples: 56.0, ips: 299.19523 samples/s, eta: 14:17:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:42:38] ppocr INFO: epoch: [402/500], global_step: 279000, lr: 0.000100, edit distance: 0.001440, exp_rate: 0.936535, exp_rate<=1 : 0.976478, exp_rate<=2 : 0.994703, exp_rate<=3 : 0.997977, loss: 0.001017, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19954 s, avg_samples: 56.0, ips: 280.64775 samples/s, eta: 14:17:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:42:56] ppocr INFO: epoch: [402/500], global_step: 279100, lr: 0.000100, edit distance: 0.001435, exp_rate: 0.936804, exp_rate<=1 : 0.976531, exp_rate<=2 : 0.994656, exp_rate<=3 : 0.997958, loss: 0.001916, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17425 s, avg_samples: 56.0, ips: 321.37002 samples/s, eta: 14:16:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:43:14] ppocr INFO: epoch: [402/500], global_step: 279200, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.936429, exp_rate<=1 : 0.976403, exp_rate<=2 : 0.994582, exp_rate<=3 : 0.997920, loss: 0.001465, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18915 s, avg_samples: 56.0, ips: 296.06899 samples/s, eta: 14:16:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:43:34] ppocr INFO: epoch: [402/500], global_step: 279300, lr: 0.000100, edit distance: 0.001452, exp_rate: 0.935747, exp_rate<=1 : 0.976174, exp_rate<=2 : 0.994550, exp_rate<=3 : 0.997914, loss: 0.001555, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19813 s, avg_samples: 56.0, ips: 282.64684 samples/s, eta: 14:16:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:43:53] ppocr INFO: epoch: [402/500], global_step: 279400, lr: 0.000100, edit distance: 0.001445, exp_rate: 0.935984, exp_rate<=1 : 0.976208, exp_rate<=2 : 0.994563, exp_rate<=3 : 0.997938, loss: 0.001734, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18440 s, avg_samples: 56.0, ips: 303.68511 samples/s, eta: 14:15:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:44:08] ppocr INFO: epoch: [402/500], global_step: 279480, lr: 0.000100, edit distance: 0.001447, exp_rate: 0.935846, exp_rate<=1 : 0.976123, exp_rate<=2 : 0.994548, exp_rate<=3 : 0.997949, loss: 0.001338, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15547 s, avg_samples: 44.8, ips: 288.15608 samples/s, eta: 14:15:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:44:10] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 17:44:14] ppocr INFO: epoch: [403/500], global_step: 279500, lr: 0.000100, edit distance: 0.000373, exp_rate: 0.977476, exp_rate<=1 : 0.991477, exp_rate<=2 : 1.000000, exp_rate<=3 : 1.000000, loss: 0.000645, avg_reader_cost: 0.02512 s, avg_batch_cost: 0.05945 s, avg_samples: 11.2, ips: 188.39149 samples/s, eta: 14:15:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:44:32] ppocr INFO: epoch: [403/500], global_step: 279600, lr: 0.000100, edit distance: 0.000988, exp_rate: 0.956813, exp_rate<=1 : 0.986165, exp_rate<=2 : 0.996986, exp_rate<=3 : 0.999303, loss: 0.001120, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17628 s, avg_samples: 56.0, ips: 317.66927 samples/s, eta: 14:15:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:44:50] ppocr INFO: epoch: [403/500], global_step: 279700, lr: 0.000100, edit distance: 0.001027, exp_rate: 0.955768, exp_rate<=1 : 0.984354, exp_rate<=2 : 0.996974, exp_rate<=3 : 0.999144, loss: 0.001218, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17693 s, avg_samples: 56.0, ips: 316.50421 samples/s, eta: 14:14:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:45:09] ppocr INFO: epoch: [403/500], global_step: 279800, lr: 0.000100, edit distance: 0.001000, exp_rate: 0.953690, exp_rate<=1 : 0.983233, exp_rate<=2 : 0.996032, exp_rate<=3 : 0.998804, loss: 0.000742, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19141 s, avg_samples: 56.0, ips: 292.56338 samples/s, eta: 14:14:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:45:27] ppocr INFO: epoch: [403/500], global_step: 279900, lr: 0.000100, edit distance: 0.001017, exp_rate: 0.952728, exp_rate<=1 : 0.982578, exp_rate<=2 : 0.995782, exp_rate<=3 : 0.998872, loss: 0.002071, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17733 s, avg_samples: 56.0, ips: 315.78946 samples/s, eta: 14:14:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:45:45] ppocr INFO: epoch: [403/500], global_step: 280000, lr: 0.000100, edit distance: 0.001058, exp_rate: 0.950566, exp_rate<=1 : 0.981762, exp_rate<=2 : 0.995733, exp_rate<=3 : 0.998700, loss: 0.001367, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18415 s, avg_samples: 56.0, ips: 304.09213 samples/s, eta: 14:13:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:46:03] ppocr INFO: epoch: [403/500], global_step: 280100, lr: 0.000100, edit distance: 0.001081, exp_rate: 0.949455, exp_rate<=1 : 0.981509, exp_rate<=2 : 0.995809, exp_rate<=3 : 0.998580, loss: 0.001470, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18444 s, avg_samples: 56.0, ips: 303.62527 samples/s, eta: 14:13:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:46:22] ppocr INFO: epoch: [403/500], global_step: 280200, lr: 0.000100, edit distance: 0.001110, exp_rate: 0.948404, exp_rate<=1 : 0.980774, exp_rate<=2 : 0.995520, exp_rate<=3 : 0.998389, loss: 0.001299, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18991 s, avg_samples: 56.0, ips: 294.87491 samples/s, eta: 14:13:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:46:40] ppocr INFO: epoch: [403/500], global_step: 280300, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.947683, exp_rate<=1 : 0.980254, exp_rate<=2 : 0.995326, exp_rate<=3 : 0.998293, loss: 0.001229, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17576 s, avg_samples: 56.0, ips: 318.61931 samples/s, eta: 14:12:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:46:59] ppocr INFO: epoch: [403/500], global_step: 280400, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.946174, exp_rate<=1 : 0.979554, exp_rate<=2 : 0.995104, exp_rate<=3 : 0.998239, loss: 0.002332, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18864 s, avg_samples: 56.0, ips: 296.85590 samples/s, eta: 14:12:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:47:18] ppocr INFO: epoch: [403/500], global_step: 280500, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.945310, exp_rate<=1 : 0.979239, exp_rate<=2 : 0.994925, exp_rate<=3 : 0.998249, loss: 0.001487, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18726 s, avg_samples: 56.0, ips: 299.05265 samples/s, eta: 14:12:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:47:37] ppocr INFO: epoch: [403/500], global_step: 280600, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.944494, exp_rate<=1 : 0.978698, exp_rate<=2 : 0.994735, exp_rate<=3 : 0.998192, loss: 0.001305, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18847 s, avg_samples: 56.0, ips: 297.12385 samples/s, eta: 14:12:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:47:59] ppocr INFO: epoch: [403/500], global_step: 280700, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.943264, exp_rate<=1 : 0.978285, exp_rate<=2 : 0.994472, exp_rate<=3 : 0.998083, loss: 0.001633, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.22007 s, avg_samples: 56.0, ips: 254.46702 samples/s, eta: 14:11:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:48:16] ppocr INFO: epoch: [403/500], global_step: 280800, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.942785, exp_rate<=1 : 0.978081, exp_rate<=2 : 0.994372, exp_rate<=3 : 0.997970, loss: 0.001314, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17110 s, avg_samples: 56.0, ips: 327.30307 samples/s, eta: 14:11:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:48:35] ppocr INFO: epoch: [403/500], global_step: 280900, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.941952, exp_rate<=1 : 0.977734, exp_rate<=2 : 0.994370, exp_rate<=3 : 0.997951, loss: 0.001943, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19196 s, avg_samples: 56.0, ips: 291.73147 samples/s, eta: 14:11:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:48:53] ppocr INFO: epoch: [403/500], global_step: 281000, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.941738, exp_rate<=1 : 0.977875, exp_rate<=2 : 0.994468, exp_rate<=3 : 0.997987, loss: 0.001238, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18122 s, avg_samples: 56.0, ips: 309.01738 samples/s, eta: 14:10:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:49:13] ppocr INFO: epoch: [403/500], global_step: 281100, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.941392, exp_rate<=1 : 0.977707, exp_rate<=2 : 0.994439, exp_rate<=3 : 0.998010, loss: 0.002063, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19588 s, avg_samples: 56.0, ips: 285.89520 samples/s, eta: 14:10:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:49:32] ppocr INFO: epoch: [403/500], global_step: 281200, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.940461, exp_rate<=1 : 0.977245, exp_rate<=2 : 0.994360, exp_rate<=3 : 0.997994, loss: 0.001857, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19418 s, avg_samples: 56.0, ips: 288.39152 samples/s, eta: 14:10:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:49:51] ppocr INFO: epoch: [403/500], global_step: 281300, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.940312, exp_rate<=1 : 0.977039, exp_rate<=2 : 0.994343, exp_rate<=3 : 0.997978, loss: 0.000984, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18354 s, avg_samples: 56.0, ips: 305.11736 samples/s, eta: 14:09:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:50:09] ppocr INFO: epoch: [403/500], global_step: 281400, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.939406, exp_rate<=1 : 0.976686, exp_rate<=2 : 0.994346, exp_rate<=3 : 0.997947, loss: 0.001784, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18722 s, avg_samples: 56.0, ips: 299.11264 samples/s, eta: 14:09:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:50:29] ppocr INFO: epoch: [403/500], global_step: 281500, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.938967, exp_rate<=1 : 0.976377, exp_rate<=2 : 0.994330, exp_rate<=3 : 0.997927, loss: 0.001754, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19479 s, avg_samples: 56.0, ips: 287.48283 samples/s, eta: 14:09:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:50:49] ppocr INFO: epoch: [403/500], global_step: 281600, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.938669, exp_rate<=1 : 0.976272, exp_rate<=2 : 0.994324, exp_rate<=3 : 0.997952, loss: 0.001789, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20017 s, avg_samples: 56.0, ips: 279.76446 samples/s, eta: 14:08:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:51:07] ppocr INFO: epoch: [403/500], global_step: 281700, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.938865, exp_rate<=1 : 0.976321, exp_rate<=2 : 0.994412, exp_rate<=3 : 0.998013, loss: 0.001168, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18356 s, avg_samples: 56.0, ips: 305.08283 samples/s, eta: 14:08:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:51:27] ppocr INFO: epoch: [403/500], global_step: 281800, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.938979, exp_rate<=1 : 0.976278, exp_rate<=2 : 0.994423, exp_rate<=3 : 0.997993, loss: 0.001499, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19755 s, avg_samples: 56.0, ips: 283.47646 samples/s, eta: 14:08:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:51:47] ppocr INFO: epoch: [403/500], global_step: 281900, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.939009, exp_rate<=1 : 0.976295, exp_rate<=2 : 0.994488, exp_rate<=3 : 0.998015, loss: 0.001443, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19610 s, avg_samples: 56.0, ips: 285.57448 samples/s, eta: 14:07:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:52:05] ppocr INFO: epoch: [403/500], global_step: 282000, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.938753, exp_rate<=1 : 0.976082, exp_rate<=2 : 0.994435, exp_rate<=3 : 0.997989, loss: 0.001126, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18476 s, avg_samples: 56.0, ips: 303.09613 samples/s, eta: 14:07:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:52:24] ppocr INFO: epoch: [403/500], global_step: 282100, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.938362, exp_rate<=1 : 0.975918, exp_rate<=2 : 0.994334, exp_rate<=3 : 0.997936, loss: 0.002049, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18961 s, avg_samples: 56.0, ips: 295.34421 samples/s, eta: 14:07:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:52:42] ppocr INFO: epoch: [403/500], global_step: 282200, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.937821, exp_rate<=1 : 0.975607, exp_rate<=2 : 0.994230, exp_rate<=3 : 0.997881, loss: 0.001704, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18370 s, avg_samples: 56.0, ips: 304.84811 samples/s, eta: 14:06:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:52:46] ppocr INFO: epoch: [403/500], global_step: 282220, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.937672, exp_rate<=1 : 0.975551, exp_rate<=2 : 0.994202, exp_rate<=3 : 0.997850, loss: 0.002801, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03947 s, avg_samples: 11.2, ips: 283.78860 samples/s, eta: 14:06:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:52:49] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 17:53:04] ppocr INFO: epoch: [404/500], global_step: 282300, lr: 0.000100, edit distance: 0.001598, exp_rate: 0.939718, exp_rate<=1 : 0.974857, exp_rate<=2 : 0.995807, exp_rate<=3 : 0.999156, loss: 0.001635, avg_reader_cost: 0.02773 s, avg_batch_cost: 0.17706 s, avg_samples: 44.8, ips: 253.02600 samples/s, eta: 14:06:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:53:22] ppocr INFO: epoch: [404/500], global_step: 282400, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.947319, exp_rate<=1 : 0.977744, exp_rate<=2 : 0.995599, exp_rate<=3 : 0.998516, loss: 0.000669, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18084 s, avg_samples: 56.0, ips: 309.66548 samples/s, eta: 14:06:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:53:41] ppocr INFO: epoch: [404/500], global_step: 282500, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.947604, exp_rate<=1 : 0.979019, exp_rate<=2 : 0.995495, exp_rate<=3 : 0.998718, loss: 0.001206, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18993 s, avg_samples: 56.0, ips: 294.84291 samples/s, eta: 14:05:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:54:00] ppocr INFO: epoch: [404/500], global_step: 282600, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.945967, exp_rate<=1 : 0.978248, exp_rate<=2 : 0.995048, exp_rate<=3 : 0.998526, loss: 0.001409, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18960 s, avg_samples: 56.0, ips: 295.36407 samples/s, eta: 14:05:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:54:20] ppocr INFO: epoch: [404/500], global_step: 282700, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.944185, exp_rate<=1 : 0.977071, exp_rate<=2 : 0.994484, exp_rate<=3 : 0.998433, loss: 0.001133, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19559 s, avg_samples: 56.0, ips: 286.31063 samples/s, eta: 14:05:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:54:39] ppocr INFO: epoch: [404/500], global_step: 282800, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.944777, exp_rate<=1 : 0.977414, exp_rate<=2 : 0.994663, exp_rate<=3 : 0.998416, loss: 0.001300, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18763 s, avg_samples: 56.0, ips: 298.45787 samples/s, eta: 14:05:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:54:57] ppocr INFO: epoch: [404/500], global_step: 282900, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.944296, exp_rate<=1 : 0.977586, exp_rate<=2 : 0.994826, exp_rate<=3 : 0.998397, loss: 0.001408, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18872 s, avg_samples: 56.0, ips: 296.73736 samples/s, eta: 14:04:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:55:16] ppocr INFO: epoch: [404/500], global_step: 283000, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.944838, exp_rate<=1 : 0.977915, exp_rate<=2 : 0.994651, exp_rate<=3 : 0.998257, loss: 0.001326, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18698 s, avg_samples: 56.0, ips: 299.49318 samples/s, eta: 14:04:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:55:34] ppocr INFO: epoch: [404/500], global_step: 283100, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.944704, exp_rate<=1 : 0.977869, exp_rate<=2 : 0.994735, exp_rate<=3 : 0.998215, loss: 0.000918, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17896 s, avg_samples: 56.0, ips: 312.91580 samples/s, eta: 14:04:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:55:53] ppocr INFO: epoch: [404/500], global_step: 283200, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.945306, exp_rate<=1 : 0.978129, exp_rate<=2 : 0.994837, exp_rate<=3 : 0.998307, loss: 0.001532, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18522 s, avg_samples: 56.0, ips: 302.34268 samples/s, eta: 14:03:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:56:13] ppocr INFO: epoch: [404/500], global_step: 283300, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.944013, exp_rate<=1 : 0.977248, exp_rate<=2 : 0.994421, exp_rate<=3 : 0.998046, loss: 0.001579, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20201 s, avg_samples: 56.0, ips: 277.21267 samples/s, eta: 14:03:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:56:32] ppocr INFO: epoch: [404/500], global_step: 283400, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.942795, exp_rate<=1 : 0.976818, exp_rate<=2 : 0.994355, exp_rate<=3 : 0.997971, loss: 0.001514, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19562 s, avg_samples: 56.0, ips: 286.26313 samples/s, eta: 14:03:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:56:51] ppocr INFO: epoch: [404/500], global_step: 283500, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.942874, exp_rate<=1 : 0.976771, exp_rate<=2 : 0.994541, exp_rate<=3 : 0.998071, loss: 0.001837, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18241 s, avg_samples: 56.0, ips: 307.00248 samples/s, eta: 14:02:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:57:10] ppocr INFO: epoch: [404/500], global_step: 283600, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.941845, exp_rate<=1 : 0.976530, exp_rate<=2 : 0.994478, exp_rate<=3 : 0.998070, loss: 0.002097, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19047 s, avg_samples: 56.0, ips: 294.01507 samples/s, eta: 14:02:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:57:28] ppocr INFO: epoch: [404/500], global_step: 283700, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.941234, exp_rate<=1 : 0.976540, exp_rate<=2 : 0.994589, exp_rate<=3 : 0.998139, loss: 0.002197, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18497 s, avg_samples: 56.0, ips: 302.75133 samples/s, eta: 14:02:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:57:47] ppocr INFO: epoch: [404/500], global_step: 283800, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.940827, exp_rate<=1 : 0.976557, exp_rate<=2 : 0.994651, exp_rate<=3 : 0.998166, loss: 0.002144, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18532 s, avg_samples: 56.0, ips: 302.17887 samples/s, eta: 14:01:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:58:06] ppocr INFO: epoch: [404/500], global_step: 283900, lr: 0.000100, edit distance: 0.001308, exp_rate: 0.940441, exp_rate<=1 : 0.976293, exp_rate<=2 : 0.994634, exp_rate<=3 : 0.998145, loss: 0.001446, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19616 s, avg_samples: 56.0, ips: 285.47549 samples/s, eta: 14:01:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:58:24] ppocr INFO: epoch: [404/500], global_step: 284000, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.940604, exp_rate<=1 : 0.976501, exp_rate<=2 : 0.994655, exp_rate<=3 : 0.998161, loss: 0.001553, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17553 s, avg_samples: 56.0, ips: 319.02625 samples/s, eta: 14:01:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:58:42] ppocr INFO: epoch: [404/500], global_step: 284100, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.940772, exp_rate<=1 : 0.976722, exp_rate<=2 : 0.994678, exp_rate<=3 : 0.998167, loss: 0.001274, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17903 s, avg_samples: 56.0, ips: 312.80440 samples/s, eta: 14:00:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:59:00] ppocr INFO: epoch: [404/500], global_step: 284200, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.940268, exp_rate<=1 : 0.976537, exp_rate<=2 : 0.994654, exp_rate<=3 : 0.998113, loss: 0.001904, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18295 s, avg_samples: 56.0, ips: 306.10056 samples/s, eta: 14:00:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:59:18] ppocr INFO: epoch: [404/500], global_step: 284300, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.940352, exp_rate<=1 : 0.976710, exp_rate<=2 : 0.994707, exp_rate<=3 : 0.998139, loss: 0.001242, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17961 s, avg_samples: 56.0, ips: 311.78695 samples/s, eta: 14:00:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:59:36] ppocr INFO: epoch: [404/500], global_step: 284400, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.940753, exp_rate<=1 : 0.976899, exp_rate<=2 : 0.994741, exp_rate<=3 : 0.998163, loss: 0.001737, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17582 s, avg_samples: 56.0, ips: 318.50112 samples/s, eta: 13:59:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 17:59:55] ppocr INFO: epoch: [404/500], global_step: 284500, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.940572, exp_rate<=1 : 0.976880, exp_rate<=2 : 0.994733, exp_rate<=3 : 0.998167, loss: 0.001436, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18831 s, avg_samples: 56.0, ips: 297.37948 samples/s, eta: 13:59:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:00:13] ppocr INFO: epoch: [404/500], global_step: 284600, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.940445, exp_rate<=1 : 0.976776, exp_rate<=2 : 0.994684, exp_rate<=3 : 0.998154, loss: 0.001463, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18543 s, avg_samples: 56.0, ips: 302.00019 samples/s, eta: 13:59:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:00:32] ppocr INFO: epoch: [404/500], global_step: 284700, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.940406, exp_rate<=1 : 0.976797, exp_rate<=2 : 0.994683, exp_rate<=3 : 0.998163, loss: 0.001815, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18432 s, avg_samples: 56.0, ips: 303.81456 samples/s, eta: 13:58:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:00:50] ppocr INFO: epoch: [404/500], global_step: 284800, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.940355, exp_rate<=1 : 0.976860, exp_rate<=2 : 0.994713, exp_rate<=3 : 0.998168, loss: 0.002117, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18036 s, avg_samples: 56.0, ips: 310.48243 samples/s, eta: 13:58:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:01:10] ppocr INFO: epoch: [404/500], global_step: 284900, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.939710, exp_rate<=1 : 0.976486, exp_rate<=2 : 0.994649, exp_rate<=3 : 0.998177, loss: 0.001832, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19924 s, avg_samples: 56.0, ips: 281.06703 samples/s, eta: 13:58:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:01:22] ppocr INFO: epoch: [404/500], global_step: 284960, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.939509, exp_rate<=1 : 0.976274, exp_rate<=2 : 0.994589, exp_rate<=3 : 0.998160, loss: 0.001949, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11959 s, avg_samples: 33.6, ips: 280.96844 samples/s, eta: 13:57:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:01:24] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 18:01:31] ppocr INFO: epoch: [405/500], global_step: 285000, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.946786, exp_rate<=1 : 0.971129, exp_rate<=2 : 0.995603, exp_rate<=3 : 0.997802, loss: 0.001369, avg_reader_cost: 0.02467 s, avg_batch_cost: 0.09703 s, avg_samples: 22.4, ips: 230.84990 samples/s, eta: 13:57:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:01:50] ppocr INFO: epoch: [405/500], global_step: 285100, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.946081, exp_rate<=1 : 0.978016, exp_rate<=2 : 0.996003, exp_rate<=3 : 0.998420, loss: 0.000851, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19020 s, avg_samples: 56.0, ips: 294.43099 samples/s, eta: 13:57:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:02:10] ppocr INFO: epoch: [405/500], global_step: 285200, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.945800, exp_rate<=1 : 0.978205, exp_rate<=2 : 0.995736, exp_rate<=3 : 0.998483, loss: 0.001215, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19294 s, avg_samples: 56.0, ips: 290.23896 samples/s, eta: 13:57:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:02:28] ppocr INFO: epoch: [405/500], global_step: 285300, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.945888, exp_rate<=1 : 0.978248, exp_rate<=2 : 0.995604, exp_rate<=3 : 0.998204, loss: 0.001589, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18103 s, avg_samples: 56.0, ips: 309.34405 samples/s, eta: 13:56:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:02:46] ppocr INFO: epoch: [405/500], global_step: 285400, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.944739, exp_rate<=1 : 0.978046, exp_rate<=2 : 0.995305, exp_rate<=3 : 0.998021, loss: 0.001419, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18332 s, avg_samples: 56.0, ips: 305.48100 samples/s, eta: 13:56:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:03:05] ppocr INFO: epoch: [405/500], global_step: 285500, lr: 0.000100, edit distance: 0.001183, exp_rate: 0.945424, exp_rate<=1 : 0.978415, exp_rate<=2 : 0.995234, exp_rate<=3 : 0.998160, loss: 0.000880, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18638 s, avg_samples: 56.0, ips: 300.46046 samples/s, eta: 13:56:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:03:23] ppocr INFO: epoch: [405/500], global_step: 285600, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.945575, exp_rate<=1 : 0.978546, exp_rate<=2 : 0.995027, exp_rate<=3 : 0.998017, loss: 0.001052, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18237 s, avg_samples: 56.0, ips: 307.06106 samples/s, eta: 13:55:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:03:42] ppocr INFO: epoch: [405/500], global_step: 285700, lr: 0.000100, edit distance: 0.001151, exp_rate: 0.945453, exp_rate<=1 : 0.978476, exp_rate<=2 : 0.994933, exp_rate<=3 : 0.998038, loss: 0.000809, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18906 s, avg_samples: 56.0, ips: 296.20235 samples/s, eta: 13:55:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:04:01] ppocr INFO: epoch: [405/500], global_step: 285800, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.944866, exp_rate<=1 : 0.978393, exp_rate<=2 : 0.994958, exp_rate<=3 : 0.997971, loss: 0.000788, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18953 s, avg_samples: 56.0, ips: 295.46973 samples/s, eta: 13:55:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:04:21] ppocr INFO: epoch: [405/500], global_step: 285900, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.944346, exp_rate<=1 : 0.977957, exp_rate<=2 : 0.994694, exp_rate<=3 : 0.997951, loss: 0.000972, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19622 s, avg_samples: 56.0, ips: 285.38769 samples/s, eta: 13:54:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:04:39] ppocr INFO: epoch: [405/500], global_step: 286000, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.944479, exp_rate<=1 : 0.977944, exp_rate<=2 : 0.994637, exp_rate<=3 : 0.997938, loss: 0.001405, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18255 s, avg_samples: 56.0, ips: 306.75690 samples/s, eta: 13:54:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:04:58] ppocr INFO: epoch: [405/500], global_step: 286100, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.942859, exp_rate<=1 : 0.977016, exp_rate<=2 : 0.994266, exp_rate<=3 : 0.997759, loss: 0.001152, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18797 s, avg_samples: 56.0, ips: 297.91618 samples/s, eta: 13:54:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:05:17] ppocr INFO: epoch: [405/500], global_step: 286200, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.942804, exp_rate<=1 : 0.977098, exp_rate<=2 : 0.994333, exp_rate<=3 : 0.997794, loss: 0.001687, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19382 s, avg_samples: 56.0, ips: 288.92636 samples/s, eta: 13:54:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:05:35] ppocr INFO: epoch: [405/500], global_step: 286300, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.943069, exp_rate<=1 : 0.977289, exp_rate<=2 : 0.994441, exp_rate<=3 : 0.997846, loss: 0.002316, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18227 s, avg_samples: 56.0, ips: 307.24400 samples/s, eta: 13:53:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:05:54] ppocr INFO: epoch: [405/500], global_step: 286400, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.942476, exp_rate<=1 : 0.977008, exp_rate<=2 : 0.994272, exp_rate<=3 : 0.997798, loss: 0.001029, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18662 s, avg_samples: 56.0, ips: 300.06747 samples/s, eta: 13:53:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:06:13] ppocr INFO: epoch: [405/500], global_step: 286500, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.941569, exp_rate<=1 : 0.976667, exp_rate<=2 : 0.994137, exp_rate<=3 : 0.997801, loss: 0.001211, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19223 s, avg_samples: 56.0, ips: 291.31461 samples/s, eta: 13:53:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:06:31] ppocr INFO: epoch: [405/500], global_step: 286600, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.941698, exp_rate<=1 : 0.976804, exp_rate<=2 : 0.994299, exp_rate<=3 : 0.997882, loss: 0.001506, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17876 s, avg_samples: 56.0, ips: 313.27470 samples/s, eta: 13:52:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:06:51] ppocr INFO: epoch: [405/500], global_step: 286700, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.941214, exp_rate<=1 : 0.976709, exp_rate<=2 : 0.994289, exp_rate<=3 : 0.997922, loss: 0.001997, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19531 s, avg_samples: 56.0, ips: 286.71769 samples/s, eta: 13:52:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:07:08] ppocr INFO: epoch: [405/500], global_step: 286800, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.940867, exp_rate<=1 : 0.976666, exp_rate<=2 : 0.994303, exp_rate<=3 : 0.997942, loss: 0.001403, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17309 s, avg_samples: 56.0, ips: 323.53247 samples/s, eta: 13:52:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:07:27] ppocr INFO: epoch: [405/500], global_step: 286900, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.940558, exp_rate<=1 : 0.976602, exp_rate<=2 : 0.994279, exp_rate<=3 : 0.997920, loss: 0.001566, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18445 s, avg_samples: 56.0, ips: 303.59985 samples/s, eta: 13:51:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:07:45] ppocr INFO: epoch: [405/500], global_step: 287000, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.940151, exp_rate<=1 : 0.976463, exp_rate<=2 : 0.994346, exp_rate<=3 : 0.997949, loss: 0.001612, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18486 s, avg_samples: 56.0, ips: 302.93372 samples/s, eta: 13:51:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:08:03] ppocr INFO: epoch: [405/500], global_step: 287100, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.940292, exp_rate<=1 : 0.976561, exp_rate<=2 : 0.994426, exp_rate<=3 : 0.997954, loss: 0.001560, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18190 s, avg_samples: 56.0, ips: 307.86515 samples/s, eta: 13:51:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:08:23] ppocr INFO: epoch: [405/500], global_step: 287200, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.939922, exp_rate<=1 : 0.976435, exp_rate<=2 : 0.994400, exp_rate<=3 : 0.997960, loss: 0.001394, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19419 s, avg_samples: 56.0, ips: 288.37521 samples/s, eta: 13:50:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:08:41] ppocr INFO: epoch: [405/500], global_step: 287300, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.939624, exp_rate<=1 : 0.976327, exp_rate<=2 : 0.994342, exp_rate<=3 : 0.997985, loss: 0.001558, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18427 s, avg_samples: 56.0, ips: 303.90329 samples/s, eta: 13:50:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:09:00] ppocr INFO: epoch: [405/500], global_step: 287400, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.939232, exp_rate<=1 : 0.976250, exp_rate<=2 : 0.994350, exp_rate<=3 : 0.997965, loss: 0.002397, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19138 s, avg_samples: 56.0, ips: 292.61806 samples/s, eta: 13:50:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:09:20] ppocr INFO: epoch: [405/500], global_step: 287500, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.938787, exp_rate<=1 : 0.976266, exp_rate<=2 : 0.994354, exp_rate<=3 : 0.997988, loss: 0.001377, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19732 s, avg_samples: 56.0, ips: 283.81013 samples/s, eta: 13:49:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:09:38] ppocr INFO: epoch: [405/500], global_step: 287600, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.938861, exp_rate<=1 : 0.976355, exp_rate<=2 : 0.994420, exp_rate<=3 : 0.998025, loss: 0.002368, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17766 s, avg_samples: 56.0, ips: 315.21260 samples/s, eta: 13:49:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:09:58] ppocr INFO: epoch: [405/500], global_step: 287700, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.938308, exp_rate<=1 : 0.976226, exp_rate<=2 : 0.994368, exp_rate<=3 : 0.997960, loss: 0.001534, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20045 s, avg_samples: 56.0, ips: 279.37080 samples/s, eta: 13:49:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:10:00] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 18:10:01] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_405 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 18:10:21] ppocr INFO: epoch: [406/500], global_step: 287800, lr: 0.000100, edit distance: 0.001135, exp_rate: 0.941191, exp_rate<=1 : 0.979578, exp_rate<=2 : 0.995933, exp_rate<=3 : 0.998275, loss: 0.001069, avg_reader_cost: 0.03547 s, avg_batch_cost: 0.22978 s, avg_samples: 56.0, ips: 243.71540 samples/s, eta: 13:48:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:10:39] ppocr INFO: epoch: [406/500], global_step: 287900, lr: 0.000100, edit distance: 0.001087, exp_rate: 0.945904, exp_rate<=1 : 0.979940, exp_rate<=2 : 0.995101, exp_rate<=3 : 0.998275, loss: 0.001474, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18499 s, avg_samples: 56.0, ips: 302.71844 samples/s, eta: 13:48:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:10:59] ppocr INFO: epoch: [406/500], global_step: 288000, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.943720, exp_rate<=1 : 0.979436, exp_rate<=2 : 0.994771, exp_rate<=3 : 0.998087, loss: 0.002057, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19193 s, avg_samples: 56.0, ips: 291.77103 samples/s, eta: 13:48:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:11:17] ppocr INFO: epoch: [406/500], global_step: 288100, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.942253, exp_rate<=1 : 0.978957, exp_rate<=2 : 0.995031, exp_rate<=3 : 0.998125, loss: 0.001530, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18692 s, avg_samples: 56.0, ips: 299.59122 samples/s, eta: 13:47:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:11:37] ppocr INFO: epoch: [406/500], global_step: 288200, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.940817, exp_rate<=1 : 0.978065, exp_rate<=2 : 0.994903, exp_rate<=3 : 0.998107, loss: 0.001379, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19816 s, avg_samples: 56.0, ips: 282.59618 samples/s, eta: 13:47:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:11:56] ppocr INFO: epoch: [406/500], global_step: 288300, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.940507, exp_rate<=1 : 0.977792, exp_rate<=2 : 0.994777, exp_rate<=3 : 0.998038, loss: 0.001478, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18833 s, avg_samples: 56.0, ips: 297.35249 samples/s, eta: 13:47:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:12:16] ppocr INFO: epoch: [406/500], global_step: 288400, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.940627, exp_rate<=1 : 0.977831, exp_rate<=2 : 0.995033, exp_rate<=3 : 0.998234, loss: 0.001238, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19598 s, avg_samples: 56.0, ips: 285.73747 samples/s, eta: 13:46:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:12:34] ppocr INFO: epoch: [406/500], global_step: 288500, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.940623, exp_rate<=1 : 0.977560, exp_rate<=2 : 0.994948, exp_rate<=3 : 0.998130, loss: 0.001143, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18000 s, avg_samples: 56.0, ips: 311.11241 samples/s, eta: 13:46:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:12:52] ppocr INFO: epoch: [406/500], global_step: 288600, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.941402, exp_rate<=1 : 0.977921, exp_rate<=2 : 0.995138, exp_rate<=3 : 0.998269, loss: 0.001358, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18209 s, avg_samples: 56.0, ips: 307.53419 samples/s, eta: 13:46:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:13:10] ppocr INFO: epoch: [406/500], global_step: 288700, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.942273, exp_rate<=1 : 0.978141, exp_rate<=2 : 0.995229, exp_rate<=3 : 0.998341, loss: 0.001038, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17554 s, avg_samples: 56.0, ips: 319.01124 samples/s, eta: 13:45:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:13:29] ppocr INFO: epoch: [406/500], global_step: 288800, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.942200, exp_rate<=1 : 0.977952, exp_rate<=2 : 0.995275, exp_rate<=3 : 0.998349, loss: 0.001259, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19073 s, avg_samples: 56.0, ips: 293.60630 samples/s, eta: 13:45:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:13:46] ppocr INFO: epoch: [406/500], global_step: 288900, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942095, exp_rate<=1 : 0.978002, exp_rate<=2 : 0.995187, exp_rate<=3 : 0.998402, loss: 0.001535, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17821 s, avg_samples: 56.0, ips: 314.23388 samples/s, eta: 13:45:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:14:05] ppocr INFO: epoch: [406/500], global_step: 289000, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.941885, exp_rate<=1 : 0.977928, exp_rate<=2 : 0.995122, exp_rate<=3 : 0.998298, loss: 0.001105, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18943 s, avg_samples: 56.0, ips: 295.62453 samples/s, eta: 13:44:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:14:26] ppocr INFO: epoch: [406/500], global_step: 289100, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.940496, exp_rate<=1 : 0.977232, exp_rate<=2 : 0.994893, exp_rate<=3 : 0.998254, loss: 0.002238, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20793 s, avg_samples: 56.0, ips: 269.31511 samples/s, eta: 13:44:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:14:45] ppocr INFO: epoch: [406/500], global_step: 289200, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.940393, exp_rate<=1 : 0.977119, exp_rate<=2 : 0.994824, exp_rate<=3 : 0.998221, loss: 0.001523, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19221 s, avg_samples: 56.0, ips: 291.34806 samples/s, eta: 13:44:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:15:05] ppocr INFO: epoch: [406/500], global_step: 289300, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.939950, exp_rate<=1 : 0.976923, exp_rate<=2 : 0.994742, exp_rate<=3 : 0.998226, loss: 0.001289, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19550 s, avg_samples: 56.0, ips: 286.44400 samples/s, eta: 13:44:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:15:24] ppocr INFO: epoch: [406/500], global_step: 289400, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.940065, exp_rate<=1 : 0.976916, exp_rate<=2 : 0.994750, exp_rate<=3 : 0.998310, loss: 0.001334, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18914 s, avg_samples: 56.0, ips: 296.08057 samples/s, eta: 13:43:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:15:42] ppocr INFO: epoch: [406/500], global_step: 289500, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.940308, exp_rate<=1 : 0.976916, exp_rate<=2 : 0.994779, exp_rate<=3 : 0.998295, loss: 0.001678, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18084 s, avg_samples: 56.0, ips: 309.66296 samples/s, eta: 13:43:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:15:59] ppocr INFO: epoch: [406/500], global_step: 289600, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.940439, exp_rate<=1 : 0.976947, exp_rate<=2 : 0.994829, exp_rate<=3 : 0.998272, loss: 0.001402, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17352 s, avg_samples: 56.0, ips: 322.72549 samples/s, eta: 13:43:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:16:18] ppocr INFO: epoch: [406/500], global_step: 289700, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.940270, exp_rate<=1 : 0.976894, exp_rate<=2 : 0.994814, exp_rate<=3 : 0.998269, loss: 0.001393, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18419 s, avg_samples: 56.0, ips: 304.02758 samples/s, eta: 13:42:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:16:37] ppocr INFO: epoch: [406/500], global_step: 289800, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.940362, exp_rate<=1 : 0.976869, exp_rate<=2 : 0.994858, exp_rate<=3 : 0.998267, loss: 0.001895, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19355 s, avg_samples: 56.0, ips: 289.33148 samples/s, eta: 13:42:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:16:56] ppocr INFO: epoch: [406/500], global_step: 289900, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.939786, exp_rate<=1 : 0.976690, exp_rate<=2 : 0.994805, exp_rate<=3 : 0.998218, loss: 0.001615, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18908 s, avg_samples: 56.0, ips: 296.16380 samples/s, eta: 13:42:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:17:15] ppocr INFO: epoch: [406/500], global_step: 290000, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.939472, exp_rate<=1 : 0.976591, exp_rate<=2 : 0.994842, exp_rate<=3 : 0.998263, loss: 0.001531, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19257 s, avg_samples: 56.0, ips: 290.80277 samples/s, eta: 13:41:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:17:33] ppocr INFO: epoch: [406/500], global_step: 290100, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.939138, exp_rate<=1 : 0.976425, exp_rate<=2 : 0.994837, exp_rate<=3 : 0.998259, loss: 0.001194, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17364 s, avg_samples: 56.0, ips: 322.50995 samples/s, eta: 13:41:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:17:51] ppocr INFO: epoch: [406/500], global_step: 290200, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.938943, exp_rate<=1 : 0.976371, exp_rate<=2 : 0.994801, exp_rate<=3 : 0.998217, loss: 0.002025, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18226 s, avg_samples: 56.0, ips: 307.24981 samples/s, eta: 13:41:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:18:09] ppocr INFO: epoch: [406/500], global_step: 290300, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.938785, exp_rate<=1 : 0.976358, exp_rate<=2 : 0.994822, exp_rate<=3 : 0.998228, loss: 0.001176, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17908 s, avg_samples: 56.0, ips: 312.70345 samples/s, eta: 13:40:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:18:30] ppocr INFO: epoch: [406/500], global_step: 290400, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.938151, exp_rate<=1 : 0.975981, exp_rate<=2 : 0.994757, exp_rate<=3 : 0.998169, loss: 0.001733, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21397 s, avg_samples: 56.0, ips: 261.72000 samples/s, eta: 13:40:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:18:37] ppocr INFO: epoch: [406/500], global_step: 290440, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.938344, exp_rate<=1 : 0.976061, exp_rate<=2 : 0.994779, exp_rate<=3 : 0.998179, loss: 0.001198, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.06793 s, avg_samples: 22.4, ips: 329.75830 samples/s, eta: 13:40:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:18:39] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 18:18:51] ppocr INFO: epoch: [407/500], global_step: 290500, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.949519, exp_rate<=1 : 0.979120, exp_rate<=2 : 0.996362, exp_rate<=3 : 0.998822, loss: 0.001285, avg_reader_cost: 0.02585 s, avg_batch_cost: 0.13581 s, avg_samples: 33.6, ips: 247.40462 samples/s, eta: 13:40:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:19:09] ppocr INFO: epoch: [407/500], global_step: 290600, lr: 0.000100, edit distance: 0.001029, exp_rate: 0.950316, exp_rate<=1 : 0.980557, exp_rate<=2 : 0.995581, exp_rate<=3 : 0.998382, loss: 0.001287, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18301 s, avg_samples: 56.0, ips: 305.99363 samples/s, eta: 13:39:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:19:27] ppocr INFO: epoch: [407/500], global_step: 290700, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.950334, exp_rate<=1 : 0.981103, exp_rate<=2 : 0.995733, exp_rate<=3 : 0.998503, loss: 0.001496, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17780 s, avg_samples: 56.0, ips: 314.96017 samples/s, eta: 13:39:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:19:45] ppocr INFO: epoch: [407/500], global_step: 290800, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.949682, exp_rate<=1 : 0.980898, exp_rate<=2 : 0.995473, exp_rate<=3 : 0.998555, loss: 0.001238, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18237 s, avg_samples: 56.0, ips: 307.06648 samples/s, eta: 13:39:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:20:04] ppocr INFO: epoch: [407/500], global_step: 290900, lr: 0.000100, edit distance: 0.001134, exp_rate: 0.949137, exp_rate<=1 : 0.980946, exp_rate<=2 : 0.995640, exp_rate<=3 : 0.998533, loss: 0.001145, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18934 s, avg_samples: 56.0, ips: 295.76352 samples/s, eta: 13:38:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:20:24] ppocr INFO: epoch: [407/500], global_step: 291000, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.949157, exp_rate<=1 : 0.980686, exp_rate<=2 : 0.995666, exp_rate<=3 : 0.998493, loss: 0.001170, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19597 s, avg_samples: 56.0, ips: 285.76059 samples/s, eta: 13:38:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:20:43] ppocr INFO: epoch: [407/500], global_step: 291100, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.947700, exp_rate<=1 : 0.980000, exp_rate<=2 : 0.995624, exp_rate<=3 : 0.998479, loss: 0.002012, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18904 s, avg_samples: 56.0, ips: 296.23682 samples/s, eta: 13:38:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:21:03] ppocr INFO: epoch: [407/500], global_step: 291200, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.945456, exp_rate<=1 : 0.979249, exp_rate<=2 : 0.995381, exp_rate<=3 : 0.998360, loss: 0.001606, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20637 s, avg_samples: 56.0, ips: 271.35918 samples/s, eta: 13:37:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:21:22] ppocr INFO: epoch: [407/500], global_step: 291300, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.945283, exp_rate<=1 : 0.979338, exp_rate<=2 : 0.995169, exp_rate<=3 : 0.998148, loss: 0.000952, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18166 s, avg_samples: 56.0, ips: 308.27540 samples/s, eta: 13:37:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:21:41] ppocr INFO: epoch: [407/500], global_step: 291400, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.944973, exp_rate<=1 : 0.978736, exp_rate<=2 : 0.994989, exp_rate<=3 : 0.998104, loss: 0.001224, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19144 s, avg_samples: 56.0, ips: 292.52655 samples/s, eta: 13:37:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:21:59] ppocr INFO: epoch: [407/500], global_step: 291500, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.945128, exp_rate<=1 : 0.978801, exp_rate<=2 : 0.995065, exp_rate<=3 : 0.998147, loss: 0.001819, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18473 s, avg_samples: 56.0, ips: 303.13711 samples/s, eta: 13:36:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:22:18] ppocr INFO: epoch: [407/500], global_step: 291600, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.944298, exp_rate<=1 : 0.978463, exp_rate<=2 : 0.994906, exp_rate<=3 : 0.998052, loss: 0.001340, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18537 s, avg_samples: 56.0, ips: 302.10365 samples/s, eta: 13:36:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:22:35] ppocr INFO: epoch: [407/500], global_step: 291700, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.944322, exp_rate<=1 : 0.978599, exp_rate<=2 : 0.994899, exp_rate<=3 : 0.998057, loss: 0.001883, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17566 s, avg_samples: 56.0, ips: 318.79792 samples/s, eta: 13:36:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:22:55] ppocr INFO: epoch: [407/500], global_step: 291800, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.943911, exp_rate<=1 : 0.978308, exp_rate<=2 : 0.994853, exp_rate<=3 : 0.998094, loss: 0.001685, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19402 s, avg_samples: 56.0, ips: 288.62748 samples/s, eta: 13:35:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:23:14] ppocr INFO: epoch: [407/500], global_step: 291900, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.943596, exp_rate<=1 : 0.978196, exp_rate<=2 : 0.994800, exp_rate<=3 : 0.998115, loss: 0.002191, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18838 s, avg_samples: 56.0, ips: 297.26797 samples/s, eta: 13:35:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:23:32] ppocr INFO: epoch: [407/500], global_step: 292000, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.942804, exp_rate<=1 : 0.977966, exp_rate<=2 : 0.994865, exp_rate<=3 : 0.998146, loss: 0.001817, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17953 s, avg_samples: 56.0, ips: 311.92373 samples/s, eta: 13:35:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:23:50] ppocr INFO: epoch: [407/500], global_step: 292100, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.942519, exp_rate<=1 : 0.977869, exp_rate<=2 : 0.994902, exp_rate<=3 : 0.998142, loss: 0.001352, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18551 s, avg_samples: 56.0, ips: 301.87075 samples/s, eta: 13:34:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:24:09] ppocr INFO: epoch: [407/500], global_step: 292200, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.942261, exp_rate<=1 : 0.977798, exp_rate<=2 : 0.994812, exp_rate<=3 : 0.998115, loss: 0.001288, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18787 s, avg_samples: 56.0, ips: 298.07767 samples/s, eta: 13:34:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:24:30] ppocr INFO: epoch: [407/500], global_step: 292300, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.941835, exp_rate<=1 : 0.977658, exp_rate<=2 : 0.994824, exp_rate<=3 : 0.998128, loss: 0.001762, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20590 s, avg_samples: 56.0, ips: 271.97436 samples/s, eta: 13:34:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:24:48] ppocr INFO: epoch: [407/500], global_step: 292400, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.940938, exp_rate<=1 : 0.977325, exp_rate<=2 : 0.994763, exp_rate<=3 : 0.998125, loss: 0.001891, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18822 s, avg_samples: 56.0, ips: 297.52310 samples/s, eta: 13:34:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:25:09] ppocr INFO: epoch: [407/500], global_step: 292500, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.939617, exp_rate<=1 : 0.976736, exp_rate<=2 : 0.994662, exp_rate<=3 : 0.998075, loss: 0.002031, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20052 s, avg_samples: 56.0, ips: 279.26775 samples/s, eta: 13:33:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:25:28] ppocr INFO: epoch: [407/500], global_step: 292600, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.939279, exp_rate<=1 : 0.976651, exp_rate<=2 : 0.994583, exp_rate<=3 : 0.998038, loss: 0.001300, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19227 s, avg_samples: 56.0, ips: 291.25289 samples/s, eta: 13:33:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:25:46] ppocr INFO: epoch: [407/500], global_step: 292700, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.939197, exp_rate<=1 : 0.976703, exp_rate<=2 : 0.994640, exp_rate<=3 : 0.998066, loss: 0.001734, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17731 s, avg_samples: 56.0, ips: 315.82706 samples/s, eta: 13:33:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:26:04] ppocr INFO: epoch: [407/500], global_step: 292800, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.938995, exp_rate<=1 : 0.976514, exp_rate<=2 : 0.994635, exp_rate<=3 : 0.998047, loss: 0.001696, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18759 s, avg_samples: 56.0, ips: 298.52128 samples/s, eta: 13:32:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:26:23] ppocr INFO: epoch: [407/500], global_step: 292900, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.939039, exp_rate<=1 : 0.976535, exp_rate<=2 : 0.994638, exp_rate<=3 : 0.998071, loss: 0.001744, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18230 s, avg_samples: 56.0, ips: 307.18192 samples/s, eta: 13:32:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:26:41] ppocr INFO: epoch: [407/500], global_step: 293000, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.938969, exp_rate<=1 : 0.976510, exp_rate<=2 : 0.994668, exp_rate<=3 : 0.998107, loss: 0.002393, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18382 s, avg_samples: 56.0, ips: 304.64178 samples/s, eta: 13:32:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:27:01] ppocr INFO: epoch: [407/500], global_step: 293100, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.938538, exp_rate<=1 : 0.976383, exp_rate<=2 : 0.994586, exp_rate<=3 : 0.998061, loss: 0.001785, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19686 s, avg_samples: 56.0, ips: 284.46277 samples/s, eta: 13:31:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:27:16] ppocr INFO: epoch: [407/500], global_step: 293180, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.938532, exp_rate<=1 : 0.976325, exp_rate<=2 : 0.994554, exp_rate<=3 : 0.998093, loss: 0.001137, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14834 s, avg_samples: 44.8, ips: 302.01163 samples/s, eta: 13:31:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:27:18] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 18:27:21] ppocr INFO: epoch: [408/500], global_step: 293200, lr: 0.000100, edit distance: 0.000964, exp_rate: 0.960794, exp_rate<=1 : 0.987866, exp_rate<=2 : 0.998444, exp_rate<=3 : 1.000000, loss: 0.001217, avg_reader_cost: 0.02561 s, avg_batch_cost: 0.05923 s, avg_samples: 11.2, ips: 189.08007 samples/s, eta: 13:31:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:27:39] ppocr INFO: epoch: [408/500], global_step: 293300, lr: 0.000100, edit distance: 0.001065, exp_rate: 0.951000, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.995745, exp_rate<=3 : 0.998183, loss: 0.000881, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17963 s, avg_samples: 56.0, ips: 311.74854 samples/s, eta: 13:31:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:28:00] ppocr INFO: epoch: [408/500], global_step: 293400, lr: 0.000100, edit distance: 0.001118, exp_rate: 0.944859, exp_rate<=1 : 0.979895, exp_rate<=2 : 0.994345, exp_rate<=3 : 0.997879, loss: 0.001047, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20283 s, avg_samples: 56.0, ips: 276.09258 samples/s, eta: 13:30:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:28:18] ppocr INFO: epoch: [408/500], global_step: 293500, lr: 0.000100, edit distance: 0.001051, exp_rate: 0.949752, exp_rate<=1 : 0.981967, exp_rate<=2 : 0.995202, exp_rate<=3 : 0.998203, loss: 0.000797, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18660 s, avg_samples: 56.0, ips: 300.10937 samples/s, eta: 13:30:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:28:37] ppocr INFO: epoch: [408/500], global_step: 293600, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.947030, exp_rate<=1 : 0.980982, exp_rate<=2 : 0.995301, exp_rate<=3 : 0.998317, loss: 0.001338, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18892 s, avg_samples: 56.0, ips: 296.41616 samples/s, eta: 13:30:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:28:56] ppocr INFO: epoch: [408/500], global_step: 293700, lr: 0.000100, edit distance: 0.001119, exp_rate: 0.945907, exp_rate<=1 : 0.980236, exp_rate<=2 : 0.995368, exp_rate<=3 : 0.998451, loss: 0.001364, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18654 s, avg_samples: 56.0, ips: 300.21074 samples/s, eta: 13:29:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:29:14] ppocr INFO: epoch: [408/500], global_step: 293800, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.945666, exp_rate<=1 : 0.979838, exp_rate<=2 : 0.995376, exp_rate<=3 : 0.998473, loss: 0.001751, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18255 s, avg_samples: 56.0, ips: 306.76019 samples/s, eta: 13:29:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:29:33] ppocr INFO: epoch: [408/500], global_step: 293900, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.945380, exp_rate<=1 : 0.979617, exp_rate<=2 : 0.995457, exp_rate<=3 : 0.998512, loss: 0.002074, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18345 s, avg_samples: 56.0, ips: 305.26144 samples/s, eta: 13:29:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:29:54] ppocr INFO: epoch: [408/500], global_step: 294000, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.943995, exp_rate<=1 : 0.979228, exp_rate<=2 : 0.995145, exp_rate<=3 : 0.998346, loss: 0.001258, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21550 s, avg_samples: 56.0, ips: 259.86025 samples/s, eta: 13:28:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:30:13] ppocr INFO: epoch: [408/500], global_step: 294100, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.943722, exp_rate<=1 : 0.979078, exp_rate<=2 : 0.995116, exp_rate<=3 : 0.998294, loss: 0.001165, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18906 s, avg_samples: 56.0, ips: 296.20433 samples/s, eta: 13:28:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:30:32] ppocr INFO: epoch: [408/500], global_step: 294200, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.943343, exp_rate<=1 : 0.978761, exp_rate<=2 : 0.995217, exp_rate<=3 : 0.998365, loss: 0.001196, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18855 s, avg_samples: 56.0, ips: 296.99595 samples/s, eta: 13:28:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:30:51] ppocr INFO: epoch: [408/500], global_step: 294300, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942825, exp_rate<=1 : 0.978464, exp_rate<=2 : 0.995125, exp_rate<=3 : 0.998279, loss: 0.001716, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19360 s, avg_samples: 56.0, ips: 289.26215 samples/s, eta: 13:27:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:31:11] ppocr INFO: epoch: [408/500], global_step: 294400, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.942853, exp_rate<=1 : 0.978589, exp_rate<=2 : 0.995102, exp_rate<=3 : 0.998304, loss: 0.001495, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19166 s, avg_samples: 56.0, ips: 292.17798 samples/s, eta: 13:27:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:31:29] ppocr INFO: epoch: [408/500], global_step: 294500, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.943240, exp_rate<=1 : 0.978881, exp_rate<=2 : 0.995237, exp_rate<=3 : 0.998365, loss: 0.001127, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18821 s, avg_samples: 56.0, ips: 297.54026 samples/s, eta: 13:27:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:31:48] ppocr INFO: epoch: [408/500], global_step: 294600, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.943277, exp_rate<=1 : 0.978908, exp_rate<=2 : 0.995147, exp_rate<=3 : 0.998366, loss: 0.001354, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18433 s, avg_samples: 56.0, ips: 303.80504 samples/s, eta: 13:27:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:32:06] ppocr INFO: epoch: [408/500], global_step: 294700, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.943478, exp_rate<=1 : 0.978982, exp_rate<=2 : 0.995159, exp_rate<=3 : 0.998373, loss: 0.001393, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18451 s, avg_samples: 56.0, ips: 303.50274 samples/s, eta: 13:26:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:32:24] ppocr INFO: epoch: [408/500], global_step: 294800, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.943832, exp_rate<=1 : 0.979171, exp_rate<=2 : 0.995231, exp_rate<=3 : 0.998392, loss: 0.001208, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17604 s, avg_samples: 56.0, ips: 318.10330 samples/s, eta: 13:26:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:32:44] ppocr INFO: epoch: [408/500], global_step: 294900, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.943729, exp_rate<=1 : 0.978999, exp_rate<=2 : 0.995100, exp_rate<=3 : 0.998366, loss: 0.001021, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19836 s, avg_samples: 56.0, ips: 282.31161 samples/s, eta: 13:26:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:33:02] ppocr INFO: epoch: [408/500], global_step: 295000, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.943601, exp_rate<=1 : 0.978903, exp_rate<=2 : 0.995099, exp_rate<=3 : 0.998402, loss: 0.001530, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18069 s, avg_samples: 56.0, ips: 309.91951 samples/s, eta: 13:25:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:33:20] ppocr INFO: epoch: [408/500], global_step: 295100, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.943377, exp_rate<=1 : 0.978859, exp_rate<=2 : 0.995136, exp_rate<=3 : 0.998394, loss: 0.001717, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18537 s, avg_samples: 56.0, ips: 302.10457 samples/s, eta: 13:25:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:33:41] ppocr INFO: epoch: [408/500], global_step: 295200, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.942819, exp_rate<=1 : 0.978683, exp_rate<=2 : 0.995088, exp_rate<=3 : 0.998410, loss: 0.001498, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20243 s, avg_samples: 56.0, ips: 276.63902 samples/s, eta: 13:25:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:33:59] ppocr INFO: epoch: [408/500], global_step: 295300, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.942362, exp_rate<=1 : 0.978545, exp_rate<=2 : 0.995008, exp_rate<=3 : 0.998384, loss: 0.001427, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18570 s, avg_samples: 56.0, ips: 301.56045 samples/s, eta: 13:24:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:34:17] ppocr INFO: epoch: [408/500], global_step: 295400, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.942353, exp_rate<=1 : 0.978544, exp_rate<=2 : 0.994984, exp_rate<=3 : 0.998398, loss: 0.001599, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17629 s, avg_samples: 56.0, ips: 317.66113 samples/s, eta: 13:24:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:34:36] ppocr INFO: epoch: [408/500], global_step: 295500, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.941771, exp_rate<=1 : 0.978241, exp_rate<=2 : 0.994938, exp_rate<=3 : 0.998351, loss: 0.001410, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18590 s, avg_samples: 56.0, ips: 301.23065 samples/s, eta: 13:24:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:34:54] ppocr INFO: epoch: [408/500], global_step: 295600, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.941502, exp_rate<=1 : 0.978119, exp_rate<=2 : 0.994916, exp_rate<=3 : 0.998358, loss: 0.001253, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18666 s, avg_samples: 56.0, ips: 300.00478 samples/s, eta: 13:23:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:35:14] ppocr INFO: epoch: [408/500], global_step: 295700, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.941270, exp_rate<=1 : 0.978023, exp_rate<=2 : 0.994877, exp_rate<=3 : 0.998341, loss: 0.002233, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19604 s, avg_samples: 56.0, ips: 285.65760 samples/s, eta: 13:23:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:35:31] ppocr INFO: epoch: [408/500], global_step: 295800, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.941163, exp_rate<=1 : 0.977989, exp_rate<=2 : 0.994894, exp_rate<=3 : 0.998352, loss: 0.002498, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17243 s, avg_samples: 56.0, ips: 324.76644 samples/s, eta: 13:23:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:35:49] ppocr INFO: epoch: [408/500], global_step: 295900, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941020, exp_rate<=1 : 0.977802, exp_rate<=2 : 0.994835, exp_rate<=3 : 0.998348, loss: 0.001042, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18121 s, avg_samples: 56.0, ips: 309.03773 samples/s, eta: 13:22:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:35:53] ppocr INFO: epoch: [408/500], global_step: 295920, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.941102, exp_rate<=1 : 0.977832, exp_rate<=2 : 0.994860, exp_rate<=3 : 0.998357, loss: 0.001485, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03528 s, avg_samples: 11.2, ips: 317.43525 samples/s, eta: 13:22:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:35:55] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 18:36:10] ppocr INFO: epoch: [409/500], global_step: 296000, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.937944, exp_rate<=1 : 0.978977, exp_rate<=2 : 0.995280, exp_rate<=3 : 0.998987, loss: 0.001297, avg_reader_cost: 0.02488 s, avg_batch_cost: 0.17022 s, avg_samples: 44.8, ips: 263.18140 samples/s, eta: 13:22:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:36:28] ppocr INFO: epoch: [409/500], global_step: 296100, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.938737, exp_rate<=1 : 0.978736, exp_rate<=2 : 0.995640, exp_rate<=3 : 0.998699, loss: 0.001822, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17767 s, avg_samples: 56.0, ips: 315.18766 samples/s, eta: 13:22:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:36:47] ppocr INFO: epoch: [409/500], global_step: 296200, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.937940, exp_rate<=1 : 0.976955, exp_rate<=2 : 0.995620, exp_rate<=3 : 0.998614, loss: 0.001452, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19066 s, avg_samples: 56.0, ips: 293.71832 samples/s, eta: 13:21:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:37:05] ppocr INFO: epoch: [409/500], global_step: 296300, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.936259, exp_rate<=1 : 0.975235, exp_rate<=2 : 0.994628, exp_rate<=3 : 0.998207, loss: 0.002169, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18513 s, avg_samples: 56.0, ips: 302.49722 samples/s, eta: 13:21:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:37:25] ppocr INFO: epoch: [409/500], global_step: 296400, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.934212, exp_rate<=1 : 0.973692, exp_rate<=2 : 0.993927, exp_rate<=3 : 0.998131, loss: 0.001603, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19716 s, avg_samples: 56.0, ips: 284.03487 samples/s, eta: 13:21:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:37:43] ppocr INFO: epoch: [409/500], global_step: 296500, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.936653, exp_rate<=1 : 0.974784, exp_rate<=2 : 0.994310, exp_rate<=3 : 0.998208, loss: 0.001653, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18372 s, avg_samples: 56.0, ips: 304.81553 samples/s, eta: 13:20:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:38:01] ppocr INFO: epoch: [409/500], global_step: 296600, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.937695, exp_rate<=1 : 0.975048, exp_rate<=2 : 0.994311, exp_rate<=3 : 0.998139, loss: 0.000903, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18030 s, avg_samples: 56.0, ips: 310.59563 samples/s, eta: 13:20:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:38:21] ppocr INFO: epoch: [409/500], global_step: 296700, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.937459, exp_rate<=1 : 0.974611, exp_rate<=2 : 0.994153, exp_rate<=3 : 0.998082, loss: 0.002003, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19451 s, avg_samples: 56.0, ips: 287.90482 samples/s, eta: 13:20:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:38:40] ppocr INFO: epoch: [409/500], global_step: 296800, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.937085, exp_rate<=1 : 0.974259, exp_rate<=2 : 0.994075, exp_rate<=3 : 0.998031, loss: 0.000887, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19512 s, avg_samples: 56.0, ips: 287.00158 samples/s, eta: 13:19:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:38:59] ppocr INFO: epoch: [409/500], global_step: 296900, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.937012, exp_rate<=1 : 0.974503, exp_rate<=2 : 0.994127, exp_rate<=3 : 0.998158, loss: 0.001198, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18826 s, avg_samples: 56.0, ips: 297.46615 samples/s, eta: 13:19:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:39:20] ppocr INFO: epoch: [409/500], global_step: 297000, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.936790, exp_rate<=1 : 0.974415, exp_rate<=2 : 0.994075, exp_rate<=3 : 0.998125, loss: 0.001549, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20239 s, avg_samples: 56.0, ips: 276.69167 samples/s, eta: 13:19:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:39:39] ppocr INFO: epoch: [409/500], global_step: 297100, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.936930, exp_rate<=1 : 0.974635, exp_rate<=2 : 0.994207, exp_rate<=3 : 0.998177, loss: 0.001335, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19511 s, avg_samples: 56.0, ips: 287.02242 samples/s, eta: 13:18:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:39:58] ppocr INFO: epoch: [409/500], global_step: 297200, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.937320, exp_rate<=1 : 0.974897, exp_rate<=2 : 0.994244, exp_rate<=3 : 0.998173, loss: 0.001313, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.73667 samples/s, eta: 13:18:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:40:17] ppocr INFO: epoch: [409/500], global_step: 297300, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.936029, exp_rate<=1 : 0.974479, exp_rate<=2 : 0.993969, exp_rate<=3 : 0.998013, loss: 0.002801, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19076 s, avg_samples: 56.0, ips: 293.55723 samples/s, eta: 13:18:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:40:36] ppocr INFO: epoch: [409/500], global_step: 297400, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.934422, exp_rate<=1 : 0.973639, exp_rate<=2 : 0.993678, exp_rate<=3 : 0.997778, loss: 0.001741, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18622 s, avg_samples: 56.0, ips: 300.71154 samples/s, eta: 13:17:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:40:57] ppocr INFO: epoch: [409/500], global_step: 297500, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.934229, exp_rate<=1 : 0.973595, exp_rate<=2 : 0.993675, exp_rate<=3 : 0.997736, loss: 0.001673, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20534 s, avg_samples: 56.0, ips: 272.72412 samples/s, eta: 13:17:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:41:15] ppocr INFO: epoch: [409/500], global_step: 297600, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.934397, exp_rate<=1 : 0.973682, exp_rate<=2 : 0.993720, exp_rate<=3 : 0.997738, loss: 0.001311, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18159 s, avg_samples: 56.0, ips: 308.39470 samples/s, eta: 13:17:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:41:33] ppocr INFO: epoch: [409/500], global_step: 297700, lr: 0.000100, edit distance: 0.001340, exp_rate: 0.935001, exp_rate<=1 : 0.974012, exp_rate<=2 : 0.993808, exp_rate<=3 : 0.997788, loss: 0.002122, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17692 s, avg_samples: 56.0, ips: 316.52618 samples/s, eta: 13:17:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:41:51] ppocr INFO: epoch: [409/500], global_step: 297800, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.935102, exp_rate<=1 : 0.974252, exp_rate<=2 : 0.993882, exp_rate<=3 : 0.997846, loss: 0.001534, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18014 s, avg_samples: 56.0, ips: 310.86299 samples/s, eta: 13:16:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:42:08] ppocr INFO: epoch: [409/500], global_step: 297900, lr: 0.000100, edit distance: 0.001352, exp_rate: 0.935584, exp_rate<=1 : 0.974498, exp_rate<=2 : 0.993974, exp_rate<=3 : 0.997879, loss: 0.001659, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17053 s, avg_samples: 56.0, ips: 328.39254 samples/s, eta: 13:16:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:42:27] ppocr INFO: epoch: [409/500], global_step: 298000, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.935378, exp_rate<=1 : 0.974447, exp_rate<=2 : 0.993947, exp_rate<=3 : 0.997801, loss: 0.001322, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19373 s, avg_samples: 56.0, ips: 289.05520 samples/s, eta: 13:16:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:42:45] ppocr INFO: epoch: [409/500], global_step: 298100, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.935643, exp_rate<=1 : 0.974625, exp_rate<=2 : 0.993987, exp_rate<=3 : 0.997844, loss: 0.001926, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18040 s, avg_samples: 56.0, ips: 310.42012 samples/s, eta: 13:15:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:43:04] ppocr INFO: epoch: [409/500], global_step: 298200, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.935935, exp_rate<=1 : 0.974750, exp_rate<=2 : 0.994073, exp_rate<=3 : 0.997879, loss: 0.001349, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18407 s, avg_samples: 56.0, ips: 304.22887 samples/s, eta: 13:15:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:43:21] ppocr INFO: epoch: [409/500], global_step: 298300, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.936553, exp_rate<=1 : 0.975079, exp_rate<=2 : 0.994232, exp_rate<=3 : 0.997932, loss: 0.001422, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17947 s, avg_samples: 56.0, ips: 312.02485 samples/s, eta: 13:15:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:43:41] ppocr INFO: epoch: [409/500], global_step: 298400, lr: 0.000100, edit distance: 0.001351, exp_rate: 0.936435, exp_rate<=1 : 0.975048, exp_rate<=2 : 0.994162, exp_rate<=3 : 0.997911, loss: 0.001613, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19232 s, avg_samples: 56.0, ips: 291.17427 samples/s, eta: 13:14:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:44:00] ppocr INFO: epoch: [409/500], global_step: 298500, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.936415, exp_rate<=1 : 0.975082, exp_rate<=2 : 0.994198, exp_rate<=3 : 0.997937, loss: 0.001375, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19042 s, avg_samples: 56.0, ips: 294.09057 samples/s, eta: 13:14:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:44:18] ppocr INFO: epoch: [409/500], global_step: 298600, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.936467, exp_rate<=1 : 0.975105, exp_rate<=2 : 0.994189, exp_rate<=3 : 0.997953, loss: 0.001167, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18664 s, avg_samples: 56.0, ips: 300.04150 samples/s, eta: 13:14:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:44:30] ppocr INFO: epoch: [409/500], global_step: 298660, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.936501, exp_rate<=1 : 0.975154, exp_rate<=2 : 0.994187, exp_rate<=3 : 0.997961, loss: 0.001593, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11846 s, avg_samples: 33.6, ips: 283.64523 samples/s, eta: 13:13:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:44:33] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 18:44:42] ppocr INFO: epoch: [410/500], global_step: 298700, lr: 0.000100, edit distance: 0.001396, exp_rate: 0.941158, exp_rate<=1 : 0.973350, exp_rate<=2 : 0.992225, exp_rate<=3 : 0.997072, loss: 0.001112, avg_reader_cost: 0.02800 s, avg_batch_cost: 0.11174 s, avg_samples: 22.4, ips: 200.45848 samples/s, eta: 13:13:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:45:02] ppocr INFO: epoch: [410/500], global_step: 298800, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.939646, exp_rate<=1 : 0.975829, exp_rate<=2 : 0.992898, exp_rate<=3 : 0.997488, loss: 0.001503, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20566 s, avg_samples: 56.0, ips: 272.28999 samples/s, eta: 13:13:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:45:19] ppocr INFO: epoch: [410/500], global_step: 298900, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.946581, exp_rate<=1 : 0.979930, exp_rate<=2 : 0.994920, exp_rate<=3 : 0.998218, loss: 0.001581, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17174 s, avg_samples: 56.0, ips: 326.06768 samples/s, eta: 13:13:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:45:38] ppocr INFO: epoch: [410/500], global_step: 299000, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.946050, exp_rate<=1 : 0.979884, exp_rate<=2 : 0.994824, exp_rate<=3 : 0.998311, loss: 0.001722, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18752 s, avg_samples: 56.0, ips: 298.63902 samples/s, eta: 13:12:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:45:57] ppocr INFO: epoch: [410/500], global_step: 299100, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.945743, exp_rate<=1 : 0.979633, exp_rate<=2 : 0.994871, exp_rate<=3 : 0.998414, loss: 0.000993, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.74283 samples/s, eta: 13:12:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:46:17] ppocr INFO: epoch: [410/500], global_step: 299200, lr: 0.000100, edit distance: 0.001112, exp_rate: 0.947005, exp_rate<=1 : 0.980161, exp_rate<=2 : 0.995004, exp_rate<=3 : 0.998401, loss: 0.001050, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19364 s, avg_samples: 56.0, ips: 289.20195 samples/s, eta: 13:12:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:46:35] ppocr INFO: epoch: [410/500], global_step: 299300, lr: 0.000100, edit distance: 0.001124, exp_rate: 0.946372, exp_rate<=1 : 0.979228, exp_rate<=2 : 0.994937, exp_rate<=3 : 0.998272, loss: 0.001446, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18686 s, avg_samples: 56.0, ips: 299.68664 samples/s, eta: 13:11:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:46:54] ppocr INFO: epoch: [410/500], global_step: 299400, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.945658, exp_rate<=1 : 0.979079, exp_rate<=2 : 0.994923, exp_rate<=3 : 0.998305, loss: 0.002044, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18418 s, avg_samples: 56.0, ips: 304.04258 samples/s, eta: 13:11:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:47:12] ppocr INFO: epoch: [410/500], global_step: 299500, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.945107, exp_rate<=1 : 0.978964, exp_rate<=2 : 0.994814, exp_rate<=3 : 0.998166, loss: 0.001458, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18459 s, avg_samples: 56.0, ips: 303.38321 samples/s, eta: 13:11:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:47:32] ppocr INFO: epoch: [410/500], global_step: 299600, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.944657, exp_rate<=1 : 0.978898, exp_rate<=2 : 0.994772, exp_rate<=3 : 0.998178, loss: 0.001237, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19664 s, avg_samples: 56.0, ips: 284.78016 samples/s, eta: 13:10:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:47:52] ppocr INFO: epoch: [410/500], global_step: 299700, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.943309, exp_rate<=1 : 0.978162, exp_rate<=2 : 0.994571, exp_rate<=3 : 0.998058, loss: 0.001755, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19458 s, avg_samples: 56.0, ips: 287.80484 samples/s, eta: 13:10:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:48:10] ppocr INFO: epoch: [410/500], global_step: 299800, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.942837, exp_rate<=1 : 0.977912, exp_rate<=2 : 0.994526, exp_rate<=3 : 0.998056, loss: 0.001647, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18808 s, avg_samples: 56.0, ips: 297.74781 samples/s, eta: 13:10:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:48:28] ppocr INFO: epoch: [410/500], global_step: 299900, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.942959, exp_rate<=1 : 0.977876, exp_rate<=2 : 0.994546, exp_rate<=3 : 0.998112, loss: 0.001410, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18089 s, avg_samples: 56.0, ips: 309.58565 samples/s, eta: 13:09:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:48:47] ppocr INFO: epoch: [410/500], global_step: 300000, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.942560, exp_rate<=1 : 0.977818, exp_rate<=2 : 0.994498, exp_rate<=3 : 0.998121, loss: 0.001647, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18790 s, avg_samples: 56.0, ips: 298.03796 samples/s, eta: 13:09:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:49:06] ppocr INFO: epoch: [410/500], global_step: 300100, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.941685, exp_rate<=1 : 0.977532, exp_rate<=2 : 0.994460, exp_rate<=3 : 0.998128, loss: 0.001460, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18445 s, avg_samples: 56.0, ips: 303.59913 samples/s, eta: 13:09:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:49:25] ppocr INFO: epoch: [410/500], global_step: 300200, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.940985, exp_rate<=1 : 0.977194, exp_rate<=2 : 0.994421, exp_rate<=3 : 0.998123, loss: 0.002027, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19011 s, avg_samples: 56.0, ips: 294.57082 samples/s, eta: 13:08:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:49:43] ppocr INFO: epoch: [410/500], global_step: 300300, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.940600, exp_rate<=1 : 0.976819, exp_rate<=2 : 0.994337, exp_rate<=3 : 0.998130, loss: 0.001619, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18184 s, avg_samples: 56.0, ips: 307.96728 samples/s, eta: 13:08:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:50:01] ppocr INFO: epoch: [410/500], global_step: 300400, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.940462, exp_rate<=1 : 0.976807, exp_rate<=2 : 0.994335, exp_rate<=3 : 0.998123, loss: 0.001753, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18156 s, avg_samples: 56.0, ips: 308.43802 samples/s, eta: 13:08:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:50:20] ppocr INFO: epoch: [410/500], global_step: 300500, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.939996, exp_rate<=1 : 0.976597, exp_rate<=2 : 0.994332, exp_rate<=3 : 0.998138, loss: 0.001848, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18975 s, avg_samples: 56.0, ips: 295.12805 samples/s, eta: 13:07:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:50:39] ppocr INFO: epoch: [410/500], global_step: 300600, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.939636, exp_rate<=1 : 0.976425, exp_rate<=2 : 0.994287, exp_rate<=3 : 0.998132, loss: 0.001483, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19040 s, avg_samples: 56.0, ips: 294.11681 samples/s, eta: 13:07:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:50:57] ppocr INFO: epoch: [410/500], global_step: 300700, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.939511, exp_rate<=1 : 0.976469, exp_rate<=2 : 0.994382, exp_rate<=3 : 0.998154, loss: 0.001593, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17646 s, avg_samples: 56.0, ips: 317.35575 samples/s, eta: 13:07:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:51:15] ppocr INFO: epoch: [410/500], global_step: 300800, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.939228, exp_rate<=1 : 0.976432, exp_rate<=2 : 0.994364, exp_rate<=3 : 0.998166, loss: 0.001648, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17743 s, avg_samples: 56.0, ips: 315.61041 samples/s, eta: 13:07:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:51:33] ppocr INFO: epoch: [410/500], global_step: 300900, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939017, exp_rate<=1 : 0.976353, exp_rate<=2 : 0.994381, exp_rate<=3 : 0.998168, loss: 0.001519, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18345 s, avg_samples: 56.0, ips: 305.25311 samples/s, eta: 13:06:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:51:52] ppocr INFO: epoch: [410/500], global_step: 301000, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.938196, exp_rate<=1 : 0.975957, exp_rate<=2 : 0.994359, exp_rate<=3 : 0.998174, loss: 0.001698, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19456 s, avg_samples: 56.0, ips: 287.82485 samples/s, eta: 13:06:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:52:13] ppocr INFO: epoch: [410/500], global_step: 301100, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.937665, exp_rate<=1 : 0.975731, exp_rate<=2 : 0.994307, exp_rate<=3 : 0.998185, loss: 0.001359, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20110 s, avg_samples: 56.0, ips: 278.46682 samples/s, eta: 13:06:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:52:31] ppocr INFO: epoch: [410/500], global_step: 301200, lr: 0.000100, edit distance: 0.001333, exp_rate: 0.937491, exp_rate<=1 : 0.975715, exp_rate<=2 : 0.994322, exp_rate<=3 : 0.998210, loss: 0.001389, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18808 s, avg_samples: 56.0, ips: 297.75070 samples/s, eta: 13:05:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:52:49] ppocr INFO: epoch: [410/500], global_step: 301300, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.937697, exp_rate<=1 : 0.975902, exp_rate<=2 : 0.994394, exp_rate<=3 : 0.998257, loss: 0.001234, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17247 s, avg_samples: 56.0, ips: 324.68911 samples/s, eta: 13:05:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:53:08] ppocr INFO: epoch: [410/500], global_step: 301400, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.937229, exp_rate<=1 : 0.975800, exp_rate<=2 : 0.994304, exp_rate<=3 : 0.998183, loss: 0.001470, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19503 s, avg_samples: 56.0, ips: 287.13333 samples/s, eta: 13:05:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:53:10] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 18:53:11] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_410 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 18:53:30] ppocr INFO: epoch: [411/500], global_step: 301500, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.945755, exp_rate<=1 : 0.981271, exp_rate<=2 : 0.995959, exp_rate<=3 : 0.998571, loss: 0.001878, avg_reader_cost: 0.03265 s, avg_batch_cost: 0.21775 s, avg_samples: 56.0, ips: 257.18130 samples/s, eta: 13:04:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:53:49] ppocr INFO: epoch: [411/500], global_step: 301600, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.945476, exp_rate<=1 : 0.978210, exp_rate<=2 : 0.994002, exp_rate<=3 : 0.997756, loss: 0.001555, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18732 s, avg_samples: 56.0, ips: 298.95972 samples/s, eta: 13:04:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:54:08] ppocr INFO: epoch: [411/500], global_step: 301700, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.943297, exp_rate<=1 : 0.977499, exp_rate<=2 : 0.994180, exp_rate<=3 : 0.998019, loss: 0.000924, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19393 s, avg_samples: 56.0, ips: 288.76401 samples/s, eta: 13:04:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:54:26] ppocr INFO: epoch: [411/500], global_step: 301800, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.944820, exp_rate<=1 : 0.978631, exp_rate<=2 : 0.994652, exp_rate<=3 : 0.998285, loss: 0.001229, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17828 s, avg_samples: 56.0, ips: 314.10663 samples/s, eta: 13:03:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:54:46] ppocr INFO: epoch: [411/500], global_step: 301900, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.944088, exp_rate<=1 : 0.978459, exp_rate<=2 : 0.994800, exp_rate<=3 : 0.998243, loss: 0.001922, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19860 s, avg_samples: 56.0, ips: 281.96925 samples/s, eta: 13:03:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:55:03] ppocr INFO: epoch: [411/500], global_step: 302000, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.944619, exp_rate<=1 : 0.978855, exp_rate<=2 : 0.994975, exp_rate<=3 : 0.998361, loss: 0.002236, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16899 s, avg_samples: 56.0, ips: 331.38154 samples/s, eta: 13:03:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:55:20] ppocr INFO: epoch: [411/500], global_step: 302100, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.945210, exp_rate<=1 : 0.978805, exp_rate<=2 : 0.994931, exp_rate<=3 : 0.998345, loss: 0.001006, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17360 s, avg_samples: 56.0, ips: 322.57200 samples/s, eta: 13:02:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:55:40] ppocr INFO: epoch: [411/500], global_step: 302200, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.944430, exp_rate<=1 : 0.978362, exp_rate<=2 : 0.994948, exp_rate<=3 : 0.998347, loss: 0.001644, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19608 s, avg_samples: 56.0, ips: 285.60369 samples/s, eta: 13:02:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:55:58] ppocr INFO: epoch: [411/500], global_step: 302300, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.944183, exp_rate<=1 : 0.978734, exp_rate<=2 : 0.994987, exp_rate<=3 : 0.998361, loss: 0.001044, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18410 s, avg_samples: 56.0, ips: 304.17567 samples/s, eta: 13:02:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:56:18] ppocr INFO: epoch: [411/500], global_step: 302400, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.944515, exp_rate<=1 : 0.978946, exp_rate<=2 : 0.995072, exp_rate<=3 : 0.998323, loss: 0.000899, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19522 s, avg_samples: 56.0, ips: 286.85350 samples/s, eta: 13:01:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:56:36] ppocr INFO: epoch: [411/500], global_step: 302500, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.944319, exp_rate<=1 : 0.978642, exp_rate<=2 : 0.995026, exp_rate<=3 : 0.998349, loss: 0.001236, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18403 s, avg_samples: 56.0, ips: 304.30074 samples/s, eta: 13:01:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:56:55] ppocr INFO: epoch: [411/500], global_step: 302600, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.944516, exp_rate<=1 : 0.978760, exp_rate<=2 : 0.994986, exp_rate<=3 : 0.998275, loss: 0.001574, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18408 s, avg_samples: 56.0, ips: 304.21610 samples/s, eta: 13:01:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:57:12] ppocr INFO: epoch: [411/500], global_step: 302700, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.944275, exp_rate<=1 : 0.978606, exp_rate<=2 : 0.994898, exp_rate<=3 : 0.998201, loss: 0.001322, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17253 s, avg_samples: 56.0, ips: 324.58800 samples/s, eta: 13:00:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:57:31] ppocr INFO: epoch: [411/500], global_step: 302800, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.943685, exp_rate<=1 : 0.978342, exp_rate<=2 : 0.994882, exp_rate<=3 : 0.998229, loss: 0.001987, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18789 s, avg_samples: 56.0, ips: 298.05394 samples/s, eta: 13:00:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:57:50] ppocr INFO: epoch: [411/500], global_step: 302900, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.943008, exp_rate<=1 : 0.978023, exp_rate<=2 : 0.994793, exp_rate<=3 : 0.998235, loss: 0.001734, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18796 s, avg_samples: 56.0, ips: 297.93871 samples/s, eta: 13:00:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:58:10] ppocr INFO: epoch: [411/500], global_step: 303000, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.941752, exp_rate<=1 : 0.977541, exp_rate<=2 : 0.994731, exp_rate<=3 : 0.998192, loss: 0.001895, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19923 s, avg_samples: 56.0, ips: 281.08688 samples/s, eta: 12:59:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:58:28] ppocr INFO: epoch: [411/500], global_step: 303100, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.941923, exp_rate<=1 : 0.977631, exp_rate<=2 : 0.994715, exp_rate<=3 : 0.998201, loss: 0.000992, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18924 s, avg_samples: 56.0, ips: 295.91369 samples/s, eta: 12:59:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:58:46] ppocr INFO: epoch: [411/500], global_step: 303200, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941716, exp_rate<=1 : 0.977456, exp_rate<=2 : 0.994655, exp_rate<=3 : 0.998103, loss: 0.001782, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18012 s, avg_samples: 56.0, ips: 310.89975 samples/s, eta: 12:59:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:59:05] ppocr INFO: epoch: [411/500], global_step: 303300, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.941643, exp_rate<=1 : 0.977464, exp_rate<=2 : 0.994654, exp_rate<=3 : 0.998096, loss: 0.002031, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18628 s, avg_samples: 56.0, ips: 300.62093 samples/s, eta: 12:58:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:59:24] ppocr INFO: epoch: [411/500], global_step: 303400, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.940988, exp_rate<=1 : 0.977220, exp_rate<=2 : 0.994547, exp_rate<=3 : 0.998062, loss: 0.001070, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18376 s, avg_samples: 56.0, ips: 304.74885 samples/s, eta: 12:58:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 18:59:43] ppocr INFO: epoch: [411/500], global_step: 303500, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.940483, exp_rate<=1 : 0.976984, exp_rate<=2 : 0.994479, exp_rate<=3 : 0.998075, loss: 0.001858, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19848 s, avg_samples: 56.0, ips: 282.13821 samples/s, eta: 12:58:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:00:04] ppocr INFO: epoch: [411/500], global_step: 303600, lr: 0.000100, edit distance: 0.001326, exp_rate: 0.939707, exp_rate<=1 : 0.976599, exp_rate<=2 : 0.994299, exp_rate<=3 : 0.997995, loss: 0.001478, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20468 s, avg_samples: 56.0, ips: 273.59456 samples/s, eta: 12:58:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:00:23] ppocr INFO: epoch: [411/500], global_step: 303700, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.939552, exp_rate<=1 : 0.976584, exp_rate<=2 : 0.994307, exp_rate<=3 : 0.998006, loss: 0.002164, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18697 s, avg_samples: 56.0, ips: 299.52031 samples/s, eta: 12:57:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:00:41] ppocr INFO: epoch: [411/500], global_step: 303800, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.939251, exp_rate<=1 : 0.976525, exp_rate<=2 : 0.994288, exp_rate<=3 : 0.997959, loss: 0.002996, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18382 s, avg_samples: 56.0, ips: 304.65098 samples/s, eta: 12:57:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:01:01] ppocr INFO: epoch: [411/500], global_step: 303900, lr: 0.000100, edit distance: 0.001350, exp_rate: 0.938785, exp_rate<=1 : 0.976315, exp_rate<=2 : 0.994196, exp_rate<=3 : 0.997910, loss: 0.001584, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20261 s, avg_samples: 56.0, ips: 276.39485 samples/s, eta: 12:57:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:01:19] ppocr INFO: epoch: [411/500], global_step: 304000, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.938769, exp_rate<=1 : 0.976340, exp_rate<=2 : 0.994218, exp_rate<=3 : 0.997900, loss: 0.001651, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17764 s, avg_samples: 56.0, ips: 315.24339 samples/s, eta: 12:56:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:01:38] ppocr INFO: epoch: [411/500], global_step: 304100, lr: 0.000100, edit distance: 0.001348, exp_rate: 0.938775, exp_rate<=1 : 0.976447, exp_rate<=2 : 0.994268, exp_rate<=3 : 0.997909, loss: 0.001834, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18992 s, avg_samples: 56.0, ips: 294.85576 samples/s, eta: 12:56:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:01:46] ppocr INFO: epoch: [411/500], global_step: 304140, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.938584, exp_rate<=1 : 0.976382, exp_rate<=2 : 0.994212, exp_rate<=3 : 0.997899, loss: 0.001728, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07888 s, avg_samples: 22.4, ips: 283.95983 samples/s, eta: 12:56:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:01:48] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 19:02:00] ppocr INFO: epoch: [412/500], global_step: 304200, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.939666, exp_rate<=1 : 0.973383, exp_rate<=2 : 0.994312, exp_rate<=3 : 0.998685, loss: 0.001140, avg_reader_cost: 0.02523 s, avg_batch_cost: 0.13660 s, avg_samples: 33.6, ips: 245.97523 samples/s, eta: 12:56:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:02:19] ppocr INFO: epoch: [412/500], global_step: 304300, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.942472, exp_rate<=1 : 0.977350, exp_rate<=2 : 0.994603, exp_rate<=3 : 0.998009, loss: 0.001575, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19136 s, avg_samples: 56.0, ips: 292.63917 samples/s, eta: 12:55:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:02:38] ppocr INFO: epoch: [412/500], global_step: 304400, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.941332, exp_rate<=1 : 0.977472, exp_rate<=2 : 0.994393, exp_rate<=3 : 0.997790, loss: 0.001340, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19628 s, avg_samples: 56.0, ips: 285.31306 samples/s, eta: 12:55:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:02:57] ppocr INFO: epoch: [412/500], global_step: 304500, lr: 0.000100, edit distance: 0.001390, exp_rate: 0.940832, exp_rate<=1 : 0.977541, exp_rate<=2 : 0.994549, exp_rate<=3 : 0.998064, loss: 0.001576, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18637 s, avg_samples: 56.0, ips: 300.47740 samples/s, eta: 12:55:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:03:17] ppocr INFO: epoch: [412/500], global_step: 304600, lr: 0.000100, edit distance: 0.001360, exp_rate: 0.941312, exp_rate<=1 : 0.977674, exp_rate<=2 : 0.994643, exp_rate<=3 : 0.997885, loss: 0.001704, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20263 s, avg_samples: 56.0, ips: 276.37069 samples/s, eta: 12:54:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:03:35] ppocr INFO: epoch: [412/500], global_step: 304700, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.942107, exp_rate<=1 : 0.977626, exp_rate<=2 : 0.994672, exp_rate<=3 : 0.997994, loss: 0.001145, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17864 s, avg_samples: 56.0, ips: 313.48779 samples/s, eta: 12:54:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:03:55] ppocr INFO: epoch: [412/500], global_step: 304800, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.940609, exp_rate<=1 : 0.977070, exp_rate<=2 : 0.994390, exp_rate<=3 : 0.997976, loss: 0.001608, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19505 s, avg_samples: 56.0, ips: 287.10196 samples/s, eta: 12:54:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:04:15] ppocr INFO: epoch: [412/500], global_step: 304900, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.939470, exp_rate<=1 : 0.976758, exp_rate<=2 : 0.994221, exp_rate<=3 : 0.997882, loss: 0.001272, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19832 s, avg_samples: 56.0, ips: 282.37824 samples/s, eta: 12:53:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:04:34] ppocr INFO: epoch: [412/500], global_step: 305000, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.939811, exp_rate<=1 : 0.976846, exp_rate<=2 : 0.994216, exp_rate<=3 : 0.997895, loss: 0.001262, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18846 s, avg_samples: 56.0, ips: 297.15196 samples/s, eta: 12:53:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:04:52] ppocr INFO: epoch: [412/500], global_step: 305100, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.940407, exp_rate<=1 : 0.976760, exp_rate<=2 : 0.994336, exp_rate<=3 : 0.997933, loss: 0.000820, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18192 s, avg_samples: 56.0, ips: 307.83514 samples/s, eta: 12:53:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:05:10] ppocr INFO: epoch: [412/500], global_step: 305200, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.941175, exp_rate<=1 : 0.977031, exp_rate<=2 : 0.994339, exp_rate<=3 : 0.997939, loss: 0.001856, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18237 s, avg_samples: 56.0, ips: 307.07538 samples/s, eta: 12:52:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:05:28] ppocr INFO: epoch: [412/500], global_step: 305300, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.940461, exp_rate<=1 : 0.976726, exp_rate<=2 : 0.994406, exp_rate<=3 : 0.997968, loss: 0.002223, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18456 s, avg_samples: 56.0, ips: 303.42679 samples/s, eta: 12:52:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:05:47] ppocr INFO: epoch: [412/500], global_step: 305400, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.940991, exp_rate<=1 : 0.976809, exp_rate<=2 : 0.994431, exp_rate<=3 : 0.998029, loss: 0.001659, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18375 s, avg_samples: 56.0, ips: 304.76880 samples/s, eta: 12:52:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:06:05] ppocr INFO: epoch: [412/500], global_step: 305500, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.941552, exp_rate<=1 : 0.977171, exp_rate<=2 : 0.994547, exp_rate<=3 : 0.998145, loss: 0.001245, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17749 s, avg_samples: 56.0, ips: 315.51095 samples/s, eta: 12:51:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:06:23] ppocr INFO: epoch: [412/500], global_step: 305600, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.940264, exp_rate<=1 : 0.976654, exp_rate<=2 : 0.994212, exp_rate<=3 : 0.997945, loss: 0.001902, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18769 s, avg_samples: 56.0, ips: 298.36451 samples/s, eta: 12:51:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:06:41] ppocr INFO: epoch: [412/500], global_step: 305700, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.940125, exp_rate<=1 : 0.976762, exp_rate<=2 : 0.994261, exp_rate<=3 : 0.997980, loss: 0.001652, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17276 s, avg_samples: 56.0, ips: 324.14199 samples/s, eta: 12:51:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:07:02] ppocr INFO: epoch: [412/500], global_step: 305800, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939184, exp_rate<=1 : 0.976398, exp_rate<=2 : 0.994192, exp_rate<=3 : 0.997913, loss: 0.001492, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20852 s, avg_samples: 56.0, ips: 268.56104 samples/s, eta: 12:50:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:07:20] ppocr INFO: epoch: [412/500], global_step: 305900, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.938983, exp_rate<=1 : 0.976221, exp_rate<=2 : 0.994199, exp_rate<=3 : 0.997890, loss: 0.001749, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18458 s, avg_samples: 56.0, ips: 303.39919 samples/s, eta: 12:50:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:07:39] ppocr INFO: epoch: [412/500], global_step: 306000, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.938698, exp_rate<=1 : 0.975978, exp_rate<=2 : 0.994168, exp_rate<=3 : 0.997927, loss: 0.001830, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19318 s, avg_samples: 56.0, ips: 289.88820 samples/s, eta: 12:50:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:07:58] ppocr INFO: epoch: [412/500], global_step: 306100, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.938467, exp_rate<=1 : 0.975858, exp_rate<=2 : 0.994180, exp_rate<=3 : 0.997949, loss: 0.001020, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18487 s, avg_samples: 56.0, ips: 302.91841 samples/s, eta: 12:49:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:08:16] ppocr INFO: epoch: [412/500], global_step: 306200, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.938807, exp_rate<=1 : 0.975871, exp_rate<=2 : 0.994261, exp_rate<=3 : 0.997972, loss: 0.001607, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18337 s, avg_samples: 56.0, ips: 305.39055 samples/s, eta: 12:49:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:08:35] ppocr INFO: epoch: [412/500], global_step: 306300, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.938501, exp_rate<=1 : 0.975801, exp_rate<=2 : 0.994274, exp_rate<=3 : 0.997954, loss: 0.001983, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18999 s, avg_samples: 56.0, ips: 294.75891 samples/s, eta: 12:49:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:08:55] ppocr INFO: epoch: [412/500], global_step: 306400, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.937974, exp_rate<=1 : 0.975639, exp_rate<=2 : 0.994272, exp_rate<=3 : 0.997931, loss: 0.002008, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19216 s, avg_samples: 56.0, ips: 291.42046 samples/s, eta: 12:49:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:09:13] ppocr INFO: epoch: [412/500], global_step: 306500, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.938080, exp_rate<=1 : 0.975710, exp_rate<=2 : 0.994242, exp_rate<=3 : 0.997926, loss: 0.001506, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18698 s, avg_samples: 56.0, ips: 299.50113 samples/s, eta: 12:48:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:09:33] ppocr INFO: epoch: [412/500], global_step: 306600, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.937631, exp_rate<=1 : 0.975534, exp_rate<=2 : 0.994155, exp_rate<=3 : 0.997858, loss: 0.001525, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19488 s, avg_samples: 56.0, ips: 287.36127 samples/s, eta: 12:48:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:09:49] ppocr INFO: epoch: [412/500], global_step: 306700, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.937857, exp_rate<=1 : 0.975654, exp_rate<=2 : 0.994191, exp_rate<=3 : 0.997868, loss: 0.001388, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16697 s, avg_samples: 56.0, ips: 335.38832 samples/s, eta: 12:48:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:10:08] ppocr INFO: epoch: [412/500], global_step: 306800, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.937485, exp_rate<=1 : 0.975539, exp_rate<=2 : 0.994140, exp_rate<=3 : 0.997825, loss: 0.001609, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18959 s, avg_samples: 56.0, ips: 295.36929 samples/s, eta: 12:47:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:10:23] ppocr INFO: epoch: [412/500], global_step: 306880, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.937208, exp_rate<=1 : 0.975487, exp_rate<=2 : 0.994131, exp_rate<=3 : 0.997833, loss: 0.001496, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14998 s, avg_samples: 44.8, ips: 298.70501 samples/s, eta: 12:47:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:10:26] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 19:10:30] ppocr INFO: epoch: [413/500], global_step: 306900, lr: 0.000100, edit distance: 0.001405, exp_rate: 0.926948, exp_rate<=1 : 0.965097, exp_rate<=2 : 0.988095, exp_rate<=3 : 0.995667, loss: 0.001375, avg_reader_cost: 0.02597 s, avg_batch_cost: 0.06653 s, avg_samples: 11.2, ips: 168.34826 samples/s, eta: 12:47:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:10:48] ppocr INFO: epoch: [413/500], global_step: 307000, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.945269, exp_rate<=1 : 0.978556, exp_rate<=2 : 0.995976, exp_rate<=3 : 0.997943, loss: 0.001377, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18220 s, avg_samples: 56.0, ips: 307.34697 samples/s, eta: 12:47:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:11:08] ppocr INFO: epoch: [413/500], global_step: 307100, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.942712, exp_rate<=1 : 0.977946, exp_rate<=2 : 0.995546, exp_rate<=3 : 0.998049, loss: 0.001395, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19866 s, avg_samples: 56.0, ips: 281.88809 samples/s, eta: 12:46:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:11:29] ppocr INFO: epoch: [413/500], global_step: 307200, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.939585, exp_rate<=1 : 0.976343, exp_rate<=2 : 0.995005, exp_rate<=3 : 0.997800, loss: 0.001175, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20830 s, avg_samples: 56.0, ips: 268.83727 samples/s, eta: 12:46:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:11:48] ppocr INFO: epoch: [413/500], global_step: 307300, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.940519, exp_rate<=1 : 0.976647, exp_rate<=2 : 0.994729, exp_rate<=3 : 0.997733, loss: 0.001373, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19123 s, avg_samples: 56.0, ips: 292.84306 samples/s, eta: 12:46:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:12:07] ppocr INFO: epoch: [413/500], global_step: 307400, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.941010, exp_rate<=1 : 0.976726, exp_rate<=2 : 0.994777, exp_rate<=3 : 0.997889, loss: 0.001724, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18366 s, avg_samples: 56.0, ips: 304.90701 samples/s, eta: 12:45:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:12:26] ppocr INFO: epoch: [413/500], global_step: 307500, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.940838, exp_rate<=1 : 0.976720, exp_rate<=2 : 0.994482, exp_rate<=3 : 0.997768, loss: 0.001086, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19811 s, avg_samples: 56.0, ips: 282.66479 samples/s, eta: 12:45:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:12:47] ppocr INFO: epoch: [413/500], global_step: 307600, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.939712, exp_rate<=1 : 0.976140, exp_rate<=2 : 0.994287, exp_rate<=3 : 0.997765, loss: 0.001613, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20459 s, avg_samples: 56.0, ips: 273.71879 samples/s, eta: 12:45:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:13:06] ppocr INFO: epoch: [413/500], global_step: 307700, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.940325, exp_rate<=1 : 0.976307, exp_rate<=2 : 0.994312, exp_rate<=3 : 0.997735, loss: 0.001384, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19371 s, avg_samples: 56.0, ips: 289.08644 samples/s, eta: 12:44:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:13:24] ppocr INFO: epoch: [413/500], global_step: 307800, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.940637, exp_rate<=1 : 0.976605, exp_rate<=2 : 0.994450, exp_rate<=3 : 0.997827, loss: 0.001547, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18014 s, avg_samples: 56.0, ips: 310.87676 samples/s, eta: 12:44:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:13:42] ppocr INFO: epoch: [413/500], global_step: 307900, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.940939, exp_rate<=1 : 0.976944, exp_rate<=2 : 0.994735, exp_rate<=3 : 0.997964, loss: 0.001723, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17938 s, avg_samples: 56.0, ips: 312.18258 samples/s, eta: 12:44:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:14:00] ppocr INFO: epoch: [413/500], global_step: 308000, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.940977, exp_rate<=1 : 0.977227, exp_rate<=2 : 0.994773, exp_rate<=3 : 0.998037, loss: 0.001984, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17932 s, avg_samples: 56.0, ips: 312.28222 samples/s, eta: 12:43:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:14:19] ppocr INFO: epoch: [413/500], global_step: 308100, lr: 0.000100, edit distance: 0.001335, exp_rate: 0.941081, exp_rate<=1 : 0.977185, exp_rate<=2 : 0.994885, exp_rate<=3 : 0.998076, loss: 0.001197, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18888 s, avg_samples: 56.0, ips: 296.48088 samples/s, eta: 12:43:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:14:37] ppocr INFO: epoch: [413/500], global_step: 308200, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.941157, exp_rate<=1 : 0.977255, exp_rate<=2 : 0.994914, exp_rate<=3 : 0.998186, loss: 0.002601, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18286 s, avg_samples: 56.0, ips: 306.24764 samples/s, eta: 12:43:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:14:56] ppocr INFO: epoch: [413/500], global_step: 308300, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.940723, exp_rate<=1 : 0.977183, exp_rate<=2 : 0.994874, exp_rate<=3 : 0.998150, loss: 0.001961, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18677 s, avg_samples: 56.0, ips: 299.82877 samples/s, eta: 12:42:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:15:15] ppocr INFO: epoch: [413/500], global_step: 308400, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.939791, exp_rate<=1 : 0.976877, exp_rate<=2 : 0.994753, exp_rate<=3 : 0.998065, loss: 0.001645, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19098 s, avg_samples: 56.0, ips: 293.22872 samples/s, eta: 12:42:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:15:33] ppocr INFO: epoch: [413/500], global_step: 308500, lr: 0.000100, edit distance: 0.001344, exp_rate: 0.939854, exp_rate<=1 : 0.976877, exp_rate<=2 : 0.994710, exp_rate<=3 : 0.998083, loss: 0.001464, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17952 s, avg_samples: 56.0, ips: 311.93663 samples/s, eta: 12:42:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:15:51] ppocr INFO: epoch: [413/500], global_step: 308600, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.939365, exp_rate<=1 : 0.976718, exp_rate<=2 : 0.994687, exp_rate<=3 : 0.998079, loss: 0.001186, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17954 s, avg_samples: 56.0, ips: 311.89997 samples/s, eta: 12:41:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:16:09] ppocr INFO: epoch: [413/500], global_step: 308700, lr: 0.000100, edit distance: 0.001341, exp_rate: 0.939505, exp_rate<=1 : 0.976798, exp_rate<=2 : 0.994774, exp_rate<=3 : 0.998096, loss: 0.001916, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18058 s, avg_samples: 56.0, ips: 310.11738 samples/s, eta: 12:41:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:16:28] ppocr INFO: epoch: [413/500], global_step: 308800, lr: 0.000100, edit distance: 0.001358, exp_rate: 0.939367, exp_rate<=1 : 0.976768, exp_rate<=2 : 0.994823, exp_rate<=3 : 0.998143, loss: 0.001842, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18196 s, avg_samples: 56.0, ips: 307.76235 samples/s, eta: 12:41:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:16:47] ppocr INFO: epoch: [413/500], global_step: 308900, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.938866, exp_rate<=1 : 0.976506, exp_rate<=2 : 0.994699, exp_rate<=3 : 0.998092, loss: 0.001630, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19154 s, avg_samples: 56.0, ips: 292.37059 samples/s, eta: 12:40:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:17:06] ppocr INFO: epoch: [413/500], global_step: 309000, lr: 0.000100, edit distance: 0.001359, exp_rate: 0.939185, exp_rate<=1 : 0.976637, exp_rate<=2 : 0.994735, exp_rate<=3 : 0.998093, loss: 0.001229, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18874 s, avg_samples: 56.0, ips: 296.69862 samples/s, eta: 12:40:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:17:24] ppocr INFO: epoch: [413/500], global_step: 309100, lr: 0.000100, edit distance: 0.001354, exp_rate: 0.939474, exp_rate<=1 : 0.976578, exp_rate<=2 : 0.994667, exp_rate<=3 : 0.998029, loss: 0.001190, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18864 s, avg_samples: 56.0, ips: 296.85516 samples/s, eta: 12:40:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:17:44] ppocr INFO: epoch: [413/500], global_step: 309200, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.939323, exp_rate<=1 : 0.976461, exp_rate<=2 : 0.994666, exp_rate<=3 : 0.998031, loss: 0.001617, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19891 s, avg_samples: 56.0, ips: 281.52758 samples/s, eta: 12:40:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:18:02] ppocr INFO: epoch: [413/500], global_step: 309300, lr: 0.000100, edit distance: 0.001349, exp_rate: 0.939486, exp_rate<=1 : 0.976620, exp_rate<=2 : 0.994727, exp_rate<=3 : 0.998041, loss: 0.001912, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18058 s, avg_samples: 56.0, ips: 310.11252 samples/s, eta: 12:39:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:18:19] ppocr INFO: epoch: [413/500], global_step: 309400, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.939591, exp_rate<=1 : 0.976724, exp_rate<=2 : 0.994719, exp_rate<=3 : 0.998030, loss: 0.001473, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16945 s, avg_samples: 56.0, ips: 330.48131 samples/s, eta: 12:39:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:18:38] ppocr INFO: epoch: [413/500], global_step: 309500, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.939564, exp_rate<=1 : 0.976727, exp_rate<=2 : 0.994709, exp_rate<=3 : 0.998046, loss: 0.001872, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19023 s, avg_samples: 56.0, ips: 294.37813 samples/s, eta: 12:39:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:18:57] ppocr INFO: epoch: [413/500], global_step: 309600, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.939495, exp_rate<=1 : 0.976755, exp_rate<=2 : 0.994679, exp_rate<=3 : 0.998030, loss: 0.002310, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18561 s, avg_samples: 56.0, ips: 301.70628 samples/s, eta: 12:38:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:19:01] ppocr INFO: epoch: [413/500], global_step: 309620, lr: 0.000100, edit distance: 0.001368, exp_rate: 0.939491, exp_rate<=1 : 0.976786, exp_rate<=2 : 0.994683, exp_rate<=3 : 0.998037, loss: 0.001387, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.04090 s, avg_samples: 11.2, ips: 273.86544 samples/s, eta: 12:38:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:19:03] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 19:19:19] ppocr INFO: epoch: [414/500], global_step: 309700, lr: 0.000100, edit distance: 0.001381, exp_rate: 0.940057, exp_rate<=1 : 0.977024, exp_rate<=2 : 0.994995, exp_rate<=3 : 0.998863, loss: 0.001200, avg_reader_cost: 0.02731 s, avg_batch_cost: 0.18247 s, avg_samples: 44.8, ips: 245.51924 samples/s, eta: 12:38:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:19:39] ppocr INFO: epoch: [414/500], global_step: 309800, lr: 0.000100, edit distance: 0.001050, exp_rate: 0.945555, exp_rate<=1 : 0.978836, exp_rate<=2 : 0.996043, exp_rate<=3 : 0.999017, loss: 0.001774, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19355 s, avg_samples: 56.0, ips: 289.33084 samples/s, eta: 12:38:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:19:57] ppocr INFO: epoch: [414/500], global_step: 309900, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.945782, exp_rate<=1 : 0.979861, exp_rate<=2 : 0.996336, exp_rate<=3 : 0.998878, loss: 0.001726, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17889 s, avg_samples: 56.0, ips: 313.03837 samples/s, eta: 12:37:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:20:15] ppocr INFO: epoch: [414/500], global_step: 310000, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.946256, exp_rate<=1 : 0.980012, exp_rate<=2 : 0.996355, exp_rate<=3 : 0.999064, loss: 0.001129, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18391 s, avg_samples: 56.0, ips: 304.49154 samples/s, eta: 12:37:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:20:33] ppocr INFO: epoch: [414/500], global_step: 310100, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.946333, exp_rate<=1 : 0.979589, exp_rate<=2 : 0.996050, exp_rate<=3 : 0.998869, loss: 0.001087, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17588 s, avg_samples: 56.0, ips: 318.39159 samples/s, eta: 12:37:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:20:51] ppocr INFO: epoch: [414/500], global_step: 310200, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.944647, exp_rate<=1 : 0.979068, exp_rate<=2 : 0.995762, exp_rate<=3 : 0.998659, loss: 0.001635, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18335 s, avg_samples: 56.0, ips: 305.42573 samples/s, eta: 12:36:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:21:09] ppocr INFO: epoch: [414/500], global_step: 310300, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.944238, exp_rate<=1 : 0.979222, exp_rate<=2 : 0.995539, exp_rate<=3 : 0.998492, loss: 0.001564, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17844 s, avg_samples: 56.0, ips: 313.82580 samples/s, eta: 12:36:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:21:27] ppocr INFO: epoch: [414/500], global_step: 310400, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.943689, exp_rate<=1 : 0.978983, exp_rate<=2 : 0.995454, exp_rate<=3 : 0.998470, loss: 0.001223, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18461 s, avg_samples: 56.0, ips: 303.33591 samples/s, eta: 12:36:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:21:46] ppocr INFO: epoch: [414/500], global_step: 310500, lr: 0.000100, edit distance: 0.001270, exp_rate: 0.943468, exp_rate<=1 : 0.978999, exp_rate<=2 : 0.995587, exp_rate<=3 : 0.998508, loss: 0.001345, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18551 s, avg_samples: 56.0, ips: 301.86506 samples/s, eta: 12:35:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:22:05] ppocr INFO: epoch: [414/500], global_step: 310600, lr: 0.000100, edit distance: 0.001270, exp_rate: 0.943452, exp_rate<=1 : 0.979056, exp_rate<=2 : 0.995449, exp_rate<=3 : 0.998491, loss: 0.002281, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18548 s, avg_samples: 56.0, ips: 301.91601 samples/s, eta: 12:35:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:22:24] ppocr INFO: epoch: [414/500], global_step: 310700, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.941808, exp_rate<=1 : 0.978496, exp_rate<=2 : 0.995298, exp_rate<=3 : 0.998384, loss: 0.001470, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19236 s, avg_samples: 56.0, ips: 291.12686 samples/s, eta: 12:35:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:22:43] ppocr INFO: epoch: [414/500], global_step: 310800, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.941824, exp_rate<=1 : 0.978573, exp_rate<=2 : 0.995373, exp_rate<=3 : 0.998348, loss: 0.001449, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18968 s, avg_samples: 56.0, ips: 295.23471 samples/s, eta: 12:34:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:23:01] ppocr INFO: epoch: [414/500], global_step: 310900, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.942059, exp_rate<=1 : 0.978607, exp_rate<=2 : 0.995185, exp_rate<=3 : 0.998251, loss: 0.001267, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17807 s, avg_samples: 56.0, ips: 314.47945 samples/s, eta: 12:34:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:23:19] ppocr INFO: epoch: [414/500], global_step: 311000, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.942176, exp_rate<=1 : 0.978844, exp_rate<=2 : 0.995215, exp_rate<=3 : 0.998242, loss: 0.001869, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17989 s, avg_samples: 56.0, ips: 311.30609 samples/s, eta: 12:34:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:23:38] ppocr INFO: epoch: [414/500], global_step: 311100, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.941009, exp_rate<=1 : 0.978199, exp_rate<=2 : 0.994969, exp_rate<=3 : 0.998127, loss: 0.001991, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19727 s, avg_samples: 56.0, ips: 283.87334 samples/s, eta: 12:33:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:23:57] ppocr INFO: epoch: [414/500], global_step: 311200, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.941283, exp_rate<=1 : 0.978369, exp_rate<=2 : 0.994993, exp_rate<=3 : 0.998113, loss: 0.001364, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18553 s, avg_samples: 56.0, ips: 301.83181 samples/s, eta: 12:33:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:24:15] ppocr INFO: epoch: [414/500], global_step: 311300, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.941266, exp_rate<=1 : 0.978497, exp_rate<=2 : 0.995022, exp_rate<=3 : 0.998061, loss: 0.001180, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18278 s, avg_samples: 56.0, ips: 306.38077 samples/s, eta: 12:33:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:24:34] ppocr INFO: epoch: [414/500], global_step: 311400, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.940942, exp_rate<=1 : 0.978341, exp_rate<=2 : 0.994990, exp_rate<=3 : 0.998057, loss: 0.001755, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19226 s, avg_samples: 56.0, ips: 291.26650 samples/s, eta: 12:32:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:24:54] ppocr INFO: epoch: [414/500], global_step: 311500, lr: 0.000100, edit distance: 0.001354, exp_rate: 0.940448, exp_rate<=1 : 0.977879, exp_rate<=2 : 0.994835, exp_rate<=3 : 0.997968, loss: 0.001206, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19674 s, avg_samples: 56.0, ips: 284.64096 samples/s, eta: 12:32:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:25:13] ppocr INFO: epoch: [414/500], global_step: 311600, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.940764, exp_rate<=1 : 0.978018, exp_rate<=2 : 0.994925, exp_rate<=3 : 0.998015, loss: 0.001623, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18566 s, avg_samples: 56.0, ips: 301.63440 samples/s, eta: 12:32:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:25:32] ppocr INFO: epoch: [414/500], global_step: 311700, lr: 0.000100, edit distance: 0.001347, exp_rate: 0.940347, exp_rate<=1 : 0.977777, exp_rate<=2 : 0.994929, exp_rate<=3 : 0.998044, loss: 0.001817, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19021 s, avg_samples: 56.0, ips: 294.41325 samples/s, eta: 12:31:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:25:50] ppocr INFO: epoch: [414/500], global_step: 311800, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.940357, exp_rate<=1 : 0.977647, exp_rate<=2 : 0.994911, exp_rate<=3 : 0.998034, loss: 0.001846, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18455 s, avg_samples: 56.0, ips: 303.44119 samples/s, eta: 12:31:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:26:10] ppocr INFO: epoch: [414/500], global_step: 311900, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.939524, exp_rate<=1 : 0.977419, exp_rate<=2 : 0.994868, exp_rate<=3 : 0.998005, loss: 0.002010, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19885 s, avg_samples: 56.0, ips: 281.62425 samples/s, eta: 12:31:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:26:29] ppocr INFO: epoch: [414/500], global_step: 312000, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.939189, exp_rate<=1 : 0.977306, exp_rate<=2 : 0.994845, exp_rate<=3 : 0.997982, loss: 0.002177, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18743 s, avg_samples: 56.0, ips: 298.77346 samples/s, eta: 12:31:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:26:48] ppocr INFO: epoch: [414/500], global_step: 312100, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.938614, exp_rate<=1 : 0.977133, exp_rate<=2 : 0.994786, exp_rate<=3 : 0.997946, loss: 0.001268, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19432 s, avg_samples: 56.0, ips: 288.18114 samples/s, eta: 12:30:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:27:06] ppocr INFO: epoch: [414/500], global_step: 312200, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.938549, exp_rate<=1 : 0.977053, exp_rate<=2 : 0.994729, exp_rate<=3 : 0.997930, loss: 0.001226, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17399 s, avg_samples: 56.0, ips: 321.85218 samples/s, eta: 12:30:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:27:25] ppocr INFO: epoch: [414/500], global_step: 312300, lr: 0.000100, edit distance: 0.001373, exp_rate: 0.938619, exp_rate<=1 : 0.977049, exp_rate<=2 : 0.994724, exp_rate<=3 : 0.997954, loss: 0.001458, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18790 s, avg_samples: 56.0, ips: 298.03352 samples/s, eta: 12:30:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:27:36] ppocr INFO: epoch: [414/500], global_step: 312360, lr: 0.000100, edit distance: 0.001375, exp_rate: 0.938632, exp_rate<=1 : 0.977112, exp_rate<=2 : 0.994688, exp_rate<=3 : 0.997946, loss: 0.001288, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.10888 s, avg_samples: 33.6, ips: 308.59926 samples/s, eta: 12:29:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:27:38] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 19:27:45] ppocr INFO: epoch: [415/500], global_step: 312400, lr: 0.000100, edit distance: 0.000944, exp_rate: 0.955833, exp_rate<=1 : 0.985465, exp_rate<=2 : 0.999206, exp_rate<=3 : 1.000000, loss: 0.001667, avg_reader_cost: 0.02575 s, avg_batch_cost: 0.09694 s, avg_samples: 22.4, ips: 231.06076 samples/s, eta: 12:29:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:28:03] ppocr INFO: epoch: [415/500], global_step: 312500, lr: 0.000100, edit distance: 0.000997, exp_rate: 0.947921, exp_rate<=1 : 0.981364, exp_rate<=2 : 0.996553, exp_rate<=3 : 0.999179, loss: 0.001143, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.17951 s, avg_samples: 56.0, ips: 311.95640 samples/s, eta: 12:29:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:28:22] ppocr INFO: epoch: [415/500], global_step: 312600, lr: 0.000100, edit distance: 0.000997, exp_rate: 0.950154, exp_rate<=1 : 0.981794, exp_rate<=2 : 0.996879, exp_rate<=3 : 0.999144, loss: 0.001232, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18450 s, avg_samples: 56.0, ips: 303.52729 samples/s, eta: 12:29:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:28:40] ppocr INFO: epoch: [415/500], global_step: 312700, lr: 0.000100, edit distance: 0.001107, exp_rate: 0.948960, exp_rate<=1 : 0.981083, exp_rate<=2 : 0.996508, exp_rate<=3 : 0.998919, loss: 0.000965, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18279 s, avg_samples: 56.0, ips: 306.36338 samples/s, eta: 12:28:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:28:58] ppocr INFO: epoch: [415/500], global_step: 312800, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.949082, exp_rate<=1 : 0.980937, exp_rate<=2 : 0.996190, exp_rate<=3 : 0.998867, loss: 0.001803, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17733 s, avg_samples: 56.0, ips: 315.78842 samples/s, eta: 12:28:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:29:16] ppocr INFO: epoch: [415/500], global_step: 312900, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.948572, exp_rate<=1 : 0.981263, exp_rate<=2 : 0.996297, exp_rate<=3 : 0.998828, loss: 0.001490, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18357 s, avg_samples: 56.0, ips: 305.05391 samples/s, eta: 12:28:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:29:37] ppocr INFO: epoch: [415/500], global_step: 313000, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.946145, exp_rate<=1 : 0.980021, exp_rate<=2 : 0.995758, exp_rate<=3 : 0.998520, loss: 0.001155, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20417 s, avg_samples: 56.0, ips: 274.28151 samples/s, eta: 12:27:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:29:55] ppocr INFO: epoch: [415/500], global_step: 313100, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.946576, exp_rate<=1 : 0.980164, exp_rate<=2 : 0.995695, exp_rate<=3 : 0.998521, loss: 0.001069, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18824 s, avg_samples: 56.0, ips: 297.49002 samples/s, eta: 12:27:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:30:14] ppocr INFO: epoch: [415/500], global_step: 313200, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.945943, exp_rate<=1 : 0.979887, exp_rate<=2 : 0.995579, exp_rate<=3 : 0.998454, loss: 0.001247, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18743 s, avg_samples: 56.0, ips: 298.77505 samples/s, eta: 12:27:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:30:34] ppocr INFO: epoch: [415/500], global_step: 313300, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.945332, exp_rate<=1 : 0.979686, exp_rate<=2 : 0.995485, exp_rate<=3 : 0.998388, loss: 0.001318, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19526 s, avg_samples: 56.0, ips: 286.79164 samples/s, eta: 12:26:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:30:51] ppocr INFO: epoch: [415/500], global_step: 313400, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.945414, exp_rate<=1 : 0.979563, exp_rate<=2 : 0.995380, exp_rate<=3 : 0.998305, loss: 0.001685, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17461 s, avg_samples: 56.0, ips: 320.71361 samples/s, eta: 12:26:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:31:10] ppocr INFO: epoch: [415/500], global_step: 313500, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.945046, exp_rate<=1 : 0.979277, exp_rate<=2 : 0.995384, exp_rate<=3 : 0.998294, loss: 0.001767, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18343 s, avg_samples: 56.0, ips: 305.28968 samples/s, eta: 12:26:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:31:29] ppocr INFO: epoch: [415/500], global_step: 313600, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.944176, exp_rate<=1 : 0.978884, exp_rate<=2 : 0.995166, exp_rate<=3 : 0.998199, loss: 0.001557, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19581 s, avg_samples: 56.0, ips: 285.99679 samples/s, eta: 12:25:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:31:47] ppocr INFO: epoch: [415/500], global_step: 313700, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.943969, exp_rate<=1 : 0.978594, exp_rate<=2 : 0.995235, exp_rate<=3 : 0.998253, loss: 0.001397, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17956 s, avg_samples: 56.0, ips: 311.86505 samples/s, eta: 12:25:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:32:05] ppocr INFO: epoch: [415/500], global_step: 313800, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.943846, exp_rate<=1 : 0.978704, exp_rate<=2 : 0.995174, exp_rate<=3 : 0.998287, loss: 0.001383, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18294 s, avg_samples: 56.0, ips: 306.10520 samples/s, eta: 12:25:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:32:24] ppocr INFO: epoch: [415/500], global_step: 313900, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.943792, exp_rate<=1 : 0.978702, exp_rate<=2 : 0.995123, exp_rate<=3 : 0.998250, loss: 0.001855, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18593 s, avg_samples: 56.0, ips: 301.18753 samples/s, eta: 12:24:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:32:44] ppocr INFO: epoch: [415/500], global_step: 314000, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.942464, exp_rate<=1 : 0.977974, exp_rate<=2 : 0.995029, exp_rate<=3 : 0.998241, loss: 0.001682, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19780 s, avg_samples: 56.0, ips: 283.11004 samples/s, eta: 12:24:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:33:04] ppocr INFO: epoch: [415/500], global_step: 314100, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.941242, exp_rate<=1 : 0.977417, exp_rate<=2 : 0.994951, exp_rate<=3 : 0.998216, loss: 0.001796, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20429 s, avg_samples: 56.0, ips: 274.11415 samples/s, eta: 12:24:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:33:24] ppocr INFO: epoch: [415/500], global_step: 314200, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.940730, exp_rate<=1 : 0.977424, exp_rate<=2 : 0.994927, exp_rate<=3 : 0.998209, loss: 0.001390, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19893 s, avg_samples: 56.0, ips: 281.50439 samples/s, eta: 12:23:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:33:44] ppocr INFO: epoch: [415/500], global_step: 314300, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.940344, exp_rate<=1 : 0.977191, exp_rate<=2 : 0.994843, exp_rate<=3 : 0.998161, loss: 0.001551, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19437 s, avg_samples: 56.0, ips: 288.11123 samples/s, eta: 12:23:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:34:02] ppocr INFO: epoch: [415/500], global_step: 314400, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.939771, exp_rate<=1 : 0.977027, exp_rate<=2 : 0.994733, exp_rate<=3 : 0.998074, loss: 0.000889, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18139 s, avg_samples: 56.0, ips: 308.72604 samples/s, eta: 12:23:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:34:20] ppocr INFO: epoch: [415/500], global_step: 314500, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939937, exp_rate<=1 : 0.977131, exp_rate<=2 : 0.994762, exp_rate<=3 : 0.998075, loss: 0.001712, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17703 s, avg_samples: 56.0, ips: 316.33387 samples/s, eta: 12:22:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:34:38] ppocr INFO: epoch: [415/500], global_step: 314600, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.939420, exp_rate<=1 : 0.976920, exp_rate<=2 : 0.994782, exp_rate<=3 : 0.998078, loss: 0.001851, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18435 s, avg_samples: 56.0, ips: 303.77101 samples/s, eta: 12:22:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:34:59] ppocr INFO: epoch: [415/500], global_step: 314700, lr: 0.000100, edit distance: 0.001351, exp_rate: 0.938866, exp_rate<=1 : 0.976493, exp_rate<=2 : 0.994524, exp_rate<=3 : 0.997929, loss: 0.001174, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20604 s, avg_samples: 56.0, ips: 271.78897 samples/s, eta: 12:22:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:35:16] ppocr INFO: epoch: [415/500], global_step: 314800, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.938897, exp_rate<=1 : 0.976541, exp_rate<=2 : 0.994518, exp_rate<=3 : 0.997919, loss: 0.001927, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17580 s, avg_samples: 56.0, ips: 318.54927 samples/s, eta: 12:21:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:35:35] ppocr INFO: epoch: [415/500], global_step: 314900, lr: 0.000100, edit distance: 0.001366, exp_rate: 0.938683, exp_rate<=1 : 0.976533, exp_rate<=2 : 0.994531, exp_rate<=3 : 0.997932, loss: 0.002402, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19208 s, avg_samples: 56.0, ips: 291.55137 samples/s, eta: 12:21:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:35:53] ppocr INFO: epoch: [415/500], global_step: 315000, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.938636, exp_rate<=1 : 0.976475, exp_rate<=2 : 0.994521, exp_rate<=3 : 0.997909, loss: 0.001679, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17905 s, avg_samples: 56.0, ips: 312.76942 samples/s, eta: 12:21:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:36:11] ppocr INFO: epoch: [415/500], global_step: 315100, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.938862, exp_rate<=1 : 0.976584, exp_rate<=2 : 0.994575, exp_rate<=3 : 0.997966, loss: 0.001175, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17747 s, avg_samples: 56.0, ips: 315.54663 samples/s, eta: 12:21:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:36:13] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 19:36:14] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_415 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 19:36:32] ppocr INFO: epoch: [416/500], global_step: 315200, lr: 0.000100, edit distance: 0.001034, exp_rate: 0.957726, exp_rate<=1 : 0.984869, exp_rate<=2 : 0.997984, exp_rate<=3 : 0.999211, loss: 0.001054, avg_reader_cost: 0.03309 s, avg_batch_cost: 0.20896 s, avg_samples: 56.0, ips: 267.99381 samples/s, eta: 12:20:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:36:50] ppocr INFO: epoch: [416/500], global_step: 315300, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.954194, exp_rate<=1 : 0.983298, exp_rate<=2 : 0.997118, exp_rate<=3 : 0.999063, loss: 0.001698, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17767 s, avg_samples: 56.0, ips: 315.19352 samples/s, eta: 12:20:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:37:08] ppocr INFO: epoch: [416/500], global_step: 315400, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.950491, exp_rate<=1 : 0.981396, exp_rate<=2 : 0.996423, exp_rate<=3 : 0.999124, loss: 0.000953, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18471 s, avg_samples: 56.0, ips: 303.18102 samples/s, eta: 12:20:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:37:29] ppocr INFO: epoch: [416/500], global_step: 315500, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.947522, exp_rate<=1 : 0.980536, exp_rate<=2 : 0.996062, exp_rate<=3 : 0.998877, loss: 0.001344, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20352 s, avg_samples: 56.0, ips: 275.15750 samples/s, eta: 12:19:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:37:48] ppocr INFO: epoch: [416/500], global_step: 315600, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.947103, exp_rate<=1 : 0.980191, exp_rate<=2 : 0.995768, exp_rate<=3 : 0.998689, loss: 0.000877, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19537 s, avg_samples: 56.0, ips: 286.64284 samples/s, eta: 12:19:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:38:08] ppocr INFO: epoch: [416/500], global_step: 315700, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.946624, exp_rate<=1 : 0.980086, exp_rate<=2 : 0.995723, exp_rate<=3 : 0.998672, loss: 0.001476, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19426 s, avg_samples: 56.0, ips: 288.26616 samples/s, eta: 12:19:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:38:26] ppocr INFO: epoch: [416/500], global_step: 315800, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.945802, exp_rate<=1 : 0.980150, exp_rate<=2 : 0.995739, exp_rate<=3 : 0.998647, loss: 0.001266, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18078 s, avg_samples: 56.0, ips: 309.77218 samples/s, eta: 12:18:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:38:46] ppocr INFO: epoch: [416/500], global_step: 315900, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.943567, exp_rate<=1 : 0.978700, exp_rate<=2 : 0.995188, exp_rate<=3 : 0.998411, loss: 0.001692, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20654 s, avg_samples: 56.0, ips: 271.13508 samples/s, eta: 12:18:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:39:05] ppocr INFO: epoch: [416/500], global_step: 316000, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.943713, exp_rate<=1 : 0.978463, exp_rate<=2 : 0.995034, exp_rate<=3 : 0.998438, loss: 0.001688, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18768 s, avg_samples: 56.0, ips: 298.37248 samples/s, eta: 12:18:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:39:24] ppocr INFO: epoch: [416/500], global_step: 316100, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.943840, exp_rate<=1 : 0.978285, exp_rate<=2 : 0.995060, exp_rate<=3 : 0.998505, loss: 0.001252, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19139 s, avg_samples: 56.0, ips: 292.60024 samples/s, eta: 12:17:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:39:43] ppocr INFO: epoch: [416/500], global_step: 316200, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.943734, exp_rate<=1 : 0.978049, exp_rate<=2 : 0.994977, exp_rate<=3 : 0.998429, loss: 0.001289, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18844 s, avg_samples: 56.0, ips: 297.17607 samples/s, eta: 12:17:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:40:02] ppocr INFO: epoch: [416/500], global_step: 316300, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.943361, exp_rate<=1 : 0.977905, exp_rate<=2 : 0.994988, exp_rate<=3 : 0.998396, loss: 0.001152, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18628 s, avg_samples: 56.0, ips: 300.61699 samples/s, eta: 12:17:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:40:20] ppocr INFO: epoch: [416/500], global_step: 316400, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.943323, exp_rate<=1 : 0.977853, exp_rate<=2 : 0.994871, exp_rate<=3 : 0.998334, loss: 0.001763, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18346 s, avg_samples: 56.0, ips: 305.23884 samples/s, eta: 12:16:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:40:39] ppocr INFO: epoch: [416/500], global_step: 316500, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.943128, exp_rate<=1 : 0.977735, exp_rate<=2 : 0.994746, exp_rate<=3 : 0.998356, loss: 0.000639, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18375 s, avg_samples: 56.0, ips: 304.76358 samples/s, eta: 12:16:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:40:59] ppocr INFO: epoch: [416/500], global_step: 316600, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.942929, exp_rate<=1 : 0.977780, exp_rate<=2 : 0.994787, exp_rate<=3 : 0.998367, loss: 0.001657, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20022 s, avg_samples: 56.0, ips: 279.68706 samples/s, eta: 12:16:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:41:18] ppocr INFO: epoch: [416/500], global_step: 316700, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.943085, exp_rate<=1 : 0.977821, exp_rate<=2 : 0.994765, exp_rate<=3 : 0.998351, loss: 0.001290, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18859 s, avg_samples: 56.0, ips: 296.93555 samples/s, eta: 12:15:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:41:35] ppocr INFO: epoch: [416/500], global_step: 316800, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.943360, exp_rate<=1 : 0.977879, exp_rate<=2 : 0.994818, exp_rate<=3 : 0.998338, loss: 0.001394, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17769 s, avg_samples: 56.0, ips: 315.15684 samples/s, eta: 12:15:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:41:54] ppocr INFO: epoch: [416/500], global_step: 316900, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.943544, exp_rate<=1 : 0.977981, exp_rate<=2 : 0.994855, exp_rate<=3 : 0.998363, loss: 0.001283, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18546 s, avg_samples: 56.0, ips: 301.94748 samples/s, eta: 12:15:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:42:13] ppocr INFO: epoch: [416/500], global_step: 317000, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.943480, exp_rate<=1 : 0.977981, exp_rate<=2 : 0.994836, exp_rate<=3 : 0.998300, loss: 0.001141, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18694 s, avg_samples: 56.0, ips: 299.55819 samples/s, eta: 12:14:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:42:30] ppocr INFO: epoch: [416/500], global_step: 317100, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.943702, exp_rate<=1 : 0.978074, exp_rate<=2 : 0.994845, exp_rate<=3 : 0.998309, loss: 0.001790, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17551 s, avg_samples: 56.0, ips: 319.07461 samples/s, eta: 12:14:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:42:48] ppocr INFO: epoch: [416/500], global_step: 317200, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.943213, exp_rate<=1 : 0.977861, exp_rate<=2 : 0.994905, exp_rate<=3 : 0.998330, loss: 0.002401, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17417 s, avg_samples: 56.0, ips: 321.52075 samples/s, eta: 12:14:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:43:05] ppocr INFO: epoch: [416/500], global_step: 317300, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.942426, exp_rate<=1 : 0.977523, exp_rate<=2 : 0.994799, exp_rate<=3 : 0.998274, loss: 0.001385, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17729 s, avg_samples: 56.0, ips: 315.86684 samples/s, eta: 12:13:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:43:23] ppocr INFO: epoch: [416/500], global_step: 317400, lr: 0.000100, edit distance: 0.001277, exp_rate: 0.941658, exp_rate<=1 : 0.977251, exp_rate<=2 : 0.994691, exp_rate<=3 : 0.998222, loss: 0.001250, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17753 s, avg_samples: 56.0, ips: 315.44762 samples/s, eta: 12:13:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:43:40] ppocr INFO: epoch: [416/500], global_step: 317500, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.941356, exp_rate<=1 : 0.977027, exp_rate<=2 : 0.994709, exp_rate<=3 : 0.998215, loss: 0.001324, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16921 s, avg_samples: 56.0, ips: 330.94997 samples/s, eta: 12:13:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:44:00] ppocr INFO: epoch: [416/500], global_step: 317600, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.940567, exp_rate<=1 : 0.976583, exp_rate<=2 : 0.994465, exp_rate<=3 : 0.998105, loss: 0.001361, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20275 s, avg_samples: 56.0, ips: 276.20621 samples/s, eta: 12:12:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:44:20] ppocr INFO: epoch: [416/500], global_step: 317700, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.940150, exp_rate<=1 : 0.976402, exp_rate<=2 : 0.994471, exp_rate<=3 : 0.998097, loss: 0.001639, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19107 s, avg_samples: 56.0, ips: 293.08061 samples/s, eta: 12:12:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:44:40] ppocr INFO: epoch: [416/500], global_step: 317800, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939431, exp_rate<=1 : 0.976192, exp_rate<=2 : 0.994405, exp_rate<=3 : 0.998073, loss: 0.000929, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20464 s, avg_samples: 56.0, ips: 273.64645 samples/s, eta: 12:12:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:44:48] ppocr INFO: epoch: [416/500], global_step: 317840, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.939372, exp_rate<=1 : 0.976124, exp_rate<=2 : 0.994354, exp_rate<=3 : 0.998058, loss: 0.001602, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.08419 s, avg_samples: 22.4, ips: 266.06723 samples/s, eta: 12:12:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:44:51] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 19:45:01] ppocr INFO: epoch: [417/500], global_step: 317900, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.954473, exp_rate<=1 : 0.982764, exp_rate<=2 : 0.995578, exp_rate<=3 : 0.998713, loss: 0.000977, avg_reader_cost: 0.02542 s, avg_batch_cost: 0.12749 s, avg_samples: 33.6, ips: 263.54653 samples/s, eta: 12:12:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:45:20] ppocr INFO: epoch: [417/500], global_step: 318000, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.947136, exp_rate<=1 : 0.979888, exp_rate<=2 : 0.995242, exp_rate<=3 : 0.998398, loss: 0.001358, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18224 s, avg_samples: 56.0, ips: 307.28440 samples/s, eta: 12:11:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:45:39] ppocr INFO: epoch: [417/500], global_step: 318100, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.945358, exp_rate<=1 : 0.979107, exp_rate<=2 : 0.995416, exp_rate<=3 : 0.998485, loss: 0.001837, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19421 s, avg_samples: 56.0, ips: 288.34114 samples/s, eta: 12:11:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:45:57] ppocr INFO: epoch: [417/500], global_step: 318200, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.945404, exp_rate<=1 : 0.978915, exp_rate<=2 : 0.994823, exp_rate<=3 : 0.998287, loss: 0.001036, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17753 s, avg_samples: 56.0, ips: 315.43156 samples/s, eta: 12:11:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:46:16] ppocr INFO: epoch: [417/500], global_step: 318300, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.945912, exp_rate<=1 : 0.979084, exp_rate<=2 : 0.995124, exp_rate<=3 : 0.998296, loss: 0.001310, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19205 s, avg_samples: 56.0, ips: 291.58583 samples/s, eta: 12:10:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:46:34] ppocr INFO: epoch: [417/500], global_step: 318400, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.944142, exp_rate<=1 : 0.978883, exp_rate<=2 : 0.995151, exp_rate<=3 : 0.998313, loss: 0.001349, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18426 s, avg_samples: 56.0, ips: 303.91670 samples/s, eta: 12:10:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:46:53] ppocr INFO: epoch: [417/500], global_step: 318500, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.943409, exp_rate<=1 : 0.978365, exp_rate<=2 : 0.995184, exp_rate<=3 : 0.998308, loss: 0.002132, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18843 s, avg_samples: 56.0, ips: 297.19554 samples/s, eta: 12:10:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:47:12] ppocr INFO: epoch: [417/500], global_step: 318600, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.942468, exp_rate<=1 : 0.978256, exp_rate<=2 : 0.994972, exp_rate<=3 : 0.998187, loss: 0.001552, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18561 s, avg_samples: 56.0, ips: 301.71320 samples/s, eta: 12:09:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:47:31] ppocr INFO: epoch: [417/500], global_step: 318700, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.942353, exp_rate<=1 : 0.977929, exp_rate<=2 : 0.994883, exp_rate<=3 : 0.998090, loss: 0.001404, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18752 s, avg_samples: 56.0, ips: 298.63927 samples/s, eta: 12:09:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:47:49] ppocr INFO: epoch: [417/500], global_step: 318800, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.942688, exp_rate<=1 : 0.978150, exp_rate<=2 : 0.994983, exp_rate<=3 : 0.998122, loss: 0.000942, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18411 s, avg_samples: 56.0, ips: 304.16104 samples/s, eta: 12:09:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:48:08] ppocr INFO: epoch: [417/500], global_step: 318900, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.942778, exp_rate<=1 : 0.978230, exp_rate<=2 : 0.994893, exp_rate<=3 : 0.998060, loss: 0.001656, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18881 s, avg_samples: 56.0, ips: 296.60067 samples/s, eta: 12:08:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:48:29] ppocr INFO: epoch: [417/500], global_step: 319000, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.941848, exp_rate<=1 : 0.977970, exp_rate<=2 : 0.994842, exp_rate<=3 : 0.997998, loss: 0.001971, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20780 s, avg_samples: 56.0, ips: 269.48937 samples/s, eta: 12:08:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:48:48] ppocr INFO: epoch: [417/500], global_step: 319100, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.940455, exp_rate<=1 : 0.976941, exp_rate<=2 : 0.994420, exp_rate<=3 : 0.997722, loss: 0.001281, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19629 s, avg_samples: 56.0, ips: 285.28920 samples/s, eta: 12:08:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:49:07] ppocr INFO: epoch: [417/500], global_step: 319200, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.940648, exp_rate<=1 : 0.976949, exp_rate<=2 : 0.994407, exp_rate<=3 : 0.997697, loss: 0.000721, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18282 s, avg_samples: 56.0, ips: 306.31737 samples/s, eta: 12:07:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:49:24] ppocr INFO: epoch: [417/500], global_step: 319300, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.941283, exp_rate<=1 : 0.977343, exp_rate<=2 : 0.994605, exp_rate<=3 : 0.997815, loss: 0.001790, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17618 s, avg_samples: 56.0, ips: 317.86485 samples/s, eta: 12:07:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:49:43] ppocr INFO: epoch: [417/500], global_step: 319400, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.940939, exp_rate<=1 : 0.977203, exp_rate<=2 : 0.994424, exp_rate<=3 : 0.997820, loss: 0.001188, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18879 s, avg_samples: 56.0, ips: 296.62181 samples/s, eta: 12:07:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:50:03] ppocr INFO: epoch: [417/500], global_step: 319500, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.940944, exp_rate<=1 : 0.977154, exp_rate<=2 : 0.994452, exp_rate<=3 : 0.997759, loss: 0.001554, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19452 s, avg_samples: 56.0, ips: 287.89393 samples/s, eta: 12:06:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:50:21] ppocr INFO: epoch: [417/500], global_step: 319600, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.940993, exp_rate<=1 : 0.977152, exp_rate<=2 : 0.994362, exp_rate<=3 : 0.997759, loss: 0.001475, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18518 s, avg_samples: 56.0, ips: 302.40769 samples/s, eta: 12:06:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:50:40] ppocr INFO: epoch: [417/500], global_step: 319700, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.941017, exp_rate<=1 : 0.977034, exp_rate<=2 : 0.994395, exp_rate<=3 : 0.997799, loss: 0.001831, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18754 s, avg_samples: 56.0, ips: 298.60684 samples/s, eta: 12:06:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:51:00] ppocr INFO: epoch: [417/500], global_step: 319800, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.940683, exp_rate<=1 : 0.976974, exp_rate<=2 : 0.994370, exp_rate<=3 : 0.997794, loss: 0.001340, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19559 s, avg_samples: 56.0, ips: 286.31416 samples/s, eta: 12:05:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:51:19] ppocr INFO: epoch: [417/500], global_step: 319900, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.940665, exp_rate<=1 : 0.976958, exp_rate<=2 : 0.994352, exp_rate<=3 : 0.997836, loss: 0.001902, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19272 s, avg_samples: 56.0, ips: 290.57391 samples/s, eta: 12:05:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:51:39] ppocr INFO: epoch: [417/500], global_step: 320000, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.940043, exp_rate<=1 : 0.976739, exp_rate<=2 : 0.994316, exp_rate<=3 : 0.997847, loss: 0.002155, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19970 s, avg_samples: 56.0, ips: 280.42350 samples/s, eta: 12:05:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:51:58] ppocr INFO: epoch: [417/500], global_step: 320100, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.939369, exp_rate<=1 : 0.976423, exp_rate<=2 : 0.994267, exp_rate<=3 : 0.997799, loss: 0.001712, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19305 s, avg_samples: 56.0, ips: 290.08227 samples/s, eta: 12:05:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:52:16] ppocr INFO: epoch: [417/500], global_step: 320200, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.939277, exp_rate<=1 : 0.976523, exp_rate<=2 : 0.994296, exp_rate<=3 : 0.997797, loss: 0.001745, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17961 s, avg_samples: 56.0, ips: 311.78645 samples/s, eta: 12:04:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:52:35] ppocr INFO: epoch: [417/500], global_step: 320300, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.939220, exp_rate<=1 : 0.976551, exp_rate<=2 : 0.994372, exp_rate<=3 : 0.997828, loss: 0.002117, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18530 s, avg_samples: 56.0, ips: 302.21609 samples/s, eta: 12:04:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:52:52] ppocr INFO: epoch: [417/500], global_step: 320400, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.939087, exp_rate<=1 : 0.976572, exp_rate<=2 : 0.994465, exp_rate<=3 : 0.997858, loss: 0.001653, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17139 s, avg_samples: 56.0, ips: 326.73362 samples/s, eta: 12:04:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:53:11] ppocr INFO: epoch: [417/500], global_step: 320500, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.938920, exp_rate<=1 : 0.976415, exp_rate<=2 : 0.994428, exp_rate<=3 : 0.997823, loss: 0.001714, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19000 s, avg_samples: 56.0, ips: 294.73454 samples/s, eta: 12:03:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:53:25] ppocr INFO: epoch: [417/500], global_step: 320580, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.938789, exp_rate<=1 : 0.976382, exp_rate<=2 : 0.994439, exp_rate<=3 : 0.997842, loss: 0.001451, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14315 s, avg_samples: 44.8, ips: 312.95653 samples/s, eta: 12:03:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:53:27] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 19:53:31] ppocr INFO: epoch: [418/500], global_step: 320600, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.951637, exp_rate<=1 : 0.976149, exp_rate<=2 : 0.998512, exp_rate<=3 : 0.999405, loss: 0.001227, avg_reader_cost: 0.02613 s, avg_batch_cost: 0.05949 s, avg_samples: 11.2, ips: 188.27790 samples/s, eta: 12:03:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:53:49] ppocr INFO: epoch: [418/500], global_step: 320700, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.954128, exp_rate<=1 : 0.981288, exp_rate<=2 : 0.996204, exp_rate<=3 : 0.998291, loss: 0.001509, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18220 s, avg_samples: 56.0, ips: 307.35034 samples/s, eta: 12:03:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:54:09] ppocr INFO: epoch: [418/500], global_step: 320800, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.943652, exp_rate<=1 : 0.978597, exp_rate<=2 : 0.994910, exp_rate<=3 : 0.997805, loss: 0.001878, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19278 s, avg_samples: 56.0, ips: 290.48468 samples/s, eta: 12:02:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:54:28] ppocr INFO: epoch: [418/500], global_step: 320900, lr: 0.000100, edit distance: 0.001332, exp_rate: 0.939869, exp_rate<=1 : 0.977239, exp_rate<=2 : 0.993917, exp_rate<=3 : 0.997629, loss: 0.001205, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19328 s, avg_samples: 56.0, ips: 289.73618 samples/s, eta: 12:02:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:54:47] ppocr INFO: epoch: [418/500], global_step: 321000, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.940527, exp_rate<=1 : 0.977776, exp_rate<=2 : 0.994496, exp_rate<=3 : 0.997806, loss: 0.001732, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18425 s, avg_samples: 56.0, ips: 303.93470 samples/s, eta: 12:02:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:55:06] ppocr INFO: epoch: [418/500], global_step: 321100, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.939609, exp_rate<=1 : 0.977397, exp_rate<=2 : 0.994691, exp_rate<=3 : 0.997946, loss: 0.001997, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19931 s, avg_samples: 56.0, ips: 280.97414 samples/s, eta: 12:01:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:55:25] ppocr INFO: epoch: [418/500], global_step: 321200, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.939283, exp_rate<=1 : 0.976921, exp_rate<=2 : 0.994778, exp_rate<=3 : 0.997963, loss: 0.001554, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18457 s, avg_samples: 56.0, ips: 303.40925 samples/s, eta: 12:01:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:55:45] ppocr INFO: epoch: [418/500], global_step: 321300, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.939388, exp_rate<=1 : 0.976540, exp_rate<=2 : 0.994831, exp_rate<=3 : 0.997974, loss: 0.001532, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20200 s, avg_samples: 56.0, ips: 277.23427 samples/s, eta: 12:01:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:56:03] ppocr INFO: epoch: [418/500], global_step: 321400, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.940892, exp_rate<=1 : 0.976954, exp_rate<=2 : 0.995019, exp_rate<=3 : 0.998083, loss: 0.001205, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18061 s, avg_samples: 56.0, ips: 310.06232 samples/s, eta: 12:00:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:56:22] ppocr INFO: epoch: [418/500], global_step: 321500, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.941722, exp_rate<=1 : 0.977406, exp_rate<=2 : 0.995105, exp_rate<=3 : 0.998201, loss: 0.001290, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18627 s, avg_samples: 56.0, ips: 300.64172 samples/s, eta: 12:00:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:56:41] ppocr INFO: epoch: [418/500], global_step: 321600, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.941896, exp_rate<=1 : 0.977559, exp_rate<=2 : 0.995263, exp_rate<=3 : 0.998279, loss: 0.001645, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19357 s, avg_samples: 56.0, ips: 289.30248 samples/s, eta: 12:00:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:57:01] ppocr INFO: epoch: [418/500], global_step: 321700, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.941902, exp_rate<=1 : 0.977733, exp_rate<=2 : 0.995167, exp_rate<=3 : 0.998182, loss: 0.001099, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19558 s, avg_samples: 56.0, ips: 286.33279 samples/s, eta: 11:59:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:57:19] ppocr INFO: epoch: [418/500], global_step: 321800, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.942384, exp_rate<=1 : 0.977975, exp_rate<=2 : 0.995191, exp_rate<=3 : 0.998131, loss: 0.001238, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18147 s, avg_samples: 56.0, ips: 308.59262 samples/s, eta: 11:59:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:57:37] ppocr INFO: epoch: [418/500], global_step: 321900, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.942447, exp_rate<=1 : 0.978236, exp_rate<=2 : 0.995285, exp_rate<=3 : 0.998121, loss: 0.001900, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18271 s, avg_samples: 56.0, ips: 306.49254 samples/s, eta: 11:59:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:57:57] ppocr INFO: epoch: [418/500], global_step: 322000, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.941487, exp_rate<=1 : 0.977729, exp_rate<=2 : 0.995126, exp_rate<=3 : 0.998076, loss: 0.001234, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19707 s, avg_samples: 56.0, ips: 284.16153 samples/s, eta: 11:58:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:58:15] ppocr INFO: epoch: [418/500], global_step: 322100, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.941487, exp_rate<=1 : 0.978015, exp_rate<=2 : 0.995319, exp_rate<=3 : 0.998146, loss: 0.001654, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17931 s, avg_samples: 56.0, ips: 312.31676 samples/s, eta: 11:58:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:58:34] ppocr INFO: epoch: [418/500], global_step: 322200, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.941235, exp_rate<=1 : 0.978013, exp_rate<=2 : 0.995310, exp_rate<=3 : 0.998137, loss: 0.001136, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18759 s, avg_samples: 56.0, ips: 298.51723 samples/s, eta: 11:58:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:58:52] ppocr INFO: epoch: [418/500], global_step: 322300, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.941060, exp_rate<=1 : 0.978075, exp_rate<=2 : 0.995291, exp_rate<=3 : 0.998145, loss: 0.001795, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17787 s, avg_samples: 56.0, ips: 314.83352 samples/s, eta: 11:57:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:59:10] ppocr INFO: epoch: [418/500], global_step: 322400, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.940614, exp_rate<=1 : 0.977787, exp_rate<=2 : 0.995272, exp_rate<=3 : 0.998153, loss: 0.001678, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18119 s, avg_samples: 56.0, ips: 309.06889 samples/s, eta: 11:57:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:59:28] ppocr INFO: epoch: [418/500], global_step: 322500, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.939953, exp_rate<=1 : 0.977512, exp_rate<=2 : 0.995142, exp_rate<=3 : 0.998055, loss: 0.001713, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18772 s, avg_samples: 56.0, ips: 298.31829 samples/s, eta: 11:57:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 19:59:47] ppocr INFO: epoch: [418/500], global_step: 322600, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.939572, exp_rate<=1 : 0.977319, exp_rate<=2 : 0.995024, exp_rate<=3 : 0.997980, loss: 0.001545, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18098 s, avg_samples: 56.0, ips: 309.43176 samples/s, eta: 11:56:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:00:04] ppocr INFO: epoch: [418/500], global_step: 322700, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.939509, exp_rate<=1 : 0.977313, exp_rate<=2 : 0.995090, exp_rate<=3 : 0.998026, loss: 0.001370, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17768 s, avg_samples: 56.0, ips: 315.17636 samples/s, eta: 11:56:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:00:24] ppocr INFO: epoch: [418/500], global_step: 322800, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.939202, exp_rate<=1 : 0.977110, exp_rate<=2 : 0.995052, exp_rate<=3 : 0.998035, loss: 0.001934, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19132 s, avg_samples: 56.0, ips: 292.70484 samples/s, eta: 11:56:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:00:43] ppocr INFO: epoch: [418/500], global_step: 322900, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.938897, exp_rate<=1 : 0.976842, exp_rate<=2 : 0.994999, exp_rate<=3 : 0.998010, loss: 0.001556, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19170 s, avg_samples: 56.0, ips: 292.12362 samples/s, eta: 11:55:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:01:01] ppocr INFO: epoch: [418/500], global_step: 323000, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.939036, exp_rate<=1 : 0.976977, exp_rate<=2 : 0.995007, exp_rate<=3 : 0.998044, loss: 0.001196, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17855 s, avg_samples: 56.0, ips: 313.63231 samples/s, eta: 11:55:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:01:19] ppocr INFO: epoch: [418/500], global_step: 323100, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.939081, exp_rate<=1 : 0.977027, exp_rate<=2 : 0.994951, exp_rate<=3 : 0.998003, loss: 0.001569, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18463 s, avg_samples: 56.0, ips: 303.31256 samples/s, eta: 11:55:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:01:38] ppocr INFO: epoch: [418/500], global_step: 323200, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.939100, exp_rate<=1 : 0.977185, exp_rate<=2 : 0.994997, exp_rate<=3 : 0.998023, loss: 0.001412, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18849 s, avg_samples: 56.0, ips: 297.09373 samples/s, eta: 11:54:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:01:57] ppocr INFO: epoch: [418/500], global_step: 323300, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.939042, exp_rate<=1 : 0.977164, exp_rate<=2 : 0.995006, exp_rate<=3 : 0.998030, loss: 0.001320, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19141 s, avg_samples: 56.0, ips: 292.56176 samples/s, eta: 11:54:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:02:02] ppocr INFO: epoch: [418/500], global_step: 323320, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.938948, exp_rate<=1 : 0.977117, exp_rate<=2 : 0.994994, exp_rate<=3 : 0.998005, loss: 0.001285, avg_reader_cost: 0.00003 s, avg_batch_cost: 0.04498 s, avg_samples: 11.2, ips: 249.02127 samples/s, eta: 11:54:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:02:04] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 20:02:20] ppocr INFO: epoch: [419/500], global_step: 323400, lr: 0.000100, edit distance: 0.001047, exp_rate: 0.946194, exp_rate<=1 : 0.979330, exp_rate<=2 : 0.995400, exp_rate<=3 : 0.998366, loss: 0.001207, avg_reader_cost: 0.02569 s, avg_batch_cost: 0.18376 s, avg_samples: 44.8, ips: 243.79061 samples/s, eta: 11:54:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:02:39] ppocr INFO: epoch: [419/500], global_step: 323500, lr: 0.000100, edit distance: 0.001006, exp_rate: 0.948401, exp_rate<=1 : 0.982039, exp_rate<=2 : 0.996385, exp_rate<=3 : 0.998622, loss: 0.001262, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18500 s, avg_samples: 56.0, ips: 302.70108 samples/s, eta: 11:54:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:02:58] ppocr INFO: epoch: [419/500], global_step: 323600, lr: 0.000100, edit distance: 0.001023, exp_rate: 0.948946, exp_rate<=1 : 0.982077, exp_rate<=2 : 0.995826, exp_rate<=3 : 0.998279, loss: 0.002001, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18951 s, avg_samples: 56.0, ips: 295.50603 samples/s, eta: 11:53:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:03:16] ppocr INFO: epoch: [419/500], global_step: 323700, lr: 0.000100, edit distance: 0.001106, exp_rate: 0.947170, exp_rate<=1 : 0.981708, exp_rate<=2 : 0.996107, exp_rate<=3 : 0.998570, loss: 0.001151, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18631 s, avg_samples: 56.0, ips: 300.58107 samples/s, eta: 11:53:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:03:36] ppocr INFO: epoch: [419/500], global_step: 323800, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.943263, exp_rate<=1 : 0.979482, exp_rate<=2 : 0.995573, exp_rate<=3 : 0.998244, loss: 0.001387, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19829 s, avg_samples: 56.0, ips: 282.41380 samples/s, eta: 11:53:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:03:55] ppocr INFO: epoch: [419/500], global_step: 323900, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.943688, exp_rate<=1 : 0.979307, exp_rate<=2 : 0.995536, exp_rate<=3 : 0.998269, loss: 0.000892, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18497 s, avg_samples: 56.0, ips: 302.75812 samples/s, eta: 11:52:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:04:14] ppocr INFO: epoch: [419/500], global_step: 324000, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.944294, exp_rate<=1 : 0.979251, exp_rate<=2 : 0.995306, exp_rate<=3 : 0.998194, loss: 0.001573, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19228 s, avg_samples: 56.0, ips: 291.23671 samples/s, eta: 11:52:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:04:32] ppocr INFO: epoch: [419/500], global_step: 324100, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.944991, exp_rate<=1 : 0.979683, exp_rate<=2 : 0.995328, exp_rate<=3 : 0.998203, loss: 0.001533, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18345 s, avg_samples: 56.0, ips: 305.25528 samples/s, eta: 11:52:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:04:52] ppocr INFO: epoch: [419/500], global_step: 324200, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.943394, exp_rate<=1 : 0.979093, exp_rate<=2 : 0.995343, exp_rate<=3 : 0.998338, loss: 0.001438, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19632 s, avg_samples: 56.0, ips: 285.24905 samples/s, eta: 11:51:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:05:11] ppocr INFO: epoch: [419/500], global_step: 324300, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.943055, exp_rate<=1 : 0.979161, exp_rate<=2 : 0.995474, exp_rate<=3 : 0.998326, loss: 0.001160, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19179 s, avg_samples: 56.0, ips: 291.98454 samples/s, eta: 11:51:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:05:30] ppocr INFO: epoch: [419/500], global_step: 324400, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.942262, exp_rate<=1 : 0.978993, exp_rate<=2 : 0.995588, exp_rate<=3 : 0.998372, loss: 0.001288, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19366 s, avg_samples: 56.0, ips: 289.16217 samples/s, eta: 11:51:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:05:49] ppocr INFO: epoch: [419/500], global_step: 324500, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.942142, exp_rate<=1 : 0.978782, exp_rate<=2 : 0.995465, exp_rate<=3 : 0.998383, loss: 0.001081, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18816 s, avg_samples: 56.0, ips: 297.62200 samples/s, eta: 11:50:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:06:07] ppocr INFO: epoch: [419/500], global_step: 324600, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.942871, exp_rate<=1 : 0.979086, exp_rate<=2 : 0.995573, exp_rate<=3 : 0.998462, loss: 0.001757, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17272 s, avg_samples: 56.0, ips: 324.22016 samples/s, eta: 11:50:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:06:25] ppocr INFO: epoch: [419/500], global_step: 324700, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.942795, exp_rate<=1 : 0.979191, exp_rate<=2 : 0.995602, exp_rate<=3 : 0.998489, loss: 0.001537, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18117 s, avg_samples: 56.0, ips: 309.09910 samples/s, eta: 11:50:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:06:43] ppocr INFO: epoch: [419/500], global_step: 324800, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.942898, exp_rate<=1 : 0.979134, exp_rate<=2 : 0.995713, exp_rate<=3 : 0.998543, loss: 0.001099, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18715 s, avg_samples: 56.0, ips: 299.23079 samples/s, eta: 11:49:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:07:02] ppocr INFO: epoch: [419/500], global_step: 324900, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.942908, exp_rate<=1 : 0.979079, exp_rate<=2 : 0.995665, exp_rate<=3 : 0.998533, loss: 0.001930, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18233 s, avg_samples: 56.0, ips: 307.12998 samples/s, eta: 11:49:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:07:20] ppocr INFO: epoch: [419/500], global_step: 325000, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.942706, exp_rate<=1 : 0.978894, exp_rate<=2 : 0.995608, exp_rate<=3 : 0.998503, loss: 0.001350, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17914 s, avg_samples: 56.0, ips: 312.61065 samples/s, eta: 11:49:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:07:39] ppocr INFO: epoch: [419/500], global_step: 325100, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.941990, exp_rate<=1 : 0.978454, exp_rate<=2 : 0.995368, exp_rate<=3 : 0.998388, loss: 0.001498, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19289 s, avg_samples: 56.0, ips: 290.32600 samples/s, eta: 11:48:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:07:59] ppocr INFO: epoch: [419/500], global_step: 325200, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.941065, exp_rate<=1 : 0.977916, exp_rate<=2 : 0.995185, exp_rate<=3 : 0.998285, loss: 0.001621, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19717 s, avg_samples: 56.0, ips: 284.01992 samples/s, eta: 11:48:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:08:17] ppocr INFO: epoch: [419/500], global_step: 325300, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.941288, exp_rate<=1 : 0.978159, exp_rate<=2 : 0.995215, exp_rate<=3 : 0.998316, loss: 0.001962, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18854 s, avg_samples: 56.0, ips: 297.01775 samples/s, eta: 11:48:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:08:36] ppocr INFO: epoch: [419/500], global_step: 325400, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.940884, exp_rate<=1 : 0.977934, exp_rate<=2 : 0.995129, exp_rate<=3 : 0.998289, loss: 0.001935, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18965 s, avg_samples: 56.0, ips: 295.28160 samples/s, eta: 11:47:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:08:55] ppocr INFO: epoch: [419/500], global_step: 325500, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.940890, exp_rate<=1 : 0.977807, exp_rate<=2 : 0.995024, exp_rate<=3 : 0.998274, loss: 0.001105, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18097 s, avg_samples: 56.0, ips: 309.44558 samples/s, eta: 11:47:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:09:14] ppocr INFO: epoch: [419/500], global_step: 325600, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.940467, exp_rate<=1 : 0.977513, exp_rate<=2 : 0.994939, exp_rate<=3 : 0.998219, loss: 0.001031, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18968 s, avg_samples: 56.0, ips: 295.23604 samples/s, eta: 11:47:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:09:31] ppocr INFO: epoch: [419/500], global_step: 325700, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.940643, exp_rate<=1 : 0.977594, exp_rate<=2 : 0.994937, exp_rate<=3 : 0.998194, loss: 0.002005, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17603 s, avg_samples: 56.0, ips: 318.13241 samples/s, eta: 11:46:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:09:50] ppocr INFO: epoch: [419/500], global_step: 325800, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.940274, exp_rate<=1 : 0.977385, exp_rate<=2 : 0.994822, exp_rate<=3 : 0.998127, loss: 0.002228, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18735 s, avg_samples: 56.0, ips: 298.90017 samples/s, eta: 11:46:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:10:08] ppocr INFO: epoch: [419/500], global_step: 325900, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.939922, exp_rate<=1 : 0.977275, exp_rate<=2 : 0.994857, exp_rate<=3 : 0.998150, loss: 0.001577, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18192 s, avg_samples: 56.0, ips: 307.82581 samples/s, eta: 11:46:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:10:27] ppocr INFO: epoch: [419/500], global_step: 326000, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.939800, exp_rate<=1 : 0.977231, exp_rate<=2 : 0.994892, exp_rate<=3 : 0.998179, loss: 0.001443, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18513 s, avg_samples: 56.0, ips: 302.48784 samples/s, eta: 11:46:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:10:38] ppocr INFO: epoch: [419/500], global_step: 326060, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.939570, exp_rate<=1 : 0.976990, exp_rate<=2 : 0.994828, exp_rate<=3 : 0.998120, loss: 0.002529, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.11121 s, avg_samples: 33.6, ips: 302.11823 samples/s, eta: 11:45:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:10:40] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 20:10:48] ppocr INFO: epoch: [420/500], global_step: 326100, lr: 0.000100, edit distance: 0.001485, exp_rate: 0.931913, exp_rate<=1 : 0.976449, exp_rate<=2 : 0.997402, exp_rate<=3 : 1.000000, loss: 0.002366, avg_reader_cost: 0.02606 s, avg_batch_cost: 0.10040 s, avg_samples: 22.4, ips: 223.10328 samples/s, eta: 11:45:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:11:06] ppocr INFO: epoch: [420/500], global_step: 326200, lr: 0.000100, edit distance: 0.002541, exp_rate: 0.891499, exp_rate<=1 : 0.957878, exp_rate<=2 : 0.988232, exp_rate<=3 : 0.994800, loss: 0.002756, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17808 s, avg_samples: 56.0, ips: 314.47110 samples/s, eta: 11:45:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:11:24] ppocr INFO: epoch: [420/500], global_step: 326300, lr: 0.000100, edit distance: 0.002182, exp_rate: 0.905067, exp_rate<=1 : 0.962464, exp_rate<=2 : 0.989493, exp_rate<=3 : 0.995468, loss: 0.001879, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18497 s, avg_samples: 56.0, ips: 302.75484 samples/s, eta: 11:45:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:11:43] ppocr INFO: epoch: [420/500], global_step: 326400, lr: 0.000100, edit distance: 0.001999, exp_rate: 0.913789, exp_rate<=1 : 0.965688, exp_rate<=2 : 0.990491, exp_rate<=3 : 0.996331, loss: 0.001957, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18272 s, avg_samples: 56.0, ips: 306.48628 samples/s, eta: 11:44:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:12:01] ppocr INFO: epoch: [420/500], global_step: 326500, lr: 0.000100, edit distance: 0.001801, exp_rate: 0.921576, exp_rate<=1 : 0.969302, exp_rate<=2 : 0.992036, exp_rate<=3 : 0.996962, loss: 0.001075, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18039 s, avg_samples: 56.0, ips: 310.43522 samples/s, eta: 11:44:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:12:20] ppocr INFO: epoch: [420/500], global_step: 326600, lr: 0.000100, edit distance: 0.001682, exp_rate: 0.925731, exp_rate<=1 : 0.970698, exp_rate<=2 : 0.992558, exp_rate<=3 : 0.997161, loss: 0.001920, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19008 s, avg_samples: 56.0, ips: 294.61957 samples/s, eta: 11:44:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:12:40] ppocr INFO: epoch: [420/500], global_step: 326700, lr: 0.000100, edit distance: 0.001657, exp_rate: 0.927495, exp_rate<=1 : 0.971511, exp_rate<=2 : 0.992693, exp_rate<=3 : 0.997151, loss: 0.001493, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20602 s, avg_samples: 56.0, ips: 271.82106 samples/s, eta: 11:43:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:13:00] ppocr INFO: epoch: [420/500], global_step: 326800, lr: 0.000100, edit distance: 0.001652, exp_rate: 0.928999, exp_rate<=1 : 0.972496, exp_rate<=2 : 0.992995, exp_rate<=3 : 0.997266, loss: 0.001168, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19339 s, avg_samples: 56.0, ips: 289.56327 samples/s, eta: 11:43:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:13:19] ppocr INFO: epoch: [420/500], global_step: 326900, lr: 0.000100, edit distance: 0.001636, exp_rate: 0.930043, exp_rate<=1 : 0.973007, exp_rate<=2 : 0.992939, exp_rate<=3 : 0.997226, loss: 0.001581, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19347 s, avg_samples: 56.0, ips: 289.44420 samples/s, eta: 11:43:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:13:38] ppocr INFO: epoch: [420/500], global_step: 327000, lr: 0.000100, edit distance: 0.001631, exp_rate: 0.930728, exp_rate<=1 : 0.973186, exp_rate<=2 : 0.993065, exp_rate<=3 : 0.997327, loss: 0.002352, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18731 s, avg_samples: 56.0, ips: 298.97150 samples/s, eta: 11:42:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:13:56] ppocr INFO: epoch: [420/500], global_step: 327100, lr: 0.000100, edit distance: 0.001634, exp_rate: 0.931655, exp_rate<=1 : 0.973570, exp_rate<=2 : 0.993083, exp_rate<=3 : 0.997360, loss: 0.001588, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18263 s, avg_samples: 56.0, ips: 306.62665 samples/s, eta: 11:42:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:14:14] ppocr INFO: epoch: [420/500], global_step: 327200, lr: 0.000100, edit distance: 0.001605, exp_rate: 0.932832, exp_rate<=1 : 0.973917, exp_rate<=2 : 0.993217, exp_rate<=3 : 0.997381, loss: 0.001804, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18144 s, avg_samples: 56.0, ips: 308.64972 samples/s, eta: 11:42:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:14:32] ppocr INFO: epoch: [420/500], global_step: 327300, lr: 0.000100, edit distance: 0.001571, exp_rate: 0.934262, exp_rate<=1 : 0.974545, exp_rate<=2 : 0.993441, exp_rate<=3 : 0.997525, loss: 0.001652, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17513 s, avg_samples: 56.0, ips: 319.76505 samples/s, eta: 11:41:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:14:51] ppocr INFO: epoch: [420/500], global_step: 327400, lr: 0.000100, edit distance: 0.001553, exp_rate: 0.934450, exp_rate<=1 : 0.974741, exp_rate<=2 : 0.993544, exp_rate<=3 : 0.997598, loss: 0.000991, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18806 s, avg_samples: 56.0, ips: 297.78249 samples/s, eta: 11:41:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:15:11] ppocr INFO: epoch: [420/500], global_step: 327500, lr: 0.000100, edit distance: 0.001531, exp_rate: 0.935023, exp_rate<=1 : 0.975128, exp_rate<=2 : 0.993699, exp_rate<=3 : 0.997702, loss: 0.001613, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20022 s, avg_samples: 56.0, ips: 279.69705 samples/s, eta: 11:41:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:15:30] ppocr INFO: epoch: [420/500], global_step: 327600, lr: 0.000100, edit distance: 0.001517, exp_rate: 0.934906, exp_rate<=1 : 0.975094, exp_rate<=2 : 0.993701, exp_rate<=3 : 0.997724, loss: 0.000792, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19568 s, avg_samples: 56.0, ips: 286.18763 samples/s, eta: 11:40:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:15:47] ppocr INFO: epoch: [420/500], global_step: 327700, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.935952, exp_rate<=1 : 0.975551, exp_rate<=2 : 0.993845, exp_rate<=3 : 0.997766, loss: 0.001188, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16942 s, avg_samples: 56.0, ips: 330.54242 samples/s, eta: 11:40:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:16:06] ppocr INFO: epoch: [420/500], global_step: 327800, lr: 0.000100, edit distance: 0.001491, exp_rate: 0.935785, exp_rate<=1 : 0.975593, exp_rate<=2 : 0.993872, exp_rate<=3 : 0.997832, loss: 0.001696, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18627 s, avg_samples: 56.0, ips: 300.64010 samples/s, eta: 11:40:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:16:23] ppocr INFO: epoch: [420/500], global_step: 327900, lr: 0.000100, edit distance: 0.001482, exp_rate: 0.936535, exp_rate<=1 : 0.975861, exp_rate<=2 : 0.993942, exp_rate<=3 : 0.997825, loss: 0.001517, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17657 s, avg_samples: 56.0, ips: 317.14605 samples/s, eta: 11:39:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:16:42] ppocr INFO: epoch: [420/500], global_step: 328000, lr: 0.000100, edit distance: 0.001471, exp_rate: 0.936688, exp_rate<=1 : 0.975853, exp_rate<=2 : 0.993946, exp_rate<=3 : 0.997825, loss: 0.001070, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18999 s, avg_samples: 56.0, ips: 294.75820 samples/s, eta: 11:39:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:17:02] ppocr INFO: epoch: [420/500], global_step: 328100, lr: 0.000100, edit distance: 0.001458, exp_rate: 0.936857, exp_rate<=1 : 0.975865, exp_rate<=2 : 0.994002, exp_rate<=3 : 0.997865, loss: 0.001509, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19060 s, avg_samples: 56.0, ips: 293.80541 samples/s, eta: 11:39:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:17:22] ppocr INFO: epoch: [420/500], global_step: 328200, lr: 0.000100, edit distance: 0.001449, exp_rate: 0.936830, exp_rate<=1 : 0.975775, exp_rate<=2 : 0.994049, exp_rate<=3 : 0.997863, loss: 0.001143, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19995 s, avg_samples: 56.0, ips: 280.07581 samples/s, eta: 11:38:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:17:41] ppocr INFO: epoch: [420/500], global_step: 328300, lr: 0.000100, edit distance: 0.001447, exp_rate: 0.936798, exp_rate<=1 : 0.975713, exp_rate<=2 : 0.994012, exp_rate<=3 : 0.997854, loss: 0.001102, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19360 s, avg_samples: 56.0, ips: 289.25551 samples/s, eta: 11:38:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:17:58] ppocr INFO: epoch: [420/500], global_step: 328400, lr: 0.000100, edit distance: 0.001431, exp_rate: 0.937228, exp_rate<=1 : 0.975914, exp_rate<=2 : 0.994146, exp_rate<=3 : 0.997931, loss: 0.001898, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17112 s, avg_samples: 56.0, ips: 327.25830 samples/s, eta: 11:38:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:18:17] ppocr INFO: epoch: [420/500], global_step: 328500, lr: 0.000100, edit distance: 0.001434, exp_rate: 0.937228, exp_rate<=1 : 0.975836, exp_rate<=2 : 0.994158, exp_rate<=3 : 0.997940, loss: 0.002170, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18672 s, avg_samples: 56.0, ips: 299.91734 samples/s, eta: 11:37:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:18:36] ppocr INFO: epoch: [420/500], global_step: 328600, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.936785, exp_rate<=1 : 0.975663, exp_rate<=2 : 0.994106, exp_rate<=3 : 0.997923, loss: 0.001345, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18920 s, avg_samples: 56.0, ips: 295.98263 samples/s, eta: 11:37:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:18:56] ppocr INFO: epoch: [420/500], global_step: 328700, lr: 0.000100, edit distance: 0.001439, exp_rate: 0.936449, exp_rate<=1 : 0.975527, exp_rate<=2 : 0.994082, exp_rate<=3 : 0.997930, loss: 0.001399, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20635 s, avg_samples: 56.0, ips: 271.38277 samples/s, eta: 11:37:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:19:15] ppocr INFO: epoch: [420/500], global_step: 328800, lr: 0.000100, edit distance: 0.001429, exp_rate: 0.936749, exp_rate<=1 : 0.975629, exp_rate<=2 : 0.994154, exp_rate<=3 : 0.997967, loss: 0.001278, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18223 s, avg_samples: 56.0, ips: 307.30135 samples/s, eta: 11:37:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:19:17] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 20:19:17] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_420 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 20:19:38] ppocr INFO: epoch: [421/500], global_step: 328900, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.944135, exp_rate<=1 : 0.976283, exp_rate<=2 : 0.994754, exp_rate<=3 : 0.998588, loss: 0.001723, avg_reader_cost: 0.03286 s, avg_batch_cost: 0.23396 s, avg_samples: 56.0, ips: 239.35869 samples/s, eta: 11:36:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:19:56] ppocr INFO: epoch: [421/500], global_step: 329000, lr: 0.000100, edit distance: 0.001517, exp_rate: 0.937200, exp_rate<=1 : 0.974078, exp_rate<=2 : 0.993907, exp_rate<=3 : 0.997469, loss: 0.001198, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17942 s, avg_samples: 56.0, ips: 312.11191 samples/s, eta: 11:36:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:20:17] ppocr INFO: epoch: [421/500], global_step: 329100, lr: 0.000100, edit distance: 0.001519, exp_rate: 0.933613, exp_rate<=1 : 0.973379, exp_rate<=2 : 0.993277, exp_rate<=3 : 0.997391, loss: 0.001413, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21266 s, avg_samples: 56.0, ips: 263.33470 samples/s, eta: 11:36:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:20:37] ppocr INFO: epoch: [421/500], global_step: 329200, lr: 0.000100, edit distance: 0.001433, exp_rate: 0.936117, exp_rate<=1 : 0.975284, exp_rate<=2 : 0.993664, exp_rate<=3 : 0.997757, loss: 0.001373, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20103 s, avg_samples: 56.0, ips: 278.56882 samples/s, eta: 11:35:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:20:57] ppocr INFO: epoch: [421/500], global_step: 329300, lr: 0.000100, edit distance: 0.001378, exp_rate: 0.938432, exp_rate<=1 : 0.976468, exp_rate<=2 : 0.993920, exp_rate<=3 : 0.997957, loss: 0.001646, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19371 s, avg_samples: 56.0, ips: 289.08681 samples/s, eta: 11:35:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:21:15] ppocr INFO: epoch: [421/500], global_step: 329400, lr: 0.000100, edit distance: 0.001334, exp_rate: 0.939819, exp_rate<=1 : 0.977456, exp_rate<=2 : 0.994318, exp_rate<=3 : 0.998139, loss: 0.001716, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18068 s, avg_samples: 56.0, ips: 309.93789 samples/s, eta: 11:35:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:21:32] ppocr INFO: epoch: [421/500], global_step: 329500, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941341, exp_rate<=1 : 0.978097, exp_rate<=2 : 0.994644, exp_rate<=3 : 0.998210, loss: 0.001404, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17500 s, avg_samples: 56.0, ips: 319.99223 samples/s, eta: 11:34:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:21:51] ppocr INFO: epoch: [421/500], global_step: 329600, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.942515, exp_rate<=1 : 0.978838, exp_rate<=2 : 0.994750, exp_rate<=3 : 0.998230, loss: 0.001923, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18157 s, avg_samples: 56.0, ips: 308.41579 samples/s, eta: 11:34:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:22:10] ppocr INFO: epoch: [421/500], global_step: 329700, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.941791, exp_rate<=1 : 0.978130, exp_rate<=2 : 0.994686, exp_rate<=3 : 0.998316, loss: 0.001319, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19873 s, avg_samples: 56.0, ips: 281.78868 samples/s, eta: 11:34:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:22:28] ppocr INFO: epoch: [421/500], global_step: 329800, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.942100, exp_rate<=1 : 0.978494, exp_rate<=2 : 0.994765, exp_rate<=3 : 0.998395, loss: 0.001550, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18022 s, avg_samples: 56.0, ips: 310.72737 samples/s, eta: 11:33:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:22:47] ppocr INFO: epoch: [421/500], global_step: 329900, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.941662, exp_rate<=1 : 0.978469, exp_rate<=2 : 0.994730, exp_rate<=3 : 0.998420, loss: 0.001523, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18534 s, avg_samples: 56.0, ips: 302.15258 samples/s, eta: 11:33:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:23:06] ppocr INFO: epoch: [421/500], global_step: 330000, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.941129, exp_rate<=1 : 0.978387, exp_rate<=2 : 0.994727, exp_rate<=3 : 0.998420, loss: 0.001693, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18684 s, avg_samples: 56.0, ips: 299.71904 samples/s, eta: 11:33:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:23:24] ppocr INFO: epoch: [421/500], global_step: 330100, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.940139, exp_rate<=1 : 0.977769, exp_rate<=2 : 0.994534, exp_rate<=3 : 0.998343, loss: 0.001130, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18437 s, avg_samples: 56.0, ips: 303.73266 samples/s, eta: 11:32:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:23:43] ppocr INFO: epoch: [421/500], global_step: 330200, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.939796, exp_rate<=1 : 0.977419, exp_rate<=2 : 0.994489, exp_rate<=3 : 0.998258, loss: 0.001576, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18537 s, avg_samples: 56.0, ips: 302.10257 samples/s, eta: 11:32:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:24:02] ppocr INFO: epoch: [421/500], global_step: 330300, lr: 0.000100, edit distance: 0.001360, exp_rate: 0.939945, exp_rate<=1 : 0.977451, exp_rate<=2 : 0.994570, exp_rate<=3 : 0.998305, loss: 0.001656, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19051 s, avg_samples: 56.0, ips: 293.94657 samples/s, eta: 11:32:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:24:20] ppocr INFO: epoch: [421/500], global_step: 330400, lr: 0.000100, edit distance: 0.001367, exp_rate: 0.939602, exp_rate<=1 : 0.977322, exp_rate<=2 : 0.994547, exp_rate<=3 : 0.998363, loss: 0.002303, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18569 s, avg_samples: 56.0, ips: 301.57421 samples/s, eta: 11:31:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:24:40] ppocr INFO: epoch: [421/500], global_step: 330500, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.939482, exp_rate<=1 : 0.977226, exp_rate<=2 : 0.994594, exp_rate<=3 : 0.998400, loss: 0.001548, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19211 s, avg_samples: 56.0, ips: 291.49375 samples/s, eta: 11:31:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:24:58] ppocr INFO: epoch: [421/500], global_step: 330600, lr: 0.000100, edit distance: 0.001361, exp_rate: 0.939113, exp_rate<=1 : 0.977081, exp_rate<=2 : 0.994611, exp_rate<=3 : 0.998400, loss: 0.001495, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18152 s, avg_samples: 56.0, ips: 308.51071 samples/s, eta: 11:31:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:25:16] ppocr INFO: epoch: [421/500], global_step: 330700, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.939276, exp_rate<=1 : 0.977417, exp_rate<=2 : 0.994758, exp_rate<=3 : 0.998431, loss: 0.001392, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18643 s, avg_samples: 56.0, ips: 300.38553 samples/s, eta: 11:30:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:25:36] ppocr INFO: epoch: [421/500], global_step: 330800, lr: 0.000100, edit distance: 0.001355, exp_rate: 0.939189, exp_rate<=1 : 0.977456, exp_rate<=2 : 0.994789, exp_rate<=3 : 0.998428, loss: 0.001356, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19488 s, avg_samples: 56.0, ips: 287.35537 samples/s, eta: 11:30:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:25:54] ppocr INFO: epoch: [421/500], global_step: 330900, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.939134, exp_rate<=1 : 0.977488, exp_rate<=2 : 0.994855, exp_rate<=3 : 0.998443, loss: 0.001789, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18477 s, avg_samples: 56.0, ips: 303.07854 samples/s, eta: 11:30:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:26:13] ppocr INFO: epoch: [421/500], global_step: 331000, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.939040, exp_rate<=1 : 0.977475, exp_rate<=2 : 0.994795, exp_rate<=3 : 0.998379, loss: 0.001271, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18281 s, avg_samples: 56.0, ips: 306.32736 samples/s, eta: 11:30:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:26:31] ppocr INFO: epoch: [421/500], global_step: 331100, lr: 0.000100, edit distance: 0.001360, exp_rate: 0.938991, exp_rate<=1 : 0.977531, exp_rate<=2 : 0.994824, exp_rate<=3 : 0.998368, loss: 0.001300, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18439 s, avg_samples: 56.0, ips: 303.70827 samples/s, eta: 11:29:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:26:50] ppocr INFO: epoch: [421/500], global_step: 331200, lr: 0.000100, edit distance: 0.001362, exp_rate: 0.939039, exp_rate<=1 : 0.977510, exp_rate<=2 : 0.994846, exp_rate<=3 : 0.998395, loss: 0.001675, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18338 s, avg_samples: 56.0, ips: 305.38424 samples/s, eta: 11:29:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:27:07] ppocr INFO: epoch: [421/500], global_step: 331300, lr: 0.000100, edit distance: 0.001369, exp_rate: 0.938946, exp_rate<=1 : 0.977420, exp_rate<=2 : 0.994838, exp_rate<=3 : 0.998392, loss: 0.001490, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17597 s, avg_samples: 56.0, ips: 318.23568 samples/s, eta: 11:29:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:27:26] ppocr INFO: epoch: [421/500], global_step: 331400, lr: 0.000100, edit distance: 0.001365, exp_rate: 0.939020, exp_rate<=1 : 0.977451, exp_rate<=2 : 0.994853, exp_rate<=3 : 0.998391, loss: 0.001454, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18638 s, avg_samples: 56.0, ips: 300.46755 samples/s, eta: 11:28:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:27:44] ppocr INFO: epoch: [421/500], global_step: 331500, lr: 0.000100, edit distance: 0.001368, exp_rate: 0.939032, exp_rate<=1 : 0.977387, exp_rate<=2 : 0.994860, exp_rate<=3 : 0.998380, loss: 0.001046, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17743 s, avg_samples: 56.0, ips: 315.62016 samples/s, eta: 11:28:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:27:51] ppocr INFO: epoch: [421/500], global_step: 331540, lr: 0.000100, edit distance: 0.001371, exp_rate: 0.939109, exp_rate<=1 : 0.977425, exp_rate<=2 : 0.994872, exp_rate<=3 : 0.998376, loss: 0.001556, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07364 s, avg_samples: 22.4, ips: 304.17200 samples/s, eta: 11:28:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:27:53] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 20:28:06] ppocr INFO: epoch: [422/500], global_step: 331600, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.941544, exp_rate<=1 : 0.974805, exp_rate<=2 : 0.993924, exp_rate<=3 : 0.997587, loss: 0.001110, avg_reader_cost: 0.02554 s, avg_batch_cost: 0.14628 s, avg_samples: 33.6, ips: 229.70178 samples/s, eta: 11:28:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:28:24] ppocr INFO: epoch: [422/500], global_step: 331700, lr: 0.000100, edit distance: 0.001081, exp_rate: 0.945837, exp_rate<=1 : 0.978154, exp_rate<=2 : 0.995059, exp_rate<=3 : 0.998036, loss: 0.001415, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18340 s, avg_samples: 56.0, ips: 305.34988 samples/s, eta: 11:27:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:28:42] ppocr INFO: epoch: [422/500], global_step: 331800, lr: 0.000100, edit distance: 0.001124, exp_rate: 0.945292, exp_rate<=1 : 0.978975, exp_rate<=2 : 0.995500, exp_rate<=3 : 0.997897, loss: 0.001236, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18393 s, avg_samples: 56.0, ips: 304.46275 samples/s, eta: 11:27:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:29:01] ppocr INFO: epoch: [422/500], global_step: 331900, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.945295, exp_rate<=1 : 0.978662, exp_rate<=2 : 0.995353, exp_rate<=3 : 0.997979, loss: 0.001321, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18890 s, avg_samples: 56.0, ips: 296.45280 samples/s, eta: 11:27:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:29:19] ppocr INFO: epoch: [422/500], global_step: 332000, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.947057, exp_rate<=1 : 0.979595, exp_rate<=2 : 0.995701, exp_rate<=3 : 0.998115, loss: 0.001453, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17752 s, avg_samples: 56.0, ips: 315.44917 samples/s, eta: 11:26:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:29:37] ppocr INFO: epoch: [422/500], global_step: 332100, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.947682, exp_rate<=1 : 0.979760, exp_rate<=2 : 0.995491, exp_rate<=3 : 0.998183, loss: 0.000917, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18183 s, avg_samples: 56.0, ips: 307.97366 samples/s, eta: 11:26:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:29:56] ppocr INFO: epoch: [422/500], global_step: 332200, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.947258, exp_rate<=1 : 0.980044, exp_rate<=2 : 0.995720, exp_rate<=3 : 0.998330, loss: 0.001316, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19117 s, avg_samples: 56.0, ips: 292.92640 samples/s, eta: 11:26:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:30:15] ppocr INFO: epoch: [422/500], global_step: 332300, lr: 0.000100, edit distance: 0.001144, exp_rate: 0.946025, exp_rate<=1 : 0.979520, exp_rate<=2 : 0.995741, exp_rate<=3 : 0.998377, loss: 0.000961, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18905 s, avg_samples: 56.0, ips: 296.22444 samples/s, eta: 11:25:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:30:35] ppocr INFO: epoch: [422/500], global_step: 332400, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.945055, exp_rate<=1 : 0.978980, exp_rate<=2 : 0.995394, exp_rate<=3 : 0.998350, loss: 0.001347, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19786 s, avg_samples: 56.0, ips: 283.03547 samples/s, eta: 11:25:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:30:54] ppocr INFO: epoch: [422/500], global_step: 332500, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.944670, exp_rate<=1 : 0.978762, exp_rate<=2 : 0.995380, exp_rate<=3 : 0.998357, loss: 0.002014, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18408 s, avg_samples: 56.0, ips: 304.21100 samples/s, eta: 11:25:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:31:13] ppocr INFO: epoch: [422/500], global_step: 332600, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.944550, exp_rate<=1 : 0.978798, exp_rate<=2 : 0.995308, exp_rate<=3 : 0.998255, loss: 0.000993, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19380 s, avg_samples: 56.0, ips: 288.96468 samples/s, eta: 11:24:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:31:31] ppocr INFO: epoch: [422/500], global_step: 332700, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.944142, exp_rate<=1 : 0.978738, exp_rate<=2 : 0.995329, exp_rate<=3 : 0.998272, loss: 0.001443, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18238 s, avg_samples: 56.0, ips: 307.05158 samples/s, eta: 11:24:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:31:52] ppocr INFO: epoch: [422/500], global_step: 332800, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.943549, exp_rate<=1 : 0.978499, exp_rate<=2 : 0.995289, exp_rate<=3 : 0.998241, loss: 0.001215, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20235 s, avg_samples: 56.0, ips: 276.75085 samples/s, eta: 11:24:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:32:10] ppocr INFO: epoch: [422/500], global_step: 332900, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.943458, exp_rate<=1 : 0.978597, exp_rate<=2 : 0.995248, exp_rate<=3 : 0.998260, loss: 0.001473, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18386 s, avg_samples: 56.0, ips: 304.57491 samples/s, eta: 11:23:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:32:30] ppocr INFO: epoch: [422/500], global_step: 333000, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.942433, exp_rate<=1 : 0.978214, exp_rate<=2 : 0.995144, exp_rate<=3 : 0.998216, loss: 0.001429, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19899 s, avg_samples: 56.0, ips: 281.42124 samples/s, eta: 11:23:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:32:47] ppocr INFO: epoch: [422/500], global_step: 333100, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.942127, exp_rate<=1 : 0.978211, exp_rate<=2 : 0.995233, exp_rate<=3 : 0.998290, loss: 0.001908, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17390 s, avg_samples: 56.0, ips: 322.02087 samples/s, eta: 11:23:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:33:07] ppocr INFO: epoch: [422/500], global_step: 333200, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.941244, exp_rate<=1 : 0.977829, exp_rate<=2 : 0.995153, exp_rate<=3 : 0.998269, loss: 0.001523, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19251 s, avg_samples: 56.0, ips: 290.90146 samples/s, eta: 11:22:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:33:25] ppocr INFO: epoch: [422/500], global_step: 333300, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.941276, exp_rate<=1 : 0.977899, exp_rate<=2 : 0.995130, exp_rate<=3 : 0.998266, loss: 0.001913, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18718 s, avg_samples: 56.0, ips: 299.17688 samples/s, eta: 11:22:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:33:43] ppocr INFO: epoch: [422/500], global_step: 333400, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.941285, exp_rate<=1 : 0.977857, exp_rate<=2 : 0.995119, exp_rate<=3 : 0.998256, loss: 0.001484, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18024 s, avg_samples: 56.0, ips: 310.70446 samples/s, eta: 11:22:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:34:03] ppocr INFO: epoch: [422/500], global_step: 333500, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.940818, exp_rate<=1 : 0.977560, exp_rate<=2 : 0.995052, exp_rate<=3 : 0.998215, loss: 0.001548, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19199 s, avg_samples: 56.0, ips: 291.67463 samples/s, eta: 11:21:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:34:20] ppocr INFO: epoch: [422/500], global_step: 333600, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.940601, exp_rate<=1 : 0.977380, exp_rate<=2 : 0.995035, exp_rate<=3 : 0.998236, loss: 0.001396, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17529 s, avg_samples: 56.0, ips: 319.47453 samples/s, eta: 11:21:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:34:39] ppocr INFO: epoch: [422/500], global_step: 333700, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.940399, exp_rate<=1 : 0.977296, exp_rate<=2 : 0.994970, exp_rate<=3 : 0.998216, loss: 0.001217, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19066 s, avg_samples: 56.0, ips: 293.71457 samples/s, eta: 11:21:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:34:59] ppocr INFO: epoch: [422/500], global_step: 333800, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.939944, exp_rate<=1 : 0.976958, exp_rate<=2 : 0.994879, exp_rate<=3 : 0.998152, loss: 0.001078, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19884 s, avg_samples: 56.0, ips: 281.62796 samples/s, eta: 11:21:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:35:18] ppocr INFO: epoch: [422/500], global_step: 333900, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.939185, exp_rate<=1 : 0.976651, exp_rate<=2 : 0.994636, exp_rate<=3 : 0.998003, loss: 0.002050, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18423 s, avg_samples: 56.0, ips: 303.96408 samples/s, eta: 11:20:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:35:36] ppocr INFO: epoch: [422/500], global_step: 334000, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.938641, exp_rate<=1 : 0.976388, exp_rate<=2 : 0.994577, exp_rate<=3 : 0.998018, loss: 0.001957, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18956 s, avg_samples: 56.0, ips: 295.41953 samples/s, eta: 11:20:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:35:55] ppocr INFO: epoch: [422/500], global_step: 334100, lr: 0.000100, edit distance: 0.001359, exp_rate: 0.938419, exp_rate<=1 : 0.976248, exp_rate<=2 : 0.994516, exp_rate<=3 : 0.997988, loss: 0.001396, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18843 s, avg_samples: 56.0, ips: 297.18501 samples/s, eta: 11:20:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:36:14] ppocr INFO: epoch: [422/500], global_step: 334200, lr: 0.000100, edit distance: 0.001357, exp_rate: 0.938632, exp_rate<=1 : 0.976302, exp_rate<=2 : 0.994556, exp_rate<=3 : 0.998013, loss: 0.001282, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18393 s, avg_samples: 56.0, ips: 304.46408 samples/s, eta: 11:19:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:36:28] ppocr INFO: epoch: [422/500], global_step: 334280, lr: 0.000100, edit distance: 0.001356, exp_rate: 0.938682, exp_rate<=1 : 0.976318, exp_rate<=2 : 0.994550, exp_rate<=3 : 0.998025, loss: 0.001992, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14000 s, avg_samples: 44.8, ips: 320.00940 samples/s, eta: 11:19:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:36:30] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 20:36:34] ppocr INFO: epoch: [423/500], global_step: 334300, lr: 0.000100, edit distance: 0.001040, exp_rate: 0.939732, exp_rate<=1 : 0.984311, exp_rate<=2 : 0.997768, exp_rate<=3 : 0.998512, loss: 0.001736, avg_reader_cost: 0.02540 s, avg_batch_cost: 0.06535 s, avg_samples: 11.2, ips: 171.37753 samples/s, eta: 11:19:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:36:53] ppocr INFO: epoch: [423/500], global_step: 334400, lr: 0.000100, edit distance: 0.001080, exp_rate: 0.947640, exp_rate<=1 : 0.980770, exp_rate<=2 : 0.994505, exp_rate<=3 : 0.996606, loss: 0.000603, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19135 s, avg_samples: 56.0, ips: 292.65180 samples/s, eta: 11:19:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:37:12] ppocr INFO: epoch: [423/500], global_step: 334500, lr: 0.000100, edit distance: 0.001058, exp_rate: 0.950585, exp_rate<=1 : 0.981404, exp_rate<=2 : 0.995832, exp_rate<=3 : 0.997673, loss: 0.001421, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18591 s, avg_samples: 56.0, ips: 301.21811 samples/s, eta: 11:18:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:37:31] ppocr INFO: epoch: [423/500], global_step: 334600, lr: 0.000100, edit distance: 0.001047, exp_rate: 0.948413, exp_rate<=1 : 0.980601, exp_rate<=2 : 0.995284, exp_rate<=3 : 0.997815, loss: 0.000950, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18650 s, avg_samples: 56.0, ips: 300.27320 samples/s, eta: 11:18:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:37:50] ppocr INFO: epoch: [423/500], global_step: 334700, lr: 0.000100, edit distance: 0.001064, exp_rate: 0.945852, exp_rate<=1 : 0.979862, exp_rate<=2 : 0.995252, exp_rate<=3 : 0.997856, loss: 0.001208, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19290 s, avg_samples: 56.0, ips: 290.30864 samples/s, eta: 11:18:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:38:08] ppocr INFO: epoch: [423/500], global_step: 334800, lr: 0.000100, edit distance: 0.001079, exp_rate: 0.946446, exp_rate<=1 : 0.980674, exp_rate<=2 : 0.995409, exp_rate<=3 : 0.998049, loss: 0.001479, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17911 s, avg_samples: 56.0, ips: 312.66073 samples/s, eta: 11:17:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:38:26] ppocr INFO: epoch: [423/500], global_step: 334900, lr: 0.000100, edit distance: 0.001105, exp_rate: 0.946297, exp_rate<=1 : 0.980230, exp_rate<=2 : 0.995275, exp_rate<=3 : 0.997953, loss: 0.002094, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18341 s, avg_samples: 56.0, ips: 305.32044 samples/s, eta: 11:17:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:38:46] ppocr INFO: epoch: [423/500], global_step: 335000, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.944495, exp_rate<=1 : 0.979428, exp_rate<=2 : 0.995300, exp_rate<=3 : 0.998016, loss: 0.001198, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19212 s, avg_samples: 56.0, ips: 291.48307 samples/s, eta: 11:17:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:39:06] ppocr INFO: epoch: [423/500], global_step: 335100, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.942683, exp_rate<=1 : 0.978675, exp_rate<=2 : 0.994964, exp_rate<=3 : 0.997797, loss: 0.001365, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20470 s, avg_samples: 56.0, ips: 273.56879 samples/s, eta: 11:16:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:39:25] ppocr INFO: epoch: [423/500], global_step: 335200, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.943073, exp_rate<=1 : 0.978749, exp_rate<=2 : 0.995077, exp_rate<=3 : 0.997893, loss: 0.001374, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19111 s, avg_samples: 56.0, ips: 293.01874 samples/s, eta: 11:16:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:39:44] ppocr INFO: epoch: [423/500], global_step: 335300, lr: 0.000100, edit distance: 0.001157, exp_rate: 0.943273, exp_rate<=1 : 0.979020, exp_rate<=2 : 0.995176, exp_rate<=3 : 0.997952, loss: 0.001265, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18732 s, avg_samples: 56.0, ips: 298.95353 samples/s, eta: 11:16:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:40:01] ppocr INFO: epoch: [423/500], global_step: 335400, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.944021, exp_rate<=1 : 0.979385, exp_rate<=2 : 0.995256, exp_rate<=3 : 0.998038, loss: 0.000979, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17108 s, avg_samples: 56.0, ips: 327.32761 samples/s, eta: 11:15:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:40:20] ppocr INFO: epoch: [423/500], global_step: 335500, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.944150, exp_rate<=1 : 0.979353, exp_rate<=2 : 0.995264, exp_rate<=3 : 0.998028, loss: 0.001219, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18742 s, avg_samples: 56.0, ips: 298.79055 samples/s, eta: 11:15:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:40:38] ppocr INFO: epoch: [423/500], global_step: 335600, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.944386, exp_rate<=1 : 0.979407, exp_rate<=2 : 0.995318, exp_rate<=3 : 0.998096, loss: 0.000962, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17951 s, avg_samples: 56.0, ips: 311.96625 samples/s, eta: 11:15:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:40:57] ppocr INFO: epoch: [423/500], global_step: 335700, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.943972, exp_rate<=1 : 0.979222, exp_rate<=2 : 0.995354, exp_rate<=3 : 0.998034, loss: 0.001209, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19119 s, avg_samples: 56.0, ips: 292.90857 samples/s, eta: 11:14:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:41:15] ppocr INFO: epoch: [423/500], global_step: 335800, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.943928, exp_rate<=1 : 0.979294, exp_rate<=2 : 0.995343, exp_rate<=3 : 0.998043, loss: 0.001275, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17978 s, avg_samples: 56.0, ips: 311.49980 samples/s, eta: 11:14:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:41:34] ppocr INFO: epoch: [423/500], global_step: 335900, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.944020, exp_rate<=1 : 0.979272, exp_rate<=2 : 0.995375, exp_rate<=3 : 0.998083, loss: 0.001707, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19164 s, avg_samples: 56.0, ips: 292.21157 samples/s, eta: 11:14:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:41:53] ppocr INFO: epoch: [423/500], global_step: 336000, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.943855, exp_rate<=1 : 0.979179, exp_rate<=2 : 0.995446, exp_rate<=3 : 0.998145, loss: 0.001473, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19203 s, avg_samples: 56.0, ips: 291.62627 samples/s, eta: 11:13:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:42:13] ppocr INFO: epoch: [423/500], global_step: 336100, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.943282, exp_rate<=1 : 0.978901, exp_rate<=2 : 0.995330, exp_rate<=3 : 0.998123, loss: 0.001640, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20123 s, avg_samples: 56.0, ips: 278.28342 samples/s, eta: 11:13:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:42:31] ppocr INFO: epoch: [423/500], global_step: 336200, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.943246, exp_rate<=1 : 0.978922, exp_rate<=2 : 0.995280, exp_rate<=3 : 0.998121, loss: 0.001002, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17538 s, avg_samples: 56.0, ips: 319.29931 samples/s, eta: 11:13:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:42:50] ppocr INFO: epoch: [423/500], global_step: 336300, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.942486, exp_rate<=1 : 0.978487, exp_rate<=2 : 0.995172, exp_rate<=3 : 0.998082, loss: 0.001439, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19207 s, avg_samples: 56.0, ips: 291.56196 samples/s, eta: 11:13:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:43:08] ppocr INFO: epoch: [423/500], global_step: 336400, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.942706, exp_rate<=1 : 0.978467, exp_rate<=2 : 0.995172, exp_rate<=3 : 0.998105, loss: 0.001227, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17387 s, avg_samples: 56.0, ips: 322.08012 samples/s, eta: 11:12:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:43:26] ppocr INFO: epoch: [423/500], global_step: 336500, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.942638, exp_rate<=1 : 0.978497, exp_rate<=2 : 0.995169, exp_rate<=3 : 0.998111, loss: 0.001330, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18170 s, avg_samples: 56.0, ips: 308.20445 samples/s, eta: 11:12:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:43:45] ppocr INFO: epoch: [423/500], global_step: 336600, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.942226, exp_rate<=1 : 0.978365, exp_rate<=2 : 0.995111, exp_rate<=3 : 0.998063, loss: 0.001429, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18748 s, avg_samples: 56.0, ips: 298.69940 samples/s, eta: 11:12:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:44:02] ppocr INFO: epoch: [423/500], global_step: 336700, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.942533, exp_rate<=1 : 0.978498, exp_rate<=2 : 0.995141, exp_rate<=3 : 0.998069, loss: 0.001036, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16852 s, avg_samples: 56.0, ips: 332.30211 samples/s, eta: 11:11:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:44:22] ppocr INFO: epoch: [423/500], global_step: 336800, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.942375, exp_rate<=1 : 0.978384, exp_rate<=2 : 0.995081, exp_rate<=3 : 0.998051, loss: 0.001412, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20732 s, avg_samples: 56.0, ips: 270.11364 samples/s, eta: 11:11:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:44:42] ppocr INFO: epoch: [423/500], global_step: 336900, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.942169, exp_rate<=1 : 0.978339, exp_rate<=2 : 0.995101, exp_rate<=3 : 0.998050, loss: 0.002042, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19387 s, avg_samples: 56.0, ips: 288.85457 samples/s, eta: 11:11:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:45:00] ppocr INFO: epoch: [423/500], global_step: 337000, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.941948, exp_rate<=1 : 0.978237, exp_rate<=2 : 0.995036, exp_rate<=3 : 0.998036, loss: 0.001185, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18645 s, avg_samples: 56.0, ips: 300.35001 samples/s, eta: 11:10:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:45:04] ppocr INFO: epoch: [423/500], global_step: 337020, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.941927, exp_rate<=1 : 0.978196, exp_rate<=2 : 0.995024, exp_rate<=3 : 0.998039, loss: 0.001706, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03492 s, avg_samples: 11.2, ips: 320.77442 samples/s, eta: 11:10:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:45:06] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 20:45:22] ppocr INFO: epoch: [424/500], global_step: 337100, lr: 0.000100, edit distance: 0.001077, exp_rate: 0.948596, exp_rate<=1 : 0.981882, exp_rate<=2 : 0.995093, exp_rate<=3 : 0.998086, loss: 0.001699, avg_reader_cost: 0.02489 s, avg_batch_cost: 0.18106 s, avg_samples: 44.8, ips: 247.43178 samples/s, eta: 11:10:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:45:42] ppocr INFO: epoch: [424/500], global_step: 337200, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.945917, exp_rate<=1 : 0.979108, exp_rate<=2 : 0.994822, exp_rate<=3 : 0.997711, loss: 0.001177, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20277 s, avg_samples: 56.0, ips: 276.17830 samples/s, eta: 11:10:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:46:02] ppocr INFO: epoch: [424/500], global_step: 337300, lr: 0.000100, edit distance: 0.001071, exp_rate: 0.950138, exp_rate<=1 : 0.981323, exp_rate<=2 : 0.995327, exp_rate<=3 : 0.997947, loss: 0.001368, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19553 s, avg_samples: 56.0, ips: 286.39623 samples/s, eta: 11:09:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:46:20] ppocr INFO: epoch: [424/500], global_step: 337400, lr: 0.000100, edit distance: 0.001087, exp_rate: 0.949286, exp_rate<=1 : 0.981415, exp_rate<=2 : 0.995620, exp_rate<=3 : 0.998296, loss: 0.001142, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17848 s, avg_samples: 56.0, ips: 313.76130 samples/s, eta: 11:09:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:46:39] ppocr INFO: epoch: [424/500], global_step: 337500, lr: 0.000100, edit distance: 0.001116, exp_rate: 0.947692, exp_rate<=1 : 0.980994, exp_rate<=2 : 0.995698, exp_rate<=3 : 0.998409, loss: 0.001684, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19563 s, avg_samples: 56.0, ips: 286.26162 samples/s, eta: 11:09:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:47:00] ppocr INFO: epoch: [424/500], global_step: 337600, lr: 0.000100, edit distance: 0.001107, exp_rate: 0.945539, exp_rate<=1 : 0.979905, exp_rate<=2 : 0.995334, exp_rate<=3 : 0.998263, loss: 0.000892, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21095 s, avg_samples: 56.0, ips: 265.46040 samples/s, eta: 11:08:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:47:20] ppocr INFO: epoch: [424/500], global_step: 337700, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.944458, exp_rate<=1 : 0.979330, exp_rate<=2 : 0.995329, exp_rate<=3 : 0.998269, loss: 0.001440, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19138 s, avg_samples: 56.0, ips: 292.61119 samples/s, eta: 11:08:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:47:37] ppocr INFO: epoch: [424/500], global_step: 337800, lr: 0.000100, edit distance: 0.001106, exp_rate: 0.944557, exp_rate<=1 : 0.979398, exp_rate<=2 : 0.995411, exp_rate<=3 : 0.998347, loss: 0.000772, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17789 s, avg_samples: 56.0, ips: 314.79605 samples/s, eta: 11:08:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:47:56] ppocr INFO: epoch: [424/500], global_step: 337900, lr: 0.000100, edit distance: 0.001100, exp_rate: 0.944763, exp_rate<=1 : 0.979300, exp_rate<=2 : 0.995339, exp_rate<=3 : 0.998266, loss: 0.001306, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18706 s, avg_samples: 56.0, ips: 299.36227 samples/s, eta: 11:07:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:48:16] ppocr INFO: epoch: [424/500], global_step: 338000, lr: 0.000100, edit distance: 0.001099, exp_rate: 0.944444, exp_rate<=1 : 0.979176, exp_rate<=2 : 0.995310, exp_rate<=3 : 0.998243, loss: 0.001447, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19428 s, avg_samples: 56.0, ips: 288.24467 samples/s, eta: 11:07:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:48:35] ppocr INFO: epoch: [424/500], global_step: 338100, lr: 0.000100, edit distance: 0.001101, exp_rate: 0.944738, exp_rate<=1 : 0.979114, exp_rate<=2 : 0.995301, exp_rate<=3 : 0.998255, loss: 0.001273, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19143 s, avg_samples: 56.0, ips: 292.53514 samples/s, eta: 11:07:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:48:52] ppocr INFO: epoch: [424/500], global_step: 338200, lr: 0.000100, edit distance: 0.001126, exp_rate: 0.944683, exp_rate<=1 : 0.979070, exp_rate<=2 : 0.995352, exp_rate<=3 : 0.998299, loss: 0.001222, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17655 s, avg_samples: 56.0, ips: 317.18943 samples/s, eta: 11:06:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:49:10] ppocr INFO: epoch: [424/500], global_step: 338300, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.944516, exp_rate<=1 : 0.979144, exp_rate<=2 : 0.995126, exp_rate<=3 : 0.998266, loss: 0.001417, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17492 s, avg_samples: 56.0, ips: 320.14230 samples/s, eta: 11:06:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:49:29] ppocr INFO: epoch: [424/500], global_step: 338400, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.943389, exp_rate<=1 : 0.978664, exp_rate<=2 : 0.994919, exp_rate<=3 : 0.998176, loss: 0.001553, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18941 s, avg_samples: 56.0, ips: 295.65434 samples/s, eta: 11:06:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:49:47] ppocr INFO: epoch: [424/500], global_step: 338500, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942878, exp_rate<=1 : 0.978397, exp_rate<=2 : 0.994985, exp_rate<=3 : 0.998217, loss: 0.001272, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18085 s, avg_samples: 56.0, ips: 309.65073 samples/s, eta: 11:05:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:50:06] ppocr INFO: epoch: [424/500], global_step: 338600, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.941823, exp_rate<=1 : 0.977927, exp_rate<=2 : 0.994878, exp_rate<=3 : 0.998193, loss: 0.001418, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18773 s, avg_samples: 56.0, ips: 298.29677 samples/s, eta: 11:05:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:50:25] ppocr INFO: epoch: [424/500], global_step: 338700, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.941437, exp_rate<=1 : 0.977705, exp_rate<=2 : 0.994799, exp_rate<=3 : 0.998132, loss: 0.001105, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19108 s, avg_samples: 56.0, ips: 293.07518 samples/s, eta: 11:05:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:50:43] ppocr INFO: epoch: [424/500], global_step: 338800, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.941198, exp_rate<=1 : 0.977567, exp_rate<=2 : 0.994779, exp_rate<=3 : 0.998148, loss: 0.001958, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17817 s, avg_samples: 56.0, ips: 314.30191 samples/s, eta: 11:05:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:51:00] ppocr INFO: epoch: [424/500], global_step: 338900, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.941232, exp_rate<=1 : 0.977621, exp_rate<=2 : 0.994833, exp_rate<=3 : 0.998174, loss: 0.001532, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17361 s, avg_samples: 56.0, ips: 322.56964 samples/s, eta: 11:04:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:51:19] ppocr INFO: epoch: [424/500], global_step: 339000, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.940922, exp_rate<=1 : 0.977399, exp_rate<=2 : 0.994785, exp_rate<=3 : 0.998169, loss: 0.001859, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18543 s, avg_samples: 56.0, ips: 302.00709 samples/s, eta: 11:04:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:51:36] ppocr INFO: epoch: [424/500], global_step: 339100, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.940653, exp_rate<=1 : 0.977310, exp_rate<=2 : 0.994799, exp_rate<=3 : 0.998172, loss: 0.001643, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17519 s, avg_samples: 56.0, ips: 319.64570 samples/s, eta: 11:04:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:51:54] ppocr INFO: epoch: [424/500], global_step: 339200, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.940504, exp_rate<=1 : 0.977144, exp_rate<=2 : 0.994679, exp_rate<=3 : 0.998178, loss: 0.001449, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17908 s, avg_samples: 56.0, ips: 312.71727 samples/s, eta: 11:03:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:52:12] ppocr INFO: epoch: [424/500], global_step: 339300, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.940703, exp_rate<=1 : 0.977311, exp_rate<=2 : 0.994723, exp_rate<=3 : 0.998223, loss: 0.001497, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17729 s, avg_samples: 56.0, ips: 315.86284 samples/s, eta: 11:03:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:52:30] ppocr INFO: epoch: [424/500], global_step: 339400, lr: 0.000100, edit distance: 0.001318, exp_rate: 0.940395, exp_rate<=1 : 0.977131, exp_rate<=2 : 0.994621, exp_rate<=3 : 0.998200, loss: 0.001629, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18541 s, avg_samples: 56.0, ips: 302.03410 samples/s, eta: 11:03:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:52:50] ppocr INFO: epoch: [424/500], global_step: 339500, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.939938, exp_rate<=1 : 0.976914, exp_rate<=2 : 0.994563, exp_rate<=3 : 0.998194, loss: 0.001576, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19938 s, avg_samples: 56.0, ips: 280.87747 samples/s, eta: 11:02:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:53:11] ppocr INFO: epoch: [424/500], global_step: 339600, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.939650, exp_rate<=1 : 0.976773, exp_rate<=2 : 0.994491, exp_rate<=3 : 0.998173, loss: 0.001605, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20334 s, avg_samples: 56.0, ips: 275.40619 samples/s, eta: 11:02:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:53:30] ppocr INFO: epoch: [424/500], global_step: 339700, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.939357, exp_rate<=1 : 0.976579, exp_rate<=2 : 0.994441, exp_rate<=3 : 0.998161, loss: 0.000812, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19114 s, avg_samples: 56.0, ips: 292.97467 samples/s, eta: 11:02:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:53:41] ppocr INFO: epoch: [424/500], global_step: 339760, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.939482, exp_rate<=1 : 0.976646, exp_rate<=2 : 0.994414, exp_rate<=3 : 0.998145, loss: 0.001611, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11321 s, avg_samples: 33.6, ips: 296.79019 samples/s, eta: 11:01:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:53:43] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 20:53:51] ppocr INFO: epoch: [425/500], global_step: 339800, lr: 0.000100, edit distance: 0.001330, exp_rate: 0.955717, exp_rate<=1 : 0.981349, exp_rate<=2 : 0.996030, exp_rate<=3 : 0.999206, loss: 0.001779, avg_reader_cost: 0.02516 s, avg_batch_cost: 0.09747 s, avg_samples: 22.4, ips: 229.81232 samples/s, eta: 11:01:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:54:10] ppocr INFO: epoch: [425/500], global_step: 339900, lr: 0.000100, edit distance: 0.001427, exp_rate: 0.941895, exp_rate<=1 : 0.978368, exp_rate<=2 : 0.993519, exp_rate<=3 : 0.997668, loss: 0.001334, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18755 s, avg_samples: 56.0, ips: 298.58529 samples/s, eta: 11:01:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:54:28] ppocr INFO: epoch: [425/500], global_step: 340000, lr: 0.000100, edit distance: 0.001327, exp_rate: 0.942022, exp_rate<=1 : 0.977248, exp_rate<=2 : 0.994386, exp_rate<=3 : 0.997970, loss: 0.002327, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18381 s, avg_samples: 56.0, ips: 304.65983 samples/s, eta: 11:01:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:54:47] ppocr INFO: epoch: [425/500], global_step: 340100, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.938756, exp_rate<=1 : 0.976524, exp_rate<=2 : 0.994408, exp_rate<=3 : 0.998109, loss: 0.001394, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18948 s, avg_samples: 56.0, ips: 295.54096 samples/s, eta: 11:00:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:55:05] ppocr INFO: epoch: [425/500], global_step: 340200, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.941510, exp_rate<=1 : 0.977859, exp_rate<=2 : 0.994770, exp_rate<=3 : 0.998273, loss: 0.001554, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18349 s, avg_samples: 56.0, ips: 305.19141 samples/s, eta: 11:00:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:55:24] ppocr INFO: epoch: [425/500], global_step: 340300, lr: 0.000100, edit distance: 0.001345, exp_rate: 0.940403, exp_rate<=1 : 0.977695, exp_rate<=2 : 0.994683, exp_rate<=3 : 0.998182, loss: 0.001692, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18772 s, avg_samples: 56.0, ips: 298.31179 samples/s, eta: 11:00:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:55:43] ppocr INFO: epoch: [425/500], global_step: 340400, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.941054, exp_rate<=1 : 0.978036, exp_rate<=2 : 0.994956, exp_rate<=3 : 0.998312, loss: 0.001414, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18834 s, avg_samples: 56.0, ips: 297.33430 samples/s, eta: 10:59:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:56:01] ppocr INFO: epoch: [425/500], global_step: 340500, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.941650, exp_rate<=1 : 0.978191, exp_rate<=2 : 0.995023, exp_rate<=3 : 0.998311, loss: 0.001298, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17388 s, avg_samples: 56.0, ips: 322.06337 samples/s, eta: 10:59:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:56:19] ppocr INFO: epoch: [425/500], global_step: 340600, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.942254, exp_rate<=1 : 0.978283, exp_rate<=2 : 0.995055, exp_rate<=3 : 0.998280, loss: 0.001061, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18365 s, avg_samples: 56.0, ips: 304.92448 samples/s, eta: 10:59:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:56:38] ppocr INFO: epoch: [425/500], global_step: 340700, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.941689, exp_rate<=1 : 0.977898, exp_rate<=2 : 0.995097, exp_rate<=3 : 0.998293, loss: 0.001110, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18944 s, avg_samples: 56.0, ips: 295.60728 samples/s, eta: 10:58:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:56:57] ppocr INFO: epoch: [425/500], global_step: 340800, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.941146, exp_rate<=1 : 0.977661, exp_rate<=2 : 0.994837, exp_rate<=3 : 0.998138, loss: 0.001405, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19099 s, avg_samples: 56.0, ips: 293.20809 samples/s, eta: 10:58:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:57:17] ppocr INFO: epoch: [425/500], global_step: 340900, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.940942, exp_rate<=1 : 0.977435, exp_rate<=2 : 0.994747, exp_rate<=3 : 0.998123, loss: 0.001632, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19641 s, avg_samples: 56.0, ips: 285.11800 samples/s, eta: 10:58:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:57:35] ppocr INFO: epoch: [425/500], global_step: 341000, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.941339, exp_rate<=1 : 0.977724, exp_rate<=2 : 0.994950, exp_rate<=3 : 0.998271, loss: 0.001376, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18093 s, avg_samples: 56.0, ips: 309.51955 samples/s, eta: 10:57:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:57:54] ppocr INFO: epoch: [425/500], global_step: 341100, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.941756, exp_rate<=1 : 0.977986, exp_rate<=2 : 0.994910, exp_rate<=3 : 0.998298, loss: 0.001462, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19370 s, avg_samples: 56.0, ips: 289.10144 samples/s, eta: 10:57:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:58:11] ppocr INFO: epoch: [425/500], global_step: 341200, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.942345, exp_rate<=1 : 0.978263, exp_rate<=2 : 0.994969, exp_rate<=3 : 0.998277, loss: 0.001208, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17160 s, avg_samples: 56.0, ips: 326.33205 samples/s, eta: 10:57:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:58:30] ppocr INFO: epoch: [425/500], global_step: 341300, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.942397, exp_rate<=1 : 0.978375, exp_rate<=2 : 0.994958, exp_rate<=3 : 0.998262, loss: 0.000868, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18814 s, avg_samples: 56.0, ips: 297.64568 samples/s, eta: 10:56:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:58:49] ppocr INFO: epoch: [425/500], global_step: 341400, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.942513, exp_rate<=1 : 0.978457, exp_rate<=2 : 0.994968, exp_rate<=3 : 0.998281, loss: 0.000897, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18801 s, avg_samples: 56.0, ips: 297.85965 samples/s, eta: 10:56:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:59:09] ppocr INFO: epoch: [425/500], global_step: 341500, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.942357, exp_rate<=1 : 0.978499, exp_rate<=2 : 0.994959, exp_rate<=3 : 0.998328, loss: 0.001528, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19641 s, avg_samples: 56.0, ips: 285.11770 samples/s, eta: 10:56:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:59:26] ppocr INFO: epoch: [425/500], global_step: 341600, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.942175, exp_rate<=1 : 0.978360, exp_rate<=2 : 0.994939, exp_rate<=3 : 0.998321, loss: 0.001993, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17677 s, avg_samples: 56.0, ips: 316.80144 samples/s, eta: 10:56:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 20:59:46] ppocr INFO: epoch: [425/500], global_step: 341700, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.941642, exp_rate<=1 : 0.977990, exp_rate<=2 : 0.994801, exp_rate<=3 : 0.998235, loss: 0.000687, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19597 s, avg_samples: 56.0, ips: 285.75647 samples/s, eta: 10:55:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:00:07] ppocr INFO: epoch: [425/500], global_step: 341800, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.940817, exp_rate<=1 : 0.977440, exp_rate<=2 : 0.994676, exp_rate<=3 : 0.998167, loss: 0.001283, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20667 s, avg_samples: 56.0, ips: 270.95693 samples/s, eta: 10:55:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:00:24] ppocr INFO: epoch: [425/500], global_step: 341900, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.940915, exp_rate<=1 : 0.977497, exp_rate<=2 : 0.994697, exp_rate<=3 : 0.998220, loss: 0.001508, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17699 s, avg_samples: 56.0, ips: 316.40787 samples/s, eta: 10:55:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:00:44] ppocr INFO: epoch: [425/500], global_step: 342000, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.940455, exp_rate<=1 : 0.977329, exp_rate<=2 : 0.994622, exp_rate<=3 : 0.998197, loss: 0.001434, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19285 s, avg_samples: 56.0, ips: 290.38304 samples/s, eta: 10:54:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:01:04] ppocr INFO: epoch: [425/500], global_step: 342100, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.939819, exp_rate<=1 : 0.976961, exp_rate<=2 : 0.994524, exp_rate<=3 : 0.998185, loss: 0.001778, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.20244 s, avg_samples: 56.0, ips: 276.62549 samples/s, eta: 10:54:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:01:22] ppocr INFO: epoch: [425/500], global_step: 342200, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.939883, exp_rate<=1 : 0.977028, exp_rate<=2 : 0.994524, exp_rate<=3 : 0.998179, loss: 0.002044, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17979 s, avg_samples: 56.0, ips: 311.47450 samples/s, eta: 10:54:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:01:41] ppocr INFO: epoch: [425/500], global_step: 342300, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.939796, exp_rate<=1 : 0.976933, exp_rate<=2 : 0.994496, exp_rate<=3 : 0.998165, loss: 0.001459, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18584 s, avg_samples: 56.0, ips: 301.33559 samples/s, eta: 10:53:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:01:59] ppocr INFO: epoch: [425/500], global_step: 342400, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.940127, exp_rate<=1 : 0.977064, exp_rate<=2 : 0.994577, exp_rate<=3 : 0.998222, loss: 0.001979, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18354 s, avg_samples: 56.0, ips: 305.11593 samples/s, eta: 10:53:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:02:18] ppocr INFO: epoch: [425/500], global_step: 342500, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.940005, exp_rate<=1 : 0.976942, exp_rate<=2 : 0.994548, exp_rate<=3 : 0.998194, loss: 0.001579, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18904 s, avg_samples: 56.0, ips: 296.22962 samples/s, eta: 10:53:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:02:20] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 21:02:21] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_425 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 21:02:40] ppocr INFO: epoch: [426/500], global_step: 342600, lr: 0.000100, edit distance: 0.000964, exp_rate: 0.946839, exp_rate<=1 : 0.980704, exp_rate<=2 : 0.996240, exp_rate<=3 : 0.998761, loss: 0.001265, avg_reader_cost: 0.03286 s, avg_batch_cost: 0.22329 s, avg_samples: 56.0, ips: 250.79680 samples/s, eta: 10:52:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:03:00] ppocr INFO: epoch: [426/500], global_step: 342700, lr: 0.000100, edit distance: 0.000974, exp_rate: 0.947593, exp_rate<=1 : 0.979952, exp_rate<=2 : 0.996317, exp_rate<=3 : 0.999156, loss: 0.001007, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19837 s, avg_samples: 56.0, ips: 282.30611 samples/s, eta: 10:52:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:03:17] ppocr INFO: epoch: [426/500], global_step: 342800, lr: 0.000100, edit distance: 0.001009, exp_rate: 0.950455, exp_rate<=1 : 0.981222, exp_rate<=2 : 0.996356, exp_rate<=3 : 0.999190, loss: 0.001138, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16523 s, avg_samples: 56.0, ips: 338.92310 samples/s, eta: 10:52:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:03:36] ppocr INFO: epoch: [426/500], global_step: 342900, lr: 0.000100, edit distance: 0.001005, exp_rate: 0.950425, exp_rate<=1 : 0.980709, exp_rate<=2 : 0.995797, exp_rate<=3 : 0.998749, loss: 0.001604, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19318 s, avg_samples: 56.0, ips: 289.87908 samples/s, eta: 10:51:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:03:55] ppocr INFO: epoch: [426/500], global_step: 343000, lr: 0.000100, edit distance: 0.001017, exp_rate: 0.949006, exp_rate<=1 : 0.980075, exp_rate<=2 : 0.995527, exp_rate<=3 : 0.998500, loss: 0.001308, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18743 s, avg_samples: 56.0, ips: 298.78266 samples/s, eta: 10:51:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:04:12] ppocr INFO: epoch: [426/500], global_step: 343100, lr: 0.000100, edit distance: 0.001028, exp_rate: 0.949470, exp_rate<=1 : 0.980533, exp_rate<=2 : 0.995615, exp_rate<=3 : 0.998609, loss: 0.001039, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17716 s, avg_samples: 56.0, ips: 316.09749 samples/s, eta: 10:51:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:04:31] ppocr INFO: epoch: [426/500], global_step: 343200, lr: 0.000100, edit distance: 0.001053, exp_rate: 0.948122, exp_rate<=1 : 0.980100, exp_rate<=2 : 0.995636, exp_rate<=3 : 0.998675, loss: 0.001395, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18483 s, avg_samples: 56.0, ips: 302.97713 samples/s, eta: 10:50:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:04:50] ppocr INFO: epoch: [426/500], global_step: 343300, lr: 0.000100, edit distance: 0.001038, exp_rate: 0.948052, exp_rate<=1 : 0.979841, exp_rate<=2 : 0.995377, exp_rate<=3 : 0.998524, loss: 0.001098, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18740 s, avg_samples: 56.0, ips: 298.82533 samples/s, eta: 10:50:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:05:09] ppocr INFO: epoch: [426/500], global_step: 343400, lr: 0.000100, edit distance: 0.001046, exp_rate: 0.948052, exp_rate<=1 : 0.980056, exp_rate<=2 : 0.995425, exp_rate<=3 : 0.998518, loss: 0.001062, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19030 s, avg_samples: 56.0, ips: 294.26781 samples/s, eta: 10:50:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:05:27] ppocr INFO: epoch: [426/500], global_step: 343500, lr: 0.000100, edit distance: 0.001083, exp_rate: 0.947237, exp_rate<=1 : 0.979808, exp_rate<=2 : 0.995439, exp_rate<=3 : 0.998540, loss: 0.001309, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18279 s, avg_samples: 56.0, ips: 306.36234 samples/s, eta: 10:49:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:05:45] ppocr INFO: epoch: [426/500], global_step: 343600, lr: 0.000100, edit distance: 0.001099, exp_rate: 0.946756, exp_rate<=1 : 0.979681, exp_rate<=2 : 0.995299, exp_rate<=3 : 0.998481, loss: 0.001217, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18058 s, avg_samples: 56.0, ips: 310.11098 samples/s, eta: 10:49:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:06:05] ppocr INFO: epoch: [426/500], global_step: 343700, lr: 0.000100, edit distance: 0.001117, exp_rate: 0.945853, exp_rate<=1 : 0.979367, exp_rate<=2 : 0.995297, exp_rate<=3 : 0.998506, loss: 0.001886, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19493 s, avg_samples: 56.0, ips: 287.28442 samples/s, eta: 10:49:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:06:24] ppocr INFO: epoch: [426/500], global_step: 343800, lr: 0.000100, edit distance: 0.001126, exp_rate: 0.944569, exp_rate<=1 : 0.979000, exp_rate<=2 : 0.995124, exp_rate<=3 : 0.998496, loss: 0.001443, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19442 s, avg_samples: 56.0, ips: 288.03564 samples/s, eta: 10:48:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:06:44] ppocr INFO: epoch: [426/500], global_step: 343900, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.944465, exp_rate<=1 : 0.978963, exp_rate<=2 : 0.995065, exp_rate<=3 : 0.998462, loss: 0.000872, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19452 s, avg_samples: 56.0, ips: 287.89049 samples/s, eta: 10:48:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:07:03] ppocr INFO: epoch: [426/500], global_step: 344000, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.943985, exp_rate<=1 : 0.978662, exp_rate<=2 : 0.995097, exp_rate<=3 : 0.998471, loss: 0.002763, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19224 s, avg_samples: 56.0, ips: 291.29735 samples/s, eta: 10:48:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:07:22] ppocr INFO: epoch: [426/500], global_step: 344100, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.943246, exp_rate<=1 : 0.978365, exp_rate<=2 : 0.995011, exp_rate<=3 : 0.998385, loss: 0.000929, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19268 s, avg_samples: 56.0, ips: 290.63918 samples/s, eta: 10:48:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:07:42] ppocr INFO: epoch: [426/500], global_step: 344200, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.943049, exp_rate<=1 : 0.978309, exp_rate<=2 : 0.995050, exp_rate<=3 : 0.998426, loss: 0.001889, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19832 s, avg_samples: 56.0, ips: 282.37844 samples/s, eta: 10:47:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:08:01] ppocr INFO: epoch: [426/500], global_step: 344300, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.942271, exp_rate<=1 : 0.977947, exp_rate<=2 : 0.994956, exp_rate<=3 : 0.998331, loss: 0.001421, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19492 s, avg_samples: 56.0, ips: 287.29996 samples/s, eta: 10:47:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:08:21] ppocr INFO: epoch: [426/500], global_step: 344400, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.942003, exp_rate<=1 : 0.977845, exp_rate<=2 : 0.994892, exp_rate<=3 : 0.998301, loss: 0.001068, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19627 s, avg_samples: 56.0, ips: 285.32333 samples/s, eta: 10:47:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:08:40] ppocr INFO: epoch: [426/500], global_step: 344500, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.942143, exp_rate<=1 : 0.977905, exp_rate<=2 : 0.994911, exp_rate<=3 : 0.998273, loss: 0.001539, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19073 s, avg_samples: 56.0, ips: 293.61046 samples/s, eta: 10:46:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:08:59] ppocr INFO: epoch: [426/500], global_step: 344600, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.941996, exp_rate<=1 : 0.977788, exp_rate<=2 : 0.994887, exp_rate<=3 : 0.998236, loss: 0.001768, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18598 s, avg_samples: 56.0, ips: 301.11427 samples/s, eta: 10:46:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:09:18] ppocr INFO: epoch: [426/500], global_step: 344700, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.941709, exp_rate<=1 : 0.977742, exp_rate<=2 : 0.994851, exp_rate<=3 : 0.998179, loss: 0.002130, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18889 s, avg_samples: 56.0, ips: 296.47473 samples/s, eta: 10:46:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:09:34] ppocr INFO: epoch: [426/500], global_step: 344800, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.941798, exp_rate<=1 : 0.977824, exp_rate<=2 : 0.994906, exp_rate<=3 : 0.998198, loss: 0.001455, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16715 s, avg_samples: 56.0, ips: 335.03764 samples/s, eta: 10:45:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:09:54] ppocr INFO: epoch: [426/500], global_step: 344900, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.941231, exp_rate<=1 : 0.977556, exp_rate<=2 : 0.994890, exp_rate<=3 : 0.998207, loss: 0.001193, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.19965 s, avg_samples: 56.0, ips: 280.49322 samples/s, eta: 10:45:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:10:11] ppocr INFO: epoch: [426/500], global_step: 345000, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.941174, exp_rate<=1 : 0.977588, exp_rate<=2 : 0.994968, exp_rate<=3 : 0.998250, loss: 0.002172, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16952 s, avg_samples: 56.0, ips: 330.34992 samples/s, eta: 10:45:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:10:31] ppocr INFO: epoch: [426/500], global_step: 345100, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.940319, exp_rate<=1 : 0.977142, exp_rate<=2 : 0.994910, exp_rate<=3 : 0.998213, loss: 0.001912, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20044 s, avg_samples: 56.0, ips: 279.38155 samples/s, eta: 10:44:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:10:49] ppocr INFO: epoch: [426/500], global_step: 345200, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.940230, exp_rate<=1 : 0.977095, exp_rate<=2 : 0.994928, exp_rate<=3 : 0.998228, loss: 0.001593, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17642 s, avg_samples: 56.0, ips: 317.42587 samples/s, eta: 10:44:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:10:57] ppocr INFO: epoch: [426/500], global_step: 345240, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.939957, exp_rate<=1 : 0.976999, exp_rate<=2 : 0.994865, exp_rate<=3 : 0.998208, loss: 0.001381, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07717 s, avg_samples: 22.4, ips: 290.25776 samples/s, eta: 10:44:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:10:59] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 21:11:10] ppocr INFO: epoch: [427/500], global_step: 345300, lr: 0.000100, edit distance: 0.000805, exp_rate: 0.955357, exp_rate<=1 : 0.982669, exp_rate<=2 : 0.996908, exp_rate<=3 : 0.998822, loss: 0.000967, avg_reader_cost: 0.02615 s, avg_batch_cost: 0.13601 s, avg_samples: 33.6, ips: 247.04685 samples/s, eta: 10:44:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:11:29] ppocr INFO: epoch: [427/500], global_step: 345400, lr: 0.000100, edit distance: 0.001064, exp_rate: 0.951332, exp_rate<=1 : 0.979719, exp_rate<=2 : 0.996525, exp_rate<=3 : 0.998729, loss: 0.001918, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18615 s, avg_samples: 56.0, ips: 300.82961 samples/s, eta: 10:43:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:11:48] ppocr INFO: epoch: [427/500], global_step: 345500, lr: 0.000100, edit distance: 0.001133, exp_rate: 0.950732, exp_rate<=1 : 0.981234, exp_rate<=2 : 0.996392, exp_rate<=3 : 0.998456, loss: 0.001166, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18614 s, avg_samples: 56.0, ips: 300.84819 samples/s, eta: 10:43:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:12:06] ppocr INFO: epoch: [427/500], global_step: 345600, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.948194, exp_rate<=1 : 0.979869, exp_rate<=2 : 0.995916, exp_rate<=3 : 0.998508, loss: 0.001650, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18254 s, avg_samples: 56.0, ips: 306.78577 samples/s, eta: 10:43:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:12:25] ppocr INFO: epoch: [427/500], global_step: 345700, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.946547, exp_rate<=1 : 0.979276, exp_rate<=2 : 0.995581, exp_rate<=3 : 0.998447, loss: 0.002098, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19324 s, avg_samples: 56.0, ips: 289.79347 samples/s, eta: 10:42:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:12:43] ppocr INFO: epoch: [427/500], global_step: 345800, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.947418, exp_rate<=1 : 0.980388, exp_rate<=2 : 0.995763, exp_rate<=3 : 0.998516, loss: 0.001708, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17145 s, avg_samples: 56.0, ips: 326.63520 samples/s, eta: 10:42:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:13:03] ppocr INFO: epoch: [427/500], global_step: 345900, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.946953, exp_rate<=1 : 0.980421, exp_rate<=2 : 0.995667, exp_rate<=3 : 0.998490, loss: 0.001121, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20635 s, avg_samples: 56.0, ips: 271.38788 samples/s, eta: 10:42:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:13:23] ppocr INFO: epoch: [427/500], global_step: 346000, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.945975, exp_rate<=1 : 0.979742, exp_rate<=2 : 0.995527, exp_rate<=3 : 0.998477, loss: 0.000729, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19563 s, avg_samples: 56.0, ips: 286.24770 samples/s, eta: 10:41:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:13:41] ppocr INFO: epoch: [427/500], global_step: 346100, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.946039, exp_rate<=1 : 0.979999, exp_rate<=2 : 0.995476, exp_rate<=3 : 0.998481, loss: 0.000901, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18448 s, avg_samples: 56.0, ips: 303.55736 samples/s, eta: 10:41:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:14:01] ppocr INFO: epoch: [427/500], global_step: 346200, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.944889, exp_rate<=1 : 0.979483, exp_rate<=2 : 0.995162, exp_rate<=3 : 0.998309, loss: 0.001378, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20248 s, avg_samples: 56.0, ips: 276.56543 samples/s, eta: 10:41:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:14:19] ppocr INFO: epoch: [427/500], global_step: 346300, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.945372, exp_rate<=1 : 0.979724, exp_rate<=2 : 0.995282, exp_rate<=3 : 0.998363, loss: 0.001507, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17640 s, avg_samples: 56.0, ips: 317.46372 samples/s, eta: 10:41:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:14:39] ppocr INFO: epoch: [427/500], global_step: 346400, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.945133, exp_rate<=1 : 0.979638, exp_rate<=2 : 0.995324, exp_rate<=3 : 0.998459, loss: 0.001183, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19325 s, avg_samples: 56.0, ips: 289.78366 samples/s, eta: 10:40:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:14:58] ppocr INFO: epoch: [427/500], global_step: 346500, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.945413, exp_rate<=1 : 0.979552, exp_rate<=2 : 0.995311, exp_rate<=3 : 0.998456, loss: 0.001200, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19045 s, avg_samples: 56.0, ips: 294.04229 samples/s, eta: 10:40:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:15:17] ppocr INFO: epoch: [427/500], global_step: 346600, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.944545, exp_rate<=1 : 0.979146, exp_rate<=2 : 0.995221, exp_rate<=3 : 0.998467, loss: 0.002164, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19309 s, avg_samples: 56.0, ips: 290.01482 samples/s, eta: 10:40:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:15:35] ppocr INFO: epoch: [427/500], global_step: 346700, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.943685, exp_rate<=1 : 0.978699, exp_rate<=2 : 0.995125, exp_rate<=3 : 0.998413, loss: 0.001432, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18012 s, avg_samples: 56.0, ips: 310.90322 samples/s, eta: 10:39:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:15:53] ppocr INFO: epoch: [427/500], global_step: 346800, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.943235, exp_rate<=1 : 0.978453, exp_rate<=2 : 0.995091, exp_rate<=3 : 0.998376, loss: 0.001250, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18338 s, avg_samples: 56.0, ips: 305.37585 samples/s, eta: 10:39:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:16:11] ppocr INFO: epoch: [427/500], global_step: 346900, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.942925, exp_rate<=1 : 0.978416, exp_rate<=2 : 0.995155, exp_rate<=3 : 0.998364, loss: 0.001432, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18159 s, avg_samples: 56.0, ips: 308.39319 samples/s, eta: 10:39:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:16:31] ppocr INFO: epoch: [427/500], global_step: 347000, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.942255, exp_rate<=1 : 0.978055, exp_rate<=2 : 0.994964, exp_rate<=3 : 0.998238, loss: 0.002263, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19142 s, avg_samples: 56.0, ips: 292.55019 samples/s, eta: 10:38:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:16:49] ppocr INFO: epoch: [427/500], global_step: 347100, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.942132, exp_rate<=1 : 0.977918, exp_rate<=2 : 0.994945, exp_rate<=3 : 0.998225, loss: 0.001445, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18268 s, avg_samples: 56.0, ips: 306.55359 samples/s, eta: 10:38:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:17:07] ppocr INFO: epoch: [427/500], global_step: 347200, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.942025, exp_rate<=1 : 0.977940, exp_rate<=2 : 0.994952, exp_rate<=3 : 0.998211, loss: 0.001665, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18361 s, avg_samples: 56.0, ips: 305.00127 samples/s, eta: 10:38:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:17:27] ppocr INFO: epoch: [427/500], global_step: 347300, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.940953, exp_rate<=1 : 0.977676, exp_rate<=2 : 0.994871, exp_rate<=3 : 0.998184, loss: 0.001893, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19933 s, avg_samples: 56.0, ips: 280.94548 samples/s, eta: 10:37:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:17:47] ppocr INFO: epoch: [427/500], global_step: 347400, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.940547, exp_rate<=1 : 0.977583, exp_rate<=2 : 0.994778, exp_rate<=3 : 0.998159, loss: 0.001463, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19509 s, avg_samples: 56.0, ips: 287.04695 samples/s, eta: 10:37:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:18:06] ppocr INFO: epoch: [427/500], global_step: 347500, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.940123, exp_rate<=1 : 0.977344, exp_rate<=2 : 0.994691, exp_rate<=3 : 0.998096, loss: 0.001052, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18868 s, avg_samples: 56.0, ips: 296.80030 samples/s, eta: 10:37:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:18:25] ppocr INFO: epoch: [427/500], global_step: 347600, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.939903, exp_rate<=1 : 0.977118, exp_rate<=2 : 0.994620, exp_rate<=3 : 0.998059, loss: 0.001038, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19312 s, avg_samples: 56.0, ips: 289.96890 samples/s, eta: 10:36:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:18:43] ppocr INFO: epoch: [427/500], global_step: 347700, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.940128, exp_rate<=1 : 0.977176, exp_rate<=2 : 0.994649, exp_rate<=3 : 0.998080, loss: 0.001237, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18032 s, avg_samples: 56.0, ips: 310.56002 samples/s, eta: 10:36:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:19:03] ppocr INFO: epoch: [427/500], global_step: 347800, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.940307, exp_rate<=1 : 0.977272, exp_rate<=2 : 0.994705, exp_rate<=3 : 0.998103, loss: 0.001239, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19429 s, avg_samples: 56.0, ips: 288.22949 samples/s, eta: 10:36:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:19:20] ppocr INFO: epoch: [427/500], global_step: 347900, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.940583, exp_rate<=1 : 0.977413, exp_rate<=2 : 0.994781, exp_rate<=3 : 0.998120, loss: 0.001330, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17705 s, avg_samples: 56.0, ips: 316.29883 samples/s, eta: 10:35:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:19:36] ppocr INFO: epoch: [427/500], global_step: 347980, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.940364, exp_rate<=1 : 0.977280, exp_rate<=2 : 0.994703, exp_rate<=3 : 0.998071, loss: 0.001074, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.15319 s, avg_samples: 44.8, ips: 292.45016 samples/s, eta: 10:35:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:19:38] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 21:19:42] ppocr INFO: epoch: [428/500], global_step: 348000, lr: 0.000100, edit distance: 0.002201, exp_rate: 0.948708, exp_rate<=1 : 0.987204, exp_rate<=2 : 0.997768, exp_rate<=3 : 0.998295, loss: 0.001117, avg_reader_cost: 0.02541 s, avg_batch_cost: 0.06259 s, avg_samples: 11.2, ips: 178.93026 samples/s, eta: 10:35:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:20:00] ppocr INFO: epoch: [428/500], global_step: 348100, lr: 0.000100, edit distance: 0.001473, exp_rate: 0.947923, exp_rate<=1 : 0.981264, exp_rate<=2 : 0.996049, exp_rate<=3 : 0.998467, loss: 0.000816, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18363 s, avg_samples: 56.0, ips: 304.96247 samples/s, eta: 10:35:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:20:19] ppocr INFO: epoch: [428/500], global_step: 348200, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.949743, exp_rate<=1 : 0.981628, exp_rate<=2 : 0.996351, exp_rate<=3 : 0.998812, loss: 0.001472, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18294 s, avg_samples: 56.0, ips: 306.11166 samples/s, eta: 10:34:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:20:37] ppocr INFO: epoch: [428/500], global_step: 348300, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.948657, exp_rate<=1 : 0.981750, exp_rate<=2 : 0.996262, exp_rate<=3 : 0.998907, loss: 0.001232, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18880 s, avg_samples: 56.0, ips: 296.61706 samples/s, eta: 10:34:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:20:55] ppocr INFO: epoch: [428/500], global_step: 348400, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.949171, exp_rate<=1 : 0.981340, exp_rate<=2 : 0.996051, exp_rate<=3 : 0.998915, loss: 0.001579, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17977 s, avg_samples: 56.0, ips: 311.51035 samples/s, eta: 10:34:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:21:14] ppocr INFO: epoch: [428/500], global_step: 348500, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.948258, exp_rate<=1 : 0.981044, exp_rate<=2 : 0.995907, exp_rate<=3 : 0.998846, loss: 0.001638, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18558 s, avg_samples: 56.0, ips: 301.75280 samples/s, eta: 10:33:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:21:33] ppocr INFO: epoch: [428/500], global_step: 348600, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.945557, exp_rate<=1 : 0.980092, exp_rate<=2 : 0.995247, exp_rate<=3 : 0.998657, loss: 0.001530, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18857 s, avg_samples: 56.0, ips: 296.97375 samples/s, eta: 10:33:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:21:51] ppocr INFO: epoch: [428/500], global_step: 348700, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.946017, exp_rate<=1 : 0.980290, exp_rate<=2 : 0.995448, exp_rate<=3 : 0.998691, loss: 0.001941, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18210 s, avg_samples: 56.0, ips: 307.52364 samples/s, eta: 10:33:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:22:10] ppocr INFO: epoch: [428/500], global_step: 348800, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.944798, exp_rate<=1 : 0.979855, exp_rate<=2 : 0.995248, exp_rate<=3 : 0.998656, loss: 0.001104, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18631 s, avg_samples: 56.0, ips: 300.57700 samples/s, eta: 10:33:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:22:28] ppocr INFO: epoch: [428/500], global_step: 348900, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.944978, exp_rate<=1 : 0.979934, exp_rate<=2 : 0.995301, exp_rate<=3 : 0.998694, loss: 0.001818, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17979 s, avg_samples: 56.0, ips: 311.46758 samples/s, eta: 10:32:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:22:47] ppocr INFO: epoch: [428/500], global_step: 349000, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.944771, exp_rate<=1 : 0.979946, exp_rate<=2 : 0.995273, exp_rate<=3 : 0.998632, loss: 0.001550, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18800 s, avg_samples: 56.0, ips: 297.86905 samples/s, eta: 10:32:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:23:06] ppocr INFO: epoch: [428/500], global_step: 349100, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.944402, exp_rate<=1 : 0.979723, exp_rate<=2 : 0.995193, exp_rate<=3 : 0.998697, loss: 0.000874, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19681 s, avg_samples: 56.0, ips: 284.53801 samples/s, eta: 10:32:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:23:24] ppocr INFO: epoch: [428/500], global_step: 349200, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.944236, exp_rate<=1 : 0.979531, exp_rate<=2 : 0.995220, exp_rate<=3 : 0.998717, loss: 0.001178, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17945 s, avg_samples: 56.0, ips: 312.07090 samples/s, eta: 10:31:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:23:44] ppocr INFO: epoch: [428/500], global_step: 349300, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.943967, exp_rate<=1 : 0.979417, exp_rate<=2 : 0.995300, exp_rate<=3 : 0.998682, loss: 0.001400, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19725 s, avg_samples: 56.0, ips: 283.90552 samples/s, eta: 10:31:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:24:03] ppocr INFO: epoch: [428/500], global_step: 349400, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.943527, exp_rate<=1 : 0.978802, exp_rate<=2 : 0.995130, exp_rate<=3 : 0.998622, loss: 0.001578, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18721 s, avg_samples: 56.0, ips: 299.12352 samples/s, eta: 10:31:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:24:22] ppocr INFO: epoch: [428/500], global_step: 349500, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.943543, exp_rate<=1 : 0.978809, exp_rate<=2 : 0.995149, exp_rate<=3 : 0.998597, loss: 0.001393, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18771 s, avg_samples: 56.0, ips: 298.32606 samples/s, eta: 10:30:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:24:40] ppocr INFO: epoch: [428/500], global_step: 349600, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.943542, exp_rate<=1 : 0.978901, exp_rate<=2 : 0.995199, exp_rate<=3 : 0.998570, loss: 0.001532, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18913 s, avg_samples: 56.0, ips: 296.09965 samples/s, eta: 10:30:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:24:59] ppocr INFO: epoch: [428/500], global_step: 349700, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.943346, exp_rate<=1 : 0.978760, exp_rate<=2 : 0.995229, exp_rate<=3 : 0.998583, loss: 0.001298, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18065 s, avg_samples: 56.0, ips: 309.98731 samples/s, eta: 10:30:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:25:18] ppocr INFO: epoch: [428/500], global_step: 349800, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.942895, exp_rate<=1 : 0.978547, exp_rate<=2 : 0.995213, exp_rate<=3 : 0.998570, loss: 0.001440, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19495 s, avg_samples: 56.0, ips: 287.25768 samples/s, eta: 10:29:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:25:36] ppocr INFO: epoch: [428/500], global_step: 349900, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.942628, exp_rate<=1 : 0.978427, exp_rate<=2 : 0.995186, exp_rate<=3 : 0.998579, loss: 0.001192, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18098 s, avg_samples: 56.0, ips: 309.41839 samples/s, eta: 10:29:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:25:55] ppocr INFO: epoch: [428/500], global_step: 350000, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.942167, exp_rate<=1 : 0.978393, exp_rate<=2 : 0.995184, exp_rate<=3 : 0.998610, loss: 0.001113, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18538 s, avg_samples: 56.0, ips: 302.08450 samples/s, eta: 10:29:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:26:13] ppocr INFO: epoch: [428/500], global_step: 350100, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.942169, exp_rate<=1 : 0.978417, exp_rate<=2 : 0.995214, exp_rate<=3 : 0.998625, loss: 0.000958, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17957 s, avg_samples: 56.0, ips: 311.85058 samples/s, eta: 10:28:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:26:31] ppocr INFO: epoch: [428/500], global_step: 350200, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.942214, exp_rate<=1 : 0.978389, exp_rate<=2 : 0.995243, exp_rate<=3 : 0.998607, loss: 0.001433, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17903 s, avg_samples: 56.0, ips: 312.79055 samples/s, eta: 10:28:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:26:50] ppocr INFO: epoch: [428/500], global_step: 350300, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.941806, exp_rate<=1 : 0.978145, exp_rate<=2 : 0.995140, exp_rate<=3 : 0.998555, loss: 0.001059, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19009 s, avg_samples: 56.0, ips: 294.59664 samples/s, eta: 10:28:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:27:10] ppocr INFO: epoch: [428/500], global_step: 350400, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.941118, exp_rate<=1 : 0.977831, exp_rate<=2 : 0.994981, exp_rate<=3 : 0.998526, loss: 0.001432, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20059 s, avg_samples: 56.0, ips: 279.17334 samples/s, eta: 10:27:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:27:29] ppocr INFO: epoch: [428/500], global_step: 350500, lr: 0.000100, edit distance: 0.001308, exp_rate: 0.941194, exp_rate<=1 : 0.977834, exp_rate<=2 : 0.994961, exp_rate<=3 : 0.998520, loss: 0.001364, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19524 s, avg_samples: 56.0, ips: 286.81925 samples/s, eta: 10:27:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:27:49] ppocr INFO: epoch: [428/500], global_step: 350600, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.940712, exp_rate<=1 : 0.977602, exp_rate<=2 : 0.994928, exp_rate<=3 : 0.998497, loss: 0.001195, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20108 s, avg_samples: 56.0, ips: 278.49378 samples/s, eta: 10:27:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:28:08] ppocr INFO: epoch: [428/500], global_step: 350700, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.940823, exp_rate<=1 : 0.977642, exp_rate<=2 : 0.994929, exp_rate<=3 : 0.998475, loss: 0.001879, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18309 s, avg_samples: 56.0, ips: 305.86813 samples/s, eta: 10:26:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:28:12] ppocr INFO: epoch: [428/500], global_step: 350720, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.940686, exp_rate<=1 : 0.977615, exp_rate<=2 : 0.994920, exp_rate<=3 : 0.998468, loss: 0.001664, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03931 s, avg_samples: 11.2, ips: 284.92050 samples/s, eta: 10:26:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:28:14] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 21:28:29] ppocr INFO: epoch: [429/500], global_step: 350800, lr: 0.000100, edit distance: 0.001325, exp_rate: 0.947030, exp_rate<=1 : 0.980189, exp_rate<=2 : 0.996717, exp_rate<=3 : 0.998654, loss: 0.001347, avg_reader_cost: 0.02910 s, avg_batch_cost: 0.17748 s, avg_samples: 44.8, ips: 252.42873 samples/s, eta: 10:26:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:28:47] ppocr INFO: epoch: [429/500], global_step: 350900, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.946378, exp_rate<=1 : 0.979578, exp_rate<=2 : 0.996138, exp_rate<=3 : 0.998786, loss: 0.001287, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17939 s, avg_samples: 56.0, ips: 312.16436 samples/s, eta: 10:26:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:29:07] ppocr INFO: epoch: [429/500], global_step: 351000, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.945158, exp_rate<=1 : 0.979107, exp_rate<=2 : 0.995854, exp_rate<=3 : 0.998851, loss: 0.001698, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19298 s, avg_samples: 56.0, ips: 290.18199 samples/s, eta: 10:25:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:29:25] ppocr INFO: epoch: [429/500], global_step: 351100, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.945735, exp_rate<=1 : 0.978805, exp_rate<=2 : 0.995548, exp_rate<=3 : 0.998615, loss: 0.001346, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17935 s, avg_samples: 56.0, ips: 312.24341 samples/s, eta: 10:25:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:29:44] ppocr INFO: epoch: [429/500], global_step: 351200, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.945287, exp_rate<=1 : 0.978835, exp_rate<=2 : 0.995522, exp_rate<=3 : 0.998654, loss: 0.001971, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19008 s, avg_samples: 56.0, ips: 294.61207 samples/s, eta: 10:25:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:30:04] ppocr INFO: epoch: [429/500], global_step: 351300, lr: 0.000100, edit distance: 0.001297, exp_rate: 0.943114, exp_rate<=1 : 0.978140, exp_rate<=2 : 0.995231, exp_rate<=3 : 0.998556, loss: 0.001925, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20184 s, avg_samples: 56.0, ips: 277.45292 samples/s, eta: 10:25:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:30:22] ppocr INFO: epoch: [429/500], global_step: 351400, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.943177, exp_rate<=1 : 0.977894, exp_rate<=2 : 0.995098, exp_rate<=3 : 0.998599, loss: 0.001706, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17653 s, avg_samples: 56.0, ips: 317.22230 samples/s, eta: 10:24:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:30:41] ppocr INFO: epoch: [429/500], global_step: 351500, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.942329, exp_rate<=1 : 0.977348, exp_rate<=2 : 0.994815, exp_rate<=3 : 0.998398, loss: 0.001215, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18974 s, avg_samples: 56.0, ips: 295.14142 samples/s, eta: 10:24:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:30:59] ppocr INFO: epoch: [429/500], global_step: 351600, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.943210, exp_rate<=1 : 0.977632, exp_rate<=2 : 0.994953, exp_rate<=3 : 0.998400, loss: 0.001156, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18712 s, avg_samples: 56.0, ips: 299.27148 samples/s, eta: 10:24:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:31:19] ppocr INFO: epoch: [429/500], global_step: 351700, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.941818, exp_rate<=1 : 0.976908, exp_rate<=2 : 0.994912, exp_rate<=3 : 0.998384, loss: 0.001985, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19954 s, avg_samples: 56.0, ips: 280.64402 samples/s, eta: 10:23:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:31:38] ppocr INFO: epoch: [429/500], global_step: 351800, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.941612, exp_rate<=1 : 0.976840, exp_rate<=2 : 0.994970, exp_rate<=3 : 0.998368, loss: 0.001484, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18942 s, avg_samples: 56.0, ips: 295.64283 samples/s, eta: 10:23:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:31:57] ppocr INFO: epoch: [429/500], global_step: 351900, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.942233, exp_rate<=1 : 0.977237, exp_rate<=2 : 0.995088, exp_rate<=3 : 0.998413, loss: 0.001571, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18595 s, avg_samples: 56.0, ips: 301.15011 samples/s, eta: 10:23:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:32:16] ppocr INFO: epoch: [429/500], global_step: 352000, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.942115, exp_rate<=1 : 0.977193, exp_rate<=2 : 0.994988, exp_rate<=3 : 0.998370, loss: 0.001355, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19576 s, avg_samples: 56.0, ips: 286.05939 samples/s, eta: 10:22:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:32:35] ppocr INFO: epoch: [429/500], global_step: 352100, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.941564, exp_rate<=1 : 0.976799, exp_rate<=2 : 0.994927, exp_rate<=3 : 0.998328, loss: 0.001187, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18772 s, avg_samples: 56.0, ips: 298.30872 samples/s, eta: 10:22:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:32:53] ppocr INFO: epoch: [429/500], global_step: 352200, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.941677, exp_rate<=1 : 0.976786, exp_rate<=2 : 0.994966, exp_rate<=3 : 0.998417, loss: 0.001180, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17707 s, avg_samples: 56.0, ips: 316.26254 samples/s, eta: 10:22:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:33:14] ppocr INFO: epoch: [429/500], global_step: 352300, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.941267, exp_rate<=1 : 0.976533, exp_rate<=2 : 0.994903, exp_rate<=3 : 0.998404, loss: 0.001031, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20793 s, avg_samples: 56.0, ips: 269.32760 samples/s, eta: 10:21:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:33:33] ppocr INFO: epoch: [429/500], global_step: 352400, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.940727, exp_rate<=1 : 0.976374, exp_rate<=2 : 0.994933, exp_rate<=3 : 0.998429, loss: 0.001384, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19151 s, avg_samples: 56.0, ips: 292.40614 samples/s, eta: 10:21:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:33:51] ppocr INFO: epoch: [429/500], global_step: 352500, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.941004, exp_rate<=1 : 0.976518, exp_rate<=2 : 0.994952, exp_rate<=3 : 0.998435, loss: 0.001330, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18227 s, avg_samples: 56.0, ips: 307.24265 samples/s, eta: 10:21:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:34:11] ppocr INFO: epoch: [429/500], global_step: 352600, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.941110, exp_rate<=1 : 0.976483, exp_rate<=2 : 0.994955, exp_rate<=3 : 0.998415, loss: 0.001207, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20006 s, avg_samples: 56.0, ips: 279.91188 samples/s, eta: 10:20:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:34:29] ppocr INFO: epoch: [429/500], global_step: 352700, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.941313, exp_rate<=1 : 0.976644, exp_rate<=2 : 0.995027, exp_rate<=3 : 0.998441, loss: 0.001243, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18186 s, avg_samples: 56.0, ips: 307.92425 samples/s, eta: 10:20:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:34:47] ppocr INFO: epoch: [429/500], global_step: 352800, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.941376, exp_rate<=1 : 0.976878, exp_rate<=2 : 0.995114, exp_rate<=3 : 0.998469, loss: 0.001601, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17632 s, avg_samples: 56.0, ips: 317.60872 samples/s, eta: 10:20:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:35:06] ppocr INFO: epoch: [429/500], global_step: 352900, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.941153, exp_rate<=1 : 0.976773, exp_rate<=2 : 0.995061, exp_rate<=3 : 0.998473, loss: 0.001693, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18738 s, avg_samples: 56.0, ips: 298.86236 samples/s, eta: 10:19:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:35:24] ppocr INFO: epoch: [429/500], global_step: 353000, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.940789, exp_rate<=1 : 0.976531, exp_rate<=2 : 0.994964, exp_rate<=3 : 0.998427, loss: 0.001630, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18457 s, avg_samples: 56.0, ips: 303.41218 samples/s, eta: 10:19:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:35:43] ppocr INFO: epoch: [429/500], global_step: 353100, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.940693, exp_rate<=1 : 0.976506, exp_rate<=2 : 0.994985, exp_rate<=3 : 0.998450, loss: 0.001762, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18449 s, avg_samples: 56.0, ips: 303.54478 samples/s, eta: 10:19:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:36:02] ppocr INFO: epoch: [429/500], global_step: 353200, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.940259, exp_rate<=1 : 0.976410, exp_rate<=2 : 0.994959, exp_rate<=3 : 0.998416, loss: 0.001903, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19470 s, avg_samples: 56.0, ips: 287.62226 samples/s, eta: 10:18:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:36:21] ppocr INFO: epoch: [429/500], global_step: 353300, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.939894, exp_rate<=1 : 0.976390, exp_rate<=2 : 0.994910, exp_rate<=3 : 0.998372, loss: 0.001867, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18236 s, avg_samples: 56.0, ips: 307.08958 samples/s, eta: 10:18:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:36:38] ppocr INFO: epoch: [429/500], global_step: 353400, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.939780, exp_rate<=1 : 0.976472, exp_rate<=2 : 0.994879, exp_rate<=3 : 0.998363, loss: 0.001391, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17412 s, avg_samples: 56.0, ips: 321.60958 samples/s, eta: 10:18:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:36:50] ppocr INFO: epoch: [429/500], global_step: 353460, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.939424, exp_rate<=1 : 0.976329, exp_rate<=2 : 0.994788, exp_rate<=3 : 0.998350, loss: 0.002143, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.12381 s, avg_samples: 33.6, ips: 271.39272 samples/s, eta: 10:18:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:36:53] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 21:37:02] ppocr INFO: epoch: [430/500], global_step: 353500, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.946684, exp_rate<=1 : 0.981602, exp_rate<=2 : 0.995856, exp_rate<=3 : 0.998183, loss: 0.001489, avg_reader_cost: 0.02668 s, avg_batch_cost: 0.11357 s, avg_samples: 22.4, ips: 197.23982 samples/s, eta: 10:18:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:37:20] ppocr INFO: epoch: [430/500], global_step: 353600, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.943898, exp_rate<=1 : 0.980357, exp_rate<=2 : 0.995731, exp_rate<=3 : 0.997834, loss: 0.001275, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18145 s, avg_samples: 56.0, ips: 308.63217 samples/s, eta: 10:17:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:37:38] ppocr INFO: epoch: [430/500], global_step: 353700, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.946809, exp_rate<=1 : 0.980674, exp_rate<=2 : 0.995429, exp_rate<=3 : 0.997831, loss: 0.000957, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18024 s, avg_samples: 56.0, ips: 310.70255 samples/s, eta: 10:17:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:37:57] ppocr INFO: epoch: [430/500], global_step: 353800, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.945521, exp_rate<=1 : 0.980507, exp_rate<=2 : 0.995299, exp_rate<=3 : 0.998064, loss: 0.001685, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19382 s, avg_samples: 56.0, ips: 288.92143 samples/s, eta: 10:17:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:38:16] ppocr INFO: epoch: [430/500], global_step: 353900, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.945894, exp_rate<=1 : 0.980349, exp_rate<=2 : 0.995332, exp_rate<=3 : 0.998225, loss: 0.001359, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18219 s, avg_samples: 56.0, ips: 307.37333 samples/s, eta: 10:16:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:38:35] ppocr INFO: epoch: [430/500], global_step: 354000, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.944474, exp_rate<=1 : 0.979659, exp_rate<=2 : 0.994924, exp_rate<=3 : 0.998014, loss: 0.001154, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19508 s, avg_samples: 56.0, ips: 287.05834 samples/s, eta: 10:16:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:38:54] ppocr INFO: epoch: [430/500], global_step: 354100, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.945774, exp_rate<=1 : 0.980154, exp_rate<=2 : 0.995243, exp_rate<=3 : 0.998203, loss: 0.001388, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18973 s, avg_samples: 56.0, ips: 295.15923 samples/s, eta: 10:16:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:39:13] ppocr INFO: epoch: [430/500], global_step: 354200, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.945184, exp_rate<=1 : 0.979534, exp_rate<=2 : 0.995153, exp_rate<=3 : 0.998060, loss: 0.001621, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18964 s, avg_samples: 56.0, ips: 295.30021 samples/s, eta: 10:15:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:39:33] ppocr INFO: epoch: [430/500], global_step: 354300, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.944739, exp_rate<=1 : 0.979548, exp_rate<=2 : 0.995208, exp_rate<=3 : 0.998033, loss: 0.002154, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19506 s, avg_samples: 56.0, ips: 287.09813 samples/s, eta: 10:15:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:39:51] ppocr INFO: epoch: [430/500], global_step: 354400, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.944729, exp_rate<=1 : 0.979434, exp_rate<=2 : 0.995248, exp_rate<=3 : 0.998031, loss: 0.002695, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18003 s, avg_samples: 56.0, ips: 311.05093 samples/s, eta: 10:15:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:40:08] ppocr INFO: epoch: [430/500], global_step: 354500, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.945036, exp_rate<=1 : 0.979445, exp_rate<=2 : 0.995265, exp_rate<=3 : 0.998051, loss: 0.001423, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17740 s, avg_samples: 56.0, ips: 315.67394 samples/s, eta: 10:14:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:40:28] ppocr INFO: epoch: [430/500], global_step: 354600, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.944395, exp_rate<=1 : 0.979410, exp_rate<=2 : 0.995232, exp_rate<=3 : 0.998047, loss: 0.001744, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19700 s, avg_samples: 56.0, ips: 284.25965 samples/s, eta: 10:14:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:40:47] ppocr INFO: epoch: [430/500], global_step: 354700, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.944769, exp_rate<=1 : 0.979365, exp_rate<=2 : 0.995278, exp_rate<=3 : 0.998093, loss: 0.001614, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18717 s, avg_samples: 56.0, ips: 299.19734 samples/s, eta: 10:14:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:41:05] ppocr INFO: epoch: [430/500], global_step: 354800, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.944583, exp_rate<=1 : 0.979305, exp_rate<=2 : 0.995146, exp_rate<=3 : 0.998081, loss: 0.001730, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17944 s, avg_samples: 56.0, ips: 312.08097 samples/s, eta: 10:13:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:41:24] ppocr INFO: epoch: [430/500], global_step: 354900, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.944482, exp_rate<=1 : 0.979256, exp_rate<=2 : 0.995208, exp_rate<=3 : 0.998140, loss: 0.001146, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18924 s, avg_samples: 56.0, ips: 295.91954 samples/s, eta: 10:13:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:41:42] ppocr INFO: epoch: [430/500], global_step: 355000, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.944330, exp_rate<=1 : 0.979324, exp_rate<=2 : 0.995255, exp_rate<=3 : 0.998182, loss: 0.001357, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18588 s, avg_samples: 56.0, ips: 301.27624 samples/s, eta: 10:13:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:42:02] ppocr INFO: epoch: [430/500], global_step: 355100, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.943629, exp_rate<=1 : 0.979200, exp_rate<=2 : 0.995238, exp_rate<=3 : 0.998182, loss: 0.001543, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19303 s, avg_samples: 56.0, ips: 290.11587 samples/s, eta: 10:12:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:42:20] ppocr INFO: epoch: [430/500], global_step: 355200, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.943510, exp_rate<=1 : 0.979038, exp_rate<=2 : 0.995265, exp_rate<=3 : 0.998149, loss: 0.001253, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18562 s, avg_samples: 56.0, ips: 301.69397 samples/s, eta: 10:12:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:42:38] ppocr INFO: epoch: [430/500], global_step: 355300, lr: 0.000100, edit distance: 0.001206, exp_rate: 0.943750, exp_rate<=1 : 0.979058, exp_rate<=2 : 0.995208, exp_rate<=3 : 0.998166, loss: 0.001777, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18006 s, avg_samples: 56.0, ips: 311.00572 samples/s, eta: 10:12:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:42:56] ppocr INFO: epoch: [430/500], global_step: 355400, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.943604, exp_rate<=1 : 0.978919, exp_rate<=2 : 0.995229, exp_rate<=3 : 0.998196, loss: 0.001328, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18098 s, avg_samples: 56.0, ips: 309.41926 samples/s, eta: 10:11:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:43:16] ppocr INFO: epoch: [430/500], global_step: 355500, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.942402, exp_rate<=1 : 0.978232, exp_rate<=2 : 0.995051, exp_rate<=3 : 0.998187, loss: 0.001675, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19437 s, avg_samples: 56.0, ips: 288.10524 samples/s, eta: 10:11:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:43:34] ppocr INFO: epoch: [430/500], global_step: 355600, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.941560, exp_rate<=1 : 0.977834, exp_rate<=2 : 0.994955, exp_rate<=3 : 0.998148, loss: 0.001150, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17938 s, avg_samples: 56.0, ips: 312.18018 samples/s, eta: 10:11:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:43:52] ppocr INFO: epoch: [430/500], global_step: 355700, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.941019, exp_rate<=1 : 0.977622, exp_rate<=2 : 0.994940, exp_rate<=3 : 0.998153, loss: 0.002242, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17984 s, avg_samples: 56.0, ips: 311.38491 samples/s, eta: 10:10:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:44:10] ppocr INFO: epoch: [430/500], global_step: 355800, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.940516, exp_rate<=1 : 0.977503, exp_rate<=2 : 0.994827, exp_rate<=3 : 0.998124, loss: 0.001671, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18088 s, avg_samples: 56.0, ips: 309.60550 samples/s, eta: 10:10:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:44:28] ppocr INFO: epoch: [430/500], global_step: 355900, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.940400, exp_rate<=1 : 0.977380, exp_rate<=2 : 0.994875, exp_rate<=3 : 0.998153, loss: 0.001790, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18207 s, avg_samples: 56.0, ips: 307.58120 samples/s, eta: 10:10:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:44:47] ppocr INFO: epoch: [430/500], global_step: 356000, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.940335, exp_rate<=1 : 0.977314, exp_rate<=2 : 0.994828, exp_rate<=3 : 0.998164, loss: 0.001500, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19020 s, avg_samples: 56.0, ips: 294.42482 samples/s, eta: 10:09:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:45:07] ppocr INFO: epoch: [430/500], global_step: 356100, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.940159, exp_rate<=1 : 0.977184, exp_rate<=2 : 0.994816, exp_rate<=3 : 0.998148, loss: 0.001422, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19676 s, avg_samples: 56.0, ips: 284.60847 samples/s, eta: 10:09:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:45:25] ppocr INFO: epoch: [430/500], global_step: 356200, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.940163, exp_rate<=1 : 0.977241, exp_rate<=2 : 0.994803, exp_rate<=3 : 0.998145, loss: 0.001902, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18259 s, avg_samples: 56.0, ips: 306.69052 samples/s, eta: 10:09:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:45:27] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 21:45:28] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_430 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 21:45:47] ppocr INFO: epoch: [431/500], global_step: 356300, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.942703, exp_rate<=1 : 0.978105, exp_rate<=2 : 0.994754, exp_rate<=3 : 0.996873, loss: 0.001391, avg_reader_cost: 0.03178 s, avg_batch_cost: 0.21835 s, avg_samples: 56.0, ips: 256.46643 samples/s, eta: 10:09:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:46:05] ppocr INFO: epoch: [431/500], global_step: 356400, lr: 0.000100, edit distance: 0.001104, exp_rate: 0.949435, exp_rate<=1 : 0.981299, exp_rate<=2 : 0.996101, exp_rate<=3 : 0.998098, loss: 0.001660, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17737 s, avg_samples: 56.0, ips: 315.71770 samples/s, eta: 10:08:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:46:23] ppocr INFO: epoch: [431/500], global_step: 356500, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.947805, exp_rate<=1 : 0.981208, exp_rate<=2 : 0.995744, exp_rate<=3 : 0.998286, loss: 0.001961, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18582 s, avg_samples: 56.0, ips: 301.37131 samples/s, eta: 10:08:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:46:41] ppocr INFO: epoch: [431/500], global_step: 356600, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.947870, exp_rate<=1 : 0.981770, exp_rate<=2 : 0.995842, exp_rate<=3 : 0.998569, loss: 0.001127, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17758 s, avg_samples: 56.0, ips: 315.35120 samples/s, eta: 10:08:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:47:00] ppocr INFO: epoch: [431/500], global_step: 356700, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.947375, exp_rate<=1 : 0.980973, exp_rate<=2 : 0.995868, exp_rate<=3 : 0.998527, loss: 0.000550, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18548 s, avg_samples: 56.0, ips: 301.91949 samples/s, eta: 10:07:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:47:18] ppocr INFO: epoch: [431/500], global_step: 356800, lr: 0.000100, edit distance: 0.001129, exp_rate: 0.948092, exp_rate<=1 : 0.981082, exp_rate<=2 : 0.995833, exp_rate<=3 : 0.998620, loss: 0.001189, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18105 s, avg_samples: 56.0, ips: 309.30651 samples/s, eta: 10:07:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:47:36] ppocr INFO: epoch: [431/500], global_step: 356900, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.947149, exp_rate<=1 : 0.980836, exp_rate<=2 : 0.995855, exp_rate<=3 : 0.998595, loss: 0.001289, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18426 s, avg_samples: 56.0, ips: 303.91694 samples/s, eta: 10:07:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:47:55] ppocr INFO: epoch: [431/500], global_step: 357000, lr: 0.000100, edit distance: 0.001134, exp_rate: 0.947385, exp_rate<=1 : 0.980909, exp_rate<=2 : 0.995784, exp_rate<=3 : 0.998622, loss: 0.001537, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19135 s, avg_samples: 56.0, ips: 292.66089 samples/s, eta: 10:06:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:48:15] ppocr INFO: epoch: [431/500], global_step: 357100, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.946011, exp_rate<=1 : 0.980160, exp_rate<=2 : 0.995613, exp_rate<=3 : 0.998524, loss: 0.001920, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19310 s, avg_samples: 56.0, ips: 290.00585 samples/s, eta: 10:06:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:48:34] ppocr INFO: epoch: [431/500], global_step: 357200, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.944660, exp_rate<=1 : 0.979279, exp_rate<=2 : 0.995197, exp_rate<=3 : 0.998293, loss: 0.001541, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19332 s, avg_samples: 56.0, ips: 289.67801 samples/s, eta: 10:06:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:48:53] ppocr INFO: epoch: [431/500], global_step: 357300, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.944665, exp_rate<=1 : 0.979154, exp_rate<=2 : 0.995264, exp_rate<=3 : 0.998287, loss: 0.001331, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19037 s, avg_samples: 56.0, ips: 294.16068 samples/s, eta: 10:05:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:49:12] ppocr INFO: epoch: [431/500], global_step: 357400, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.943707, exp_rate<=1 : 0.978612, exp_rate<=2 : 0.995135, exp_rate<=3 : 0.998245, loss: 0.002001, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19194 s, avg_samples: 56.0, ips: 291.75803 samples/s, eta: 10:05:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:49:31] ppocr INFO: epoch: [431/500], global_step: 357500, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.942351, exp_rate<=1 : 0.977918, exp_rate<=2 : 0.995053, exp_rate<=3 : 0.998230, loss: 0.001442, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18792 s, avg_samples: 56.0, ips: 297.99642 samples/s, eta: 10:05:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:49:50] ppocr INFO: epoch: [431/500], global_step: 357600, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.942152, exp_rate<=1 : 0.977863, exp_rate<=2 : 0.994996, exp_rate<=3 : 0.998231, loss: 0.001521, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18466 s, avg_samples: 56.0, ips: 303.26573 samples/s, eta: 10:04:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:50:08] ppocr INFO: epoch: [431/500], global_step: 357700, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.941326, exp_rate<=1 : 0.977400, exp_rate<=2 : 0.994814, exp_rate<=3 : 0.998184, loss: 0.001399, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18424 s, avg_samples: 56.0, ips: 303.95706 samples/s, eta: 10:04:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:50:27] ppocr INFO: epoch: [431/500], global_step: 357800, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.941136, exp_rate<=1 : 0.977507, exp_rate<=2 : 0.994823, exp_rate<=3 : 0.998250, loss: 0.001537, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19152 s, avg_samples: 56.0, ips: 292.39863 samples/s, eta: 10:04:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:50:46] ppocr INFO: epoch: [431/500], global_step: 357900, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.941258, exp_rate<=1 : 0.977595, exp_rate<=2 : 0.994887, exp_rate<=3 : 0.998280, loss: 0.001177, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19041 s, avg_samples: 56.0, ips: 294.09586 samples/s, eta: 10:03:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:51:05] ppocr INFO: epoch: [431/500], global_step: 358000, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.940692, exp_rate<=1 : 0.977340, exp_rate<=2 : 0.994848, exp_rate<=3 : 0.998229, loss: 0.002114, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18639 s, avg_samples: 56.0, ips: 300.44701 samples/s, eta: 10:03:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:51:23] ppocr INFO: epoch: [431/500], global_step: 358100, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.940657, exp_rate<=1 : 0.977490, exp_rate<=2 : 0.994872, exp_rate<=3 : 0.998228, loss: 0.001410, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17704 s, avg_samples: 56.0, ips: 316.31045 samples/s, eta: 10:03:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:51:41] ppocr INFO: epoch: [431/500], global_step: 358200, lr: 0.000100, edit distance: 0.001277, exp_rate: 0.940795, exp_rate<=1 : 0.977507, exp_rate<=2 : 0.994940, exp_rate<=3 : 0.998260, loss: 0.000973, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18062 s, avg_samples: 56.0, ips: 310.04714 samples/s, eta: 10:02:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:52:01] ppocr INFO: epoch: [431/500], global_step: 358300, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.940414, exp_rate<=1 : 0.977345, exp_rate<=2 : 0.994957, exp_rate<=3 : 0.998262, loss: 0.002068, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19670 s, avg_samples: 56.0, ips: 284.70074 samples/s, eta: 10:02:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:52:20] ppocr INFO: epoch: [431/500], global_step: 358400, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.940037, exp_rate<=1 : 0.977028, exp_rate<=2 : 0.994911, exp_rate<=3 : 0.998250, loss: 0.001528, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19082 s, avg_samples: 56.0, ips: 293.47594 samples/s, eta: 10:02:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:52:39] ppocr INFO: epoch: [431/500], global_step: 358500, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.940052, exp_rate<=1 : 0.977009, exp_rate<=2 : 0.994917, exp_rate<=3 : 0.998238, loss: 0.001077, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19442 s, avg_samples: 56.0, ips: 288.03934 samples/s, eta: 10:01:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:52:57] ppocr INFO: epoch: [431/500], global_step: 358600, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.940218, exp_rate<=1 : 0.977140, exp_rate<=2 : 0.994925, exp_rate<=3 : 0.998255, loss: 0.001120, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18127 s, avg_samples: 56.0, ips: 308.92696 samples/s, eta: 10:01:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:53:18] ppocr INFO: epoch: [431/500], global_step: 358700, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.939830, exp_rate<=1 : 0.976927, exp_rate<=2 : 0.994899, exp_rate<=3 : 0.998264, loss: 0.001605, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20278 s, avg_samples: 56.0, ips: 276.16573 samples/s, eta: 10:01:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:53:36] ppocr INFO: epoch: [431/500], global_step: 358800, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.939700, exp_rate<=1 : 0.976820, exp_rate<=2 : 0.994852, exp_rate<=3 : 0.998252, loss: 0.001887, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18420 s, avg_samples: 56.0, ips: 304.01288 samples/s, eta: 10:01:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:53:55] ppocr INFO: epoch: [431/500], global_step: 358900, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.939641, exp_rate<=1 : 0.976860, exp_rate<=2 : 0.994927, exp_rate<=3 : 0.998288, loss: 0.001556, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18704 s, avg_samples: 56.0, ips: 299.39718 samples/s, eta: 10:00:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:54:02] ppocr INFO: epoch: [431/500], global_step: 358940, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.939640, exp_rate<=1 : 0.976854, exp_rate<=2 : 0.994935, exp_rate<=3 : 0.998290, loss: 0.001441, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.07212 s, avg_samples: 22.4, ips: 310.58676 samples/s, eta: 10:00:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:54:04] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 21:54:16] ppocr INFO: epoch: [432/500], global_step: 359000, lr: 0.000100, edit distance: 0.000861, exp_rate: 0.959305, exp_rate<=1 : 0.986040, exp_rate<=2 : 0.995893, exp_rate<=3 : 0.998585, loss: 0.001151, avg_reader_cost: 0.02770 s, avg_batch_cost: 0.14096 s, avg_samples: 33.6, ips: 238.36407 samples/s, eta: 10:00:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:54:34] ppocr INFO: epoch: [432/500], global_step: 359100, lr: 0.000100, edit distance: 0.001043, exp_rate: 0.954462, exp_rate<=1 : 0.982383, exp_rate<=2 : 0.994637, exp_rate<=3 : 0.998102, loss: 0.001109, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18090 s, avg_samples: 56.0, ips: 309.56754 samples/s, eta: 10:00:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:54:54] ppocr INFO: epoch: [432/500], global_step: 359200, lr: 0.000100, edit distance: 0.001059, exp_rate: 0.951373, exp_rate<=1 : 0.981180, exp_rate<=2 : 0.994867, exp_rate<=3 : 0.998168, loss: 0.001207, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19362 s, avg_samples: 56.0, ips: 289.22810 samples/s, eta: 9:59:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:55:12] ppocr INFO: epoch: [432/500], global_step: 359300, lr: 0.000100, edit distance: 0.001011, exp_rate: 0.952815, exp_rate<=1 : 0.982501, exp_rate<=2 : 0.995324, exp_rate<=3 : 0.998523, loss: 0.000944, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18763 s, avg_samples: 56.0, ips: 298.46707 samples/s, eta: 9:59:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:55:30] ppocr INFO: epoch: [432/500], global_step: 359400, lr: 0.000100, edit distance: 0.001063, exp_rate: 0.953603, exp_rate<=1 : 0.982805, exp_rate<=2 : 0.995418, exp_rate<=3 : 0.998555, loss: 0.002099, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17401 s, avg_samples: 56.0, ips: 321.82697 samples/s, eta: 9:59:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:55:49] ppocr INFO: epoch: [432/500], global_step: 359500, lr: 0.000100, edit distance: 0.001095, exp_rate: 0.951909, exp_rate<=1 : 0.982335, exp_rate<=2 : 0.995524, exp_rate<=3 : 0.998519, loss: 0.001393, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18798 s, avg_samples: 56.0, ips: 297.89855 samples/s, eta: 9:58:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:56:07] ppocr INFO: epoch: [432/500], global_step: 359600, lr: 0.000100, edit distance: 0.001087, exp_rate: 0.951205, exp_rate<=1 : 0.981896, exp_rate<=2 : 0.995515, exp_rate<=3 : 0.998545, loss: 0.001060, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18914 s, avg_samples: 56.0, ips: 296.07078 samples/s, eta: 9:58:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:56:27] ppocr INFO: epoch: [432/500], global_step: 359700, lr: 0.000100, edit distance: 0.001107, exp_rate: 0.949969, exp_rate<=1 : 0.981486, exp_rate<=2 : 0.995670, exp_rate<=3 : 0.998616, loss: 0.001033, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19537 s, avg_samples: 56.0, ips: 286.63048 samples/s, eta: 9:58:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:56:45] ppocr INFO: epoch: [432/500], global_step: 359800, lr: 0.000100, edit distance: 0.001119, exp_rate: 0.949217, exp_rate<=1 : 0.981255, exp_rate<=2 : 0.995727, exp_rate<=3 : 0.998712, loss: 0.001081, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18358 s, avg_samples: 56.0, ips: 305.04430 samples/s, eta: 9:57:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:57:05] ppocr INFO: epoch: [432/500], global_step: 359900, lr: 0.000100, edit distance: 0.001127, exp_rate: 0.948067, exp_rate<=1 : 0.980767, exp_rate<=2 : 0.995581, exp_rate<=3 : 0.998656, loss: 0.001200, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19664 s, avg_samples: 56.0, ips: 284.78841 samples/s, eta: 9:57:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:57:23] ppocr INFO: epoch: [432/500], global_step: 360000, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.947352, exp_rate<=1 : 0.980447, exp_rate<=2 : 0.995396, exp_rate<=3 : 0.998567, loss: 0.001569, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18278 s, avg_samples: 56.0, ips: 306.37623 samples/s, eta: 9:57:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:57:43] ppocr INFO: epoch: [432/500], global_step: 360100, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.946142, exp_rate<=1 : 0.979823, exp_rate<=2 : 0.995102, exp_rate<=3 : 0.998463, loss: 0.001298, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19331 s, avg_samples: 56.0, ips: 289.68312 samples/s, eta: 9:56:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:58:02] ppocr INFO: epoch: [432/500], global_step: 360200, lr: 0.000100, edit distance: 0.001180, exp_rate: 0.945429, exp_rate<=1 : 0.979445, exp_rate<=2 : 0.994970, exp_rate<=3 : 0.998380, loss: 0.001825, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18991 s, avg_samples: 56.0, ips: 294.87155 samples/s, eta: 9:56:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:58:20] ppocr INFO: epoch: [432/500], global_step: 360300, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.945053, exp_rate<=1 : 0.979207, exp_rate<=2 : 0.995006, exp_rate<=3 : 0.998347, loss: 0.001023, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18403 s, avg_samples: 56.0, ips: 304.29027 samples/s, eta: 9:56:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:58:40] ppocr INFO: epoch: [432/500], global_step: 360400, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.943769, exp_rate<=1 : 0.978479, exp_rate<=2 : 0.994897, exp_rate<=3 : 0.998263, loss: 0.002393, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19898 s, avg_samples: 56.0, ips: 281.43496 samples/s, eta: 9:55:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:58:59] ppocr INFO: epoch: [432/500], global_step: 360500, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.943035, exp_rate<=1 : 0.978077, exp_rate<=2 : 0.994823, exp_rate<=3 : 0.998247, loss: 0.002226, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18750 s, avg_samples: 56.0, ips: 298.67288 samples/s, eta: 9:55:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:59:18] ppocr INFO: epoch: [432/500], global_step: 360600, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.942364, exp_rate<=1 : 0.977969, exp_rate<=2 : 0.994822, exp_rate<=3 : 0.998305, loss: 0.001540, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19418 s, avg_samples: 56.0, ips: 288.39516 samples/s, eta: 9:55:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:59:36] ppocr INFO: epoch: [432/500], global_step: 360700, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.942880, exp_rate<=1 : 0.978210, exp_rate<=2 : 0.994930, exp_rate<=3 : 0.998333, loss: 0.001419, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17739 s, avg_samples: 56.0, ips: 315.68858 samples/s, eta: 9:54:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 21:59:54] ppocr INFO: epoch: [432/500], global_step: 360800, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.943177, exp_rate<=1 : 0.978368, exp_rate<=2 : 0.994983, exp_rate<=3 : 0.998325, loss: 0.001499, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18189 s, avg_samples: 56.0, ips: 307.87432 samples/s, eta: 9:54:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:00:12] ppocr INFO: epoch: [432/500], global_step: 360900, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.942901, exp_rate<=1 : 0.978248, exp_rate<=2 : 0.994955, exp_rate<=3 : 0.998281, loss: 0.002003, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18034 s, avg_samples: 56.0, ips: 310.52490 samples/s, eta: 9:54:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:00:31] ppocr INFO: epoch: [432/500], global_step: 361000, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.942156, exp_rate<=1 : 0.977928, exp_rate<=2 : 0.994865, exp_rate<=3 : 0.998268, loss: 0.001419, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19011 s, avg_samples: 56.0, ips: 294.55898 samples/s, eta: 9:54:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:00:50] ppocr INFO: epoch: [432/500], global_step: 361100, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.941986, exp_rate<=1 : 0.977958, exp_rate<=2 : 0.994855, exp_rate<=3 : 0.998251, loss: 0.001798, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18565 s, avg_samples: 56.0, ips: 301.63856 samples/s, eta: 9:53:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:01:09] ppocr INFO: epoch: [432/500], global_step: 361200, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.941726, exp_rate<=1 : 0.977811, exp_rate<=2 : 0.994817, exp_rate<=3 : 0.998214, loss: 0.001452, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18994 s, avg_samples: 56.0, ips: 294.83432 samples/s, eta: 9:53:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:01:27] ppocr INFO: epoch: [432/500], global_step: 361300, lr: 0.000100, edit distance: 0.001277, exp_rate: 0.941680, exp_rate<=1 : 0.977888, exp_rate<=2 : 0.994841, exp_rate<=3 : 0.998221, loss: 0.001394, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17907 s, avg_samples: 56.0, ips: 312.72688 samples/s, eta: 9:53:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:01:46] ppocr INFO: epoch: [432/500], global_step: 361400, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.941556, exp_rate<=1 : 0.977867, exp_rate<=2 : 0.994896, exp_rate<=3 : 0.998210, loss: 0.001202, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19279 s, avg_samples: 56.0, ips: 290.47262 samples/s, eta: 9:52:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:02:05] ppocr INFO: epoch: [432/500], global_step: 361500, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.941039, exp_rate<=1 : 0.977756, exp_rate<=2 : 0.994889, exp_rate<=3 : 0.998201, loss: 0.001434, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18835 s, avg_samples: 56.0, ips: 297.31299 samples/s, eta: 9:52:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:02:23] ppocr INFO: epoch: [432/500], global_step: 361600, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.941368, exp_rate<=1 : 0.977836, exp_rate<=2 : 0.994940, exp_rate<=3 : 0.998241, loss: 0.001432, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18076 s, avg_samples: 56.0, ips: 309.79565 samples/s, eta: 9:52:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:02:37] ppocr INFO: epoch: [432/500], global_step: 361680, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.941191, exp_rate<=1 : 0.977715, exp_rate<=2 : 0.994955, exp_rate<=3 : 0.998238, loss: 0.001846, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.14256 s, avg_samples: 44.8, ips: 314.24272 samples/s, eta: 9:51:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:02:40] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 22:02:44] ppocr INFO: epoch: [433/500], global_step: 361700, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.940301, exp_rate<=1 : 0.983773, exp_rate<=2 : 0.996591, exp_rate<=3 : 0.999034, loss: 0.001727, avg_reader_cost: 0.02581 s, avg_batch_cost: 0.06288 s, avg_samples: 11.2, ips: 178.11203 samples/s, eta: 9:51:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:03:02] ppocr INFO: epoch: [433/500], global_step: 361800, lr: 0.000100, edit distance: 0.001565, exp_rate: 0.941904, exp_rate<=1 : 0.978668, exp_rate<=2 : 0.993514, exp_rate<=3 : 0.998407, loss: 0.001315, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18010 s, avg_samples: 56.0, ips: 310.93134 samples/s, eta: 9:51:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:03:20] ppocr INFO: epoch: [433/500], global_step: 361900, lr: 0.000100, edit distance: 0.001485, exp_rate: 0.940418, exp_rate<=1 : 0.978141, exp_rate<=2 : 0.994048, exp_rate<=3 : 0.998021, loss: 0.001226, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18446 s, avg_samples: 56.0, ips: 303.59025 samples/s, eta: 9:51:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:03:39] ppocr INFO: epoch: [433/500], global_step: 362000, lr: 0.000100, edit distance: 0.001428, exp_rate: 0.941814, exp_rate<=1 : 0.978980, exp_rate<=2 : 0.994536, exp_rate<=3 : 0.998045, loss: 0.000612, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18745 s, avg_samples: 56.0, ips: 298.73888 samples/s, eta: 9:50:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:03:58] ppocr INFO: epoch: [433/500], global_step: 362100, lr: 0.000100, edit distance: 0.001359, exp_rate: 0.941991, exp_rate<=1 : 0.978523, exp_rate<=2 : 0.994459, exp_rate<=3 : 0.998089, loss: 0.001333, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19006 s, avg_samples: 56.0, ips: 294.63786 samples/s, eta: 9:50:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:04:17] ppocr INFO: epoch: [433/500], global_step: 362200, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.941995, exp_rate<=1 : 0.978494, exp_rate<=2 : 0.994751, exp_rate<=3 : 0.998099, loss: 0.001242, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19423 s, avg_samples: 56.0, ips: 288.32183 samples/s, eta: 9:50:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:04:36] ppocr INFO: epoch: [433/500], global_step: 362300, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.942566, exp_rate<=1 : 0.979018, exp_rate<=2 : 0.995109, exp_rate<=3 : 0.998290, loss: 0.001590, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18172 s, avg_samples: 56.0, ips: 308.16692 samples/s, eta: 9:49:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:04:55] ppocr INFO: epoch: [433/500], global_step: 362400, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.941346, exp_rate<=1 : 0.978632, exp_rate<=2 : 0.995068, exp_rate<=3 : 0.998216, loss: 0.001505, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19052 s, avg_samples: 56.0, ips: 293.93480 samples/s, eta: 9:49:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:05:15] ppocr INFO: epoch: [433/500], global_step: 362500, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.941218, exp_rate<=1 : 0.978378, exp_rate<=2 : 0.994991, exp_rate<=3 : 0.998281, loss: 0.001674, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19812 s, avg_samples: 56.0, ips: 282.65677 samples/s, eta: 9:49:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:05:34] ppocr INFO: epoch: [433/500], global_step: 362600, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.941707, exp_rate<=1 : 0.978390, exp_rate<=2 : 0.994909, exp_rate<=3 : 0.998326, loss: 0.001708, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19676 s, avg_samples: 56.0, ips: 284.60980 samples/s, eta: 9:48:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:05:54] ppocr INFO: epoch: [433/500], global_step: 362700, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.941447, exp_rate<=1 : 0.978493, exp_rate<=2 : 0.994861, exp_rate<=3 : 0.998334, loss: 0.001512, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.36532 samples/s, eta: 9:48:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:06:14] ppocr INFO: epoch: [433/500], global_step: 362800, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.941371, exp_rate<=1 : 0.978163, exp_rate<=2 : 0.994809, exp_rate<=3 : 0.998266, loss: 0.002029, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19936 s, avg_samples: 56.0, ips: 280.89475 samples/s, eta: 9:48:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:06:31] ppocr INFO: epoch: [433/500], global_step: 362900, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.942414, exp_rate<=1 : 0.978686, exp_rate<=2 : 0.995022, exp_rate<=3 : 0.998363, loss: 0.001421, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16868 s, avg_samples: 56.0, ips: 331.99017 samples/s, eta: 9:47:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:06:49] ppocr INFO: epoch: [433/500], global_step: 363000, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.942509, exp_rate<=1 : 0.978687, exp_rate<=2 : 0.994986, exp_rate<=3 : 0.998380, loss: 0.001388, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18594 s, avg_samples: 56.0, ips: 301.16696 samples/s, eta: 9:47:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:07:08] ppocr INFO: epoch: [433/500], global_step: 363100, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.941766, exp_rate<=1 : 0.978335, exp_rate<=2 : 0.994978, exp_rate<=3 : 0.998435, loss: 0.001498, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18688 s, avg_samples: 56.0, ips: 299.65294 samples/s, eta: 9:47:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:07:26] ppocr INFO: epoch: [433/500], global_step: 363200, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.942001, exp_rate<=1 : 0.978259, exp_rate<=2 : 0.995047, exp_rate<=3 : 0.998416, loss: 0.001140, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17859 s, avg_samples: 56.0, ips: 313.56006 samples/s, eta: 9:46:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:07:43] ppocr INFO: epoch: [433/500], global_step: 363300, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.942431, exp_rate<=1 : 0.978523, exp_rate<=2 : 0.995166, exp_rate<=3 : 0.998403, loss: 0.001393, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17455 s, avg_samples: 56.0, ips: 320.81930 samples/s, eta: 9:46:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:08:02] ppocr INFO: epoch: [433/500], global_step: 363400, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.942123, exp_rate<=1 : 0.978448, exp_rate<=2 : 0.995195, exp_rate<=3 : 0.998403, loss: 0.001799, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19275 s, avg_samples: 56.0, ips: 290.53135 samples/s, eta: 9:46:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:08:20] ppocr INFO: epoch: [433/500], global_step: 363500, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.942227, exp_rate<=1 : 0.978427, exp_rate<=2 : 0.995203, exp_rate<=3 : 0.998454, loss: 0.001311, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17807 s, avg_samples: 56.0, ips: 314.47664 samples/s, eta: 9:46:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:08:39] ppocr INFO: epoch: [433/500], global_step: 363600, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.942260, exp_rate<=1 : 0.978463, exp_rate<=2 : 0.995163, exp_rate<=3 : 0.998414, loss: 0.001210, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18519 s, avg_samples: 56.0, ips: 302.39690 samples/s, eta: 9:45:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:08:59] ppocr INFO: epoch: [433/500], global_step: 363700, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.941723, exp_rate<=1 : 0.978166, exp_rate<=2 : 0.995071, exp_rate<=3 : 0.998401, loss: 0.001392, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19812 s, avg_samples: 56.0, ips: 282.66284 samples/s, eta: 9:45:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:09:16] ppocr INFO: epoch: [433/500], global_step: 363800, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.941926, exp_rate<=1 : 0.978282, exp_rate<=2 : 0.995143, exp_rate<=3 : 0.998435, loss: 0.001666, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17666 s, avg_samples: 56.0, ips: 316.99235 samples/s, eta: 9:45:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:09:36] ppocr INFO: epoch: [433/500], global_step: 363900, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.941653, exp_rate<=1 : 0.978026, exp_rate<=2 : 0.995088, exp_rate<=3 : 0.998417, loss: 0.001958, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19213 s, avg_samples: 56.0, ips: 291.47338 samples/s, eta: 9:44:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:09:54] ppocr INFO: epoch: [433/500], global_step: 364000, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.941611, exp_rate<=1 : 0.978098, exp_rate<=2 : 0.995107, exp_rate<=3 : 0.998433, loss: 0.001356, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18410 s, avg_samples: 56.0, ips: 304.18794 samples/s, eta: 9:44:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:10:12] ppocr INFO: epoch: [433/500], global_step: 364100, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.941585, exp_rate<=1 : 0.977974, exp_rate<=2 : 0.995136, exp_rate<=3 : 0.998441, loss: 0.001524, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18342 s, avg_samples: 56.0, ips: 305.30382 samples/s, eta: 9:44:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:10:33] ppocr INFO: epoch: [433/500], global_step: 364200, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.941122, exp_rate<=1 : 0.977674, exp_rate<=2 : 0.995127, exp_rate<=3 : 0.998437, loss: 0.001112, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20531 s, avg_samples: 56.0, ips: 272.75645 samples/s, eta: 9:43:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:10:52] ppocr INFO: epoch: [433/500], global_step: 364300, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.940999, exp_rate<=1 : 0.977667, exp_rate<=2 : 0.995157, exp_rate<=3 : 0.998472, loss: 0.001880, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18757 s, avg_samples: 56.0, ips: 298.55293 samples/s, eta: 9:43:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:11:10] ppocr INFO: epoch: [433/500], global_step: 364400, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.941108, exp_rate<=1 : 0.977699, exp_rate<=2 : 0.995155, exp_rate<=3 : 0.998449, loss: 0.001136, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18529 s, avg_samples: 56.0, ips: 302.23228 samples/s, eta: 9:43:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:11:14] ppocr INFO: epoch: [433/500], global_step: 364420, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.941066, exp_rate<=1 : 0.977638, exp_rate<=2 : 0.995121, exp_rate<=3 : 0.998429, loss: 0.001170, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03690 s, avg_samples: 11.2, ips: 303.55086 samples/s, eta: 9:43:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:11:16] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 22:11:31] ppocr INFO: epoch: [434/500], global_step: 364500, lr: 0.000100, edit distance: 0.000823, exp_rate: 0.961307, exp_rate<=1 : 0.984408, exp_rate<=2 : 0.997610, exp_rate<=3 : 0.998657, loss: 0.000670, avg_reader_cost: 0.02515 s, avg_batch_cost: 0.17149 s, avg_samples: 44.8, ips: 261.23924 samples/s, eta: 9:42:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:11:51] ppocr INFO: epoch: [434/500], global_step: 364600, lr: 0.000100, edit distance: 0.001118, exp_rate: 0.947814, exp_rate<=1 : 0.979660, exp_rate<=2 : 0.994485, exp_rate<=3 : 0.997502, loss: 0.001568, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19514 s, avg_samples: 56.0, ips: 286.97138 samples/s, eta: 9:42:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:12:09] ppocr INFO: epoch: [434/500], global_step: 364700, lr: 0.000100, edit distance: 0.001135, exp_rate: 0.947630, exp_rate<=1 : 0.980042, exp_rate<=2 : 0.995157, exp_rate<=3 : 0.998218, loss: 0.001177, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18048 s, avg_samples: 56.0, ips: 310.29074 samples/s, eta: 9:42:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:12:28] ppocr INFO: epoch: [434/500], global_step: 364800, lr: 0.000100, edit distance: 0.001110, exp_rate: 0.946502, exp_rate<=1 : 0.979877, exp_rate<=2 : 0.995476, exp_rate<=3 : 0.998273, loss: 0.001076, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19685 s, avg_samples: 56.0, ips: 284.47433 samples/s, eta: 9:41:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:12:48] ppocr INFO: epoch: [434/500], global_step: 364900, lr: 0.000100, edit distance: 0.001097, exp_rate: 0.945955, exp_rate<=1 : 0.979705, exp_rate<=2 : 0.995159, exp_rate<=3 : 0.998111, loss: 0.001305, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19084 s, avg_samples: 56.0, ips: 293.44267 samples/s, eta: 9:41:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:13:07] ppocr INFO: epoch: [434/500], global_step: 365000, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.945895, exp_rate<=1 : 0.979547, exp_rate<=2 : 0.994983, exp_rate<=3 : 0.997993, loss: 0.000805, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19522 s, avg_samples: 56.0, ips: 286.85674 samples/s, eta: 9:41:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:13:25] ppocr INFO: epoch: [434/500], global_step: 365100, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.945476, exp_rate<=1 : 0.979680, exp_rate<=2 : 0.995042, exp_rate<=3 : 0.998014, loss: 0.001341, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18253 s, avg_samples: 56.0, ips: 306.79132 samples/s, eta: 9:40:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:13:45] ppocr INFO: epoch: [434/500], global_step: 365200, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.944917, exp_rate<=1 : 0.979646, exp_rate<=2 : 0.995155, exp_rate<=3 : 0.997975, loss: 0.001725, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19161 s, avg_samples: 56.0, ips: 292.25861 samples/s, eta: 9:40:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:14:03] ppocr INFO: epoch: [434/500], global_step: 365300, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.944788, exp_rate<=1 : 0.979712, exp_rate<=2 : 0.995197, exp_rate<=3 : 0.998047, loss: 0.001273, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18637 s, avg_samples: 56.0, ips: 300.47616 samples/s, eta: 9:40:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:14:22] ppocr INFO: epoch: [434/500], global_step: 365400, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.943669, exp_rate<=1 : 0.978827, exp_rate<=2 : 0.994821, exp_rate<=3 : 0.997920, loss: 0.001235, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18551 s, avg_samples: 56.0, ips: 301.86339 samples/s, eta: 9:39:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:14:41] ppocr INFO: epoch: [434/500], global_step: 365500, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.943624, exp_rate<=1 : 0.978787, exp_rate<=2 : 0.994846, exp_rate<=3 : 0.997933, loss: 0.001165, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19382 s, avg_samples: 56.0, ips: 288.93414 samples/s, eta: 9:39:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:15:00] ppocr INFO: epoch: [434/500], global_step: 365600, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.943898, exp_rate<=1 : 0.978818, exp_rate<=2 : 0.994966, exp_rate<=3 : 0.998062, loss: 0.001214, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18497 s, avg_samples: 56.0, ips: 302.74589 samples/s, eta: 9:39:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:15:18] ppocr INFO: epoch: [434/500], global_step: 365700, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.944134, exp_rate<=1 : 0.978727, exp_rate<=2 : 0.994907, exp_rate<=3 : 0.998052, loss: 0.001481, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18265 s, avg_samples: 56.0, ips: 306.60442 samples/s, eta: 9:38:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:15:36] ppocr INFO: epoch: [434/500], global_step: 365800, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.944420, exp_rate<=1 : 0.979004, exp_rate<=2 : 0.994968, exp_rate<=3 : 0.998080, loss: 0.001152, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17626 s, avg_samples: 56.0, ips: 317.71347 samples/s, eta: 9:38:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:15:54] ppocr INFO: epoch: [434/500], global_step: 365900, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.944444, exp_rate<=1 : 0.978838, exp_rate<=2 : 0.994913, exp_rate<=3 : 0.998102, loss: 0.001604, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18749 s, avg_samples: 56.0, ips: 298.67589 samples/s, eta: 9:38:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:16:13] ppocr INFO: epoch: [434/500], global_step: 366000, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.944629, exp_rate<=1 : 0.978921, exp_rate<=2 : 0.994966, exp_rate<=3 : 0.998114, loss: 0.001303, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18731 s, avg_samples: 56.0, ips: 298.97527 samples/s, eta: 9:38:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:16:31] ppocr INFO: epoch: [434/500], global_step: 366100, lr: 0.000100, edit distance: 0.001160, exp_rate: 0.943756, exp_rate<=1 : 0.978442, exp_rate<=2 : 0.994704, exp_rate<=3 : 0.997992, loss: 0.001298, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18052 s, avg_samples: 56.0, ips: 310.22241 samples/s, eta: 9:37:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:16:49] ppocr INFO: epoch: [434/500], global_step: 366200, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.943473, exp_rate<=1 : 0.978309, exp_rate<=2 : 0.994682, exp_rate<=3 : 0.997972, loss: 0.001898, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18186 s, avg_samples: 56.0, ips: 307.93376 samples/s, eta: 9:37:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:17:07] ppocr INFO: epoch: [434/500], global_step: 366300, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.943456, exp_rate<=1 : 0.978415, exp_rate<=2 : 0.994651, exp_rate<=3 : 0.997996, loss: 0.001781, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17208 s, avg_samples: 56.0, ips: 325.42060 samples/s, eta: 9:37:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:17:26] ppocr INFO: epoch: [434/500], global_step: 366400, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.943007, exp_rate<=1 : 0.978081, exp_rate<=2 : 0.994601, exp_rate<=3 : 0.997933, loss: 0.001963, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19178 s, avg_samples: 56.0, ips: 291.99864 samples/s, eta: 9:36:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:17:45] ppocr INFO: epoch: [434/500], global_step: 366500, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.942742, exp_rate<=1 : 0.977948, exp_rate<=2 : 0.994568, exp_rate<=3 : 0.997915, loss: 0.001373, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18801 s, avg_samples: 56.0, ips: 297.86028 samples/s, eta: 9:36:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:18:04] ppocr INFO: epoch: [434/500], global_step: 366600, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.942146, exp_rate<=1 : 0.977803, exp_rate<=2 : 0.994513, exp_rate<=3 : 0.997886, loss: 0.001601, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19143 s, avg_samples: 56.0, ips: 292.53948 samples/s, eta: 9:36:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:18:23] ppocr INFO: epoch: [434/500], global_step: 366700, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.941999, exp_rate<=1 : 0.977743, exp_rate<=2 : 0.994497, exp_rate<=3 : 0.997887, loss: 0.001419, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18687 s, avg_samples: 56.0, ips: 299.67311 samples/s, eta: 9:35:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:18:41] ppocr INFO: epoch: [434/500], global_step: 366800, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.941610, exp_rate<=1 : 0.977553, exp_rate<=2 : 0.994486, exp_rate<=3 : 0.997868, loss: 0.001439, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18698 s, avg_samples: 56.0, ips: 299.48983 samples/s, eta: 9:35:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:19:02] ppocr INFO: epoch: [434/500], global_step: 366900, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.941339, exp_rate<=1 : 0.977493, exp_rate<=2 : 0.994497, exp_rate<=3 : 0.997852, loss: 0.000954, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20368 s, avg_samples: 56.0, ips: 274.93780 samples/s, eta: 9:35:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:19:22] ppocr INFO: epoch: [434/500], global_step: 367000, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.941028, exp_rate<=1 : 0.977349, exp_rate<=2 : 0.994502, exp_rate<=3 : 0.997895, loss: 0.001329, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20050 s, avg_samples: 56.0, ips: 279.29829 samples/s, eta: 9:34:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:19:40] ppocr INFO: epoch: [434/500], global_step: 367100, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.940822, exp_rate<=1 : 0.977298, exp_rate<=2 : 0.994477, exp_rate<=3 : 0.997900, loss: 0.001468, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18270 s, avg_samples: 56.0, ips: 306.51183 samples/s, eta: 9:34:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:19:51] ppocr INFO: epoch: [434/500], global_step: 367160, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.940807, exp_rate<=1 : 0.977263, exp_rate<=2 : 0.994468, exp_rate<=3 : 0.997920, loss: 0.001504, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10985 s, avg_samples: 33.6, ips: 305.86721 samples/s, eta: 9:34:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:19:53] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 22:20:02] ppocr INFO: epoch: [435/500], global_step: 367200, lr: 0.000100, edit distance: 0.001005, exp_rate: 0.940244, exp_rate<=1 : 0.979167, exp_rate<=2 : 0.994857, exp_rate<=3 : 0.998243, loss: 0.001507, avg_reader_cost: 0.02549 s, avg_batch_cost: 0.10790 s, avg_samples: 22.4, ips: 207.59730 samples/s, eta: 9:34:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:20:20] ppocr INFO: epoch: [435/500], global_step: 367300, lr: 0.000100, edit distance: 0.001080, exp_rate: 0.942307, exp_rate<=1 : 0.979487, exp_rate<=2 : 0.995709, exp_rate<=3 : 0.998383, loss: 0.001681, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18365 s, avg_samples: 56.0, ips: 304.93027 samples/s, eta: 9:33:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:20:38] ppocr INFO: epoch: [435/500], global_step: 367400, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.941965, exp_rate<=1 : 0.979982, exp_rate<=2 : 0.996006, exp_rate<=3 : 0.998567, loss: 0.001172, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18137 s, avg_samples: 56.0, ips: 308.76646 samples/s, eta: 9:33:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:20:56] ppocr INFO: epoch: [435/500], global_step: 367500, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.943878, exp_rate<=1 : 0.980283, exp_rate<=2 : 0.995529, exp_rate<=3 : 0.998505, loss: 0.001454, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18050 s, avg_samples: 56.0, ips: 310.25757 samples/s, eta: 9:33:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:21:16] ppocr INFO: epoch: [435/500], global_step: 367600, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.943955, exp_rate<=1 : 0.980184, exp_rate<=2 : 0.995348, exp_rate<=3 : 0.998405, loss: 0.001619, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19241 s, avg_samples: 56.0, ips: 291.04078 samples/s, eta: 9:32:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:21:34] ppocr INFO: epoch: [435/500], global_step: 367700, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.944153, exp_rate<=1 : 0.980084, exp_rate<=2 : 0.995266, exp_rate<=3 : 0.998289, loss: 0.001482, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17904 s, avg_samples: 56.0, ips: 312.77420 samples/s, eta: 9:32:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:21:52] ppocr INFO: epoch: [435/500], global_step: 367800, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.944484, exp_rate<=1 : 0.979565, exp_rate<=2 : 0.995158, exp_rate<=3 : 0.998250, loss: 0.001931, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18348 s, avg_samples: 56.0, ips: 305.20403 samples/s, eta: 9:32:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:22:11] ppocr INFO: epoch: [435/500], global_step: 367900, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.943422, exp_rate<=1 : 0.978965, exp_rate<=2 : 0.995059, exp_rate<=3 : 0.998203, loss: 0.001843, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19363 s, avg_samples: 56.0, ips: 289.20686 samples/s, eta: 9:31:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:22:31] ppocr INFO: epoch: [435/500], global_step: 368000, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.942278, exp_rate<=1 : 0.978215, exp_rate<=2 : 0.994969, exp_rate<=3 : 0.998160, loss: 0.002254, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19225 s, avg_samples: 56.0, ips: 291.29264 samples/s, eta: 9:31:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:22:49] ppocr INFO: epoch: [435/500], global_step: 368100, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.942210, exp_rate<=1 : 0.978151, exp_rate<=2 : 0.994877, exp_rate<=3 : 0.998058, loss: 0.001570, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18808 s, avg_samples: 56.0, ips: 297.73963 samples/s, eta: 9:31:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:23:07] ppocr INFO: epoch: [435/500], global_step: 368200, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.943337, exp_rate<=1 : 0.978448, exp_rate<=2 : 0.994940, exp_rate<=3 : 0.998086, loss: 0.001699, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17599 s, avg_samples: 56.0, ips: 318.19221 samples/s, eta: 9:30:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:23:26] ppocr INFO: epoch: [435/500], global_step: 368300, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.943286, exp_rate<=1 : 0.978495, exp_rate<=2 : 0.994977, exp_rate<=3 : 0.998104, loss: 0.001470, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18729 s, avg_samples: 56.0, ips: 299.00299 samples/s, eta: 9:30:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:23:44] ppocr INFO: epoch: [435/500], global_step: 368400, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.943849, exp_rate<=1 : 0.978773, exp_rate<=2 : 0.995117, exp_rate<=3 : 0.998124, loss: 0.001379, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17860 s, avg_samples: 56.0, ips: 313.55514 samples/s, eta: 9:30:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:24:03] ppocr INFO: epoch: [435/500], global_step: 368500, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.943889, exp_rate<=1 : 0.978827, exp_rate<=2 : 0.995151, exp_rate<=3 : 0.998117, loss: 0.001240, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19434 s, avg_samples: 56.0, ips: 288.15372 samples/s, eta: 9:30:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:24:23] ppocr INFO: epoch: [435/500], global_step: 368600, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.943313, exp_rate<=1 : 0.978339, exp_rate<=2 : 0.994951, exp_rate<=3 : 0.998043, loss: 0.001514, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19537 s, avg_samples: 56.0, ips: 286.63743 samples/s, eta: 9:29:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:24:41] ppocr INFO: epoch: [435/500], global_step: 368700, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.942410, exp_rate<=1 : 0.978096, exp_rate<=2 : 0.994830, exp_rate<=3 : 0.998045, loss: 0.001312, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18574 s, avg_samples: 56.0, ips: 301.50037 samples/s, eta: 9:29:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:24:59] ppocr INFO: epoch: [435/500], global_step: 368800, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.942788, exp_rate<=1 : 0.978317, exp_rate<=2 : 0.994892, exp_rate<=3 : 0.998094, loss: 0.001832, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17672 s, avg_samples: 56.0, ips: 316.88612 samples/s, eta: 9:29:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:25:18] ppocr INFO: epoch: [435/500], global_step: 368900, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.941536, exp_rate<=1 : 0.977850, exp_rate<=2 : 0.994756, exp_rate<=3 : 0.998086, loss: 0.002386, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18739 s, avg_samples: 56.0, ips: 298.84850 samples/s, eta: 9:28:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:25:38] ppocr INFO: epoch: [435/500], global_step: 369000, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.940811, exp_rate<=1 : 0.977518, exp_rate<=2 : 0.994759, exp_rate<=3 : 0.998090, loss: 0.001061, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20007 s, avg_samples: 56.0, ips: 279.89531 samples/s, eta: 9:28:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:25:55] ppocr INFO: epoch: [435/500], global_step: 369100, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.940660, exp_rate<=1 : 0.977475, exp_rate<=2 : 0.994792, exp_rate<=3 : 0.998131, loss: 0.001393, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17314 s, avg_samples: 56.0, ips: 323.43774 samples/s, eta: 9:28:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:26:14] ppocr INFO: epoch: [435/500], global_step: 369200, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.940365, exp_rate<=1 : 0.977414, exp_rate<=2 : 0.994781, exp_rate<=3 : 0.998123, loss: 0.001657, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19405 s, avg_samples: 56.0, ips: 288.58311 samples/s, eta: 9:27:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:26:33] ppocr INFO: epoch: [435/500], global_step: 369300, lr: 0.000100, edit distance: 0.001337, exp_rate: 0.940475, exp_rate<=1 : 0.977525, exp_rate<=2 : 0.994867, exp_rate<=3 : 0.998146, loss: 0.001387, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18725 s, avg_samples: 56.0, ips: 299.06977 samples/s, eta: 9:27:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:26:52] ppocr INFO: epoch: [435/500], global_step: 369400, lr: 0.000100, edit distance: 0.001341, exp_rate: 0.940487, exp_rate<=1 : 0.977646, exp_rate<=2 : 0.994868, exp_rate<=3 : 0.998155, loss: 0.001483, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18852 s, avg_samples: 56.0, ips: 297.04517 samples/s, eta: 9:27:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:27:12] ppocr INFO: epoch: [435/500], global_step: 369500, lr: 0.000100, edit distance: 0.001342, exp_rate: 0.940114, exp_rate<=1 : 0.977465, exp_rate<=2 : 0.994910, exp_rate<=3 : 0.998161, loss: 0.001918, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19918 s, avg_samples: 56.0, ips: 281.15778 samples/s, eta: 9:26:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:27:30] ppocr INFO: epoch: [435/500], global_step: 369600, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.940519, exp_rate<=1 : 0.977633, exp_rate<=2 : 0.994945, exp_rate<=3 : 0.998193, loss: 0.001881, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17671 s, avg_samples: 56.0, ips: 316.89880 samples/s, eta: 9:26:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:27:49] ppocr INFO: epoch: [435/500], global_step: 369700, lr: 0.000100, edit distance: 0.001353, exp_rate: 0.940044, exp_rate<=1 : 0.977382, exp_rate<=2 : 0.994808, exp_rate<=3 : 0.998133, loss: 0.001289, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18878 s, avg_samples: 56.0, ips: 296.63583 samples/s, eta: 9:26:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:28:08] ppocr INFO: epoch: [435/500], global_step: 369800, lr: 0.000100, edit distance: 0.001360, exp_rate: 0.939799, exp_rate<=1 : 0.977267, exp_rate<=2 : 0.994793, exp_rate<=3 : 0.998122, loss: 0.002474, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19615 s, avg_samples: 56.0, ips: 285.49870 samples/s, eta: 9:25:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:28:27] ppocr INFO: epoch: [435/500], global_step: 369900, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.939574, exp_rate<=1 : 0.977099, exp_rate<=2 : 0.994640, exp_rate<=3 : 0.998067, loss: 0.001491, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18978 s, avg_samples: 56.0, ips: 295.08516 samples/s, eta: 9:25:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:28:29] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 22:28:30] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_435 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 22:28:49] ppocr INFO: epoch: [436/500], global_step: 370000, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.948133, exp_rate<=1 : 0.980418, exp_rate<=2 : 0.995103, exp_rate<=3 : 0.998394, loss: 0.001362, avg_reader_cost: 0.03244 s, avg_batch_cost: 0.21745 s, avg_samples: 56.0, ips: 257.52557 samples/s, eta: 9:25:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:29:09] ppocr INFO: epoch: [436/500], global_step: 370100, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.942538, exp_rate<=1 : 0.977128, exp_rate<=2 : 0.994676, exp_rate<=3 : 0.997597, loss: 0.001386, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19546 s, avg_samples: 56.0, ips: 286.50149 samples/s, eta: 9:24:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:29:27] ppocr INFO: epoch: [436/500], global_step: 370200, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.946848, exp_rate<=1 : 0.979774, exp_rate<=2 : 0.995636, exp_rate<=3 : 0.998156, loss: 0.001425, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17899 s, avg_samples: 56.0, ips: 312.86597 samples/s, eta: 9:24:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:29:45] ppocr INFO: epoch: [436/500], global_step: 370300, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.946918, exp_rate<=1 : 0.979228, exp_rate<=2 : 0.995764, exp_rate<=3 : 0.998294, loss: 0.001364, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18942 s, avg_samples: 56.0, ips: 295.63785 samples/s, eta: 9:24:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:30:05] ppocr INFO: epoch: [436/500], global_step: 370400, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.945830, exp_rate<=1 : 0.979133, exp_rate<=2 : 0.995626, exp_rate<=3 : 0.998107, loss: 0.001157, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19351 s, avg_samples: 56.0, ips: 289.39520 samples/s, eta: 9:23:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:30:24] ppocr INFO: epoch: [436/500], global_step: 370500, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.945945, exp_rate<=1 : 0.978911, exp_rate<=2 : 0.995513, exp_rate<=3 : 0.998004, loss: 0.001420, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19387 s, avg_samples: 56.0, ips: 288.85836 samples/s, eta: 9:23:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:30:45] ppocr INFO: epoch: [436/500], global_step: 370600, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.944849, exp_rate<=1 : 0.978610, exp_rate<=2 : 0.995585, exp_rate<=3 : 0.998119, loss: 0.001420, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20724 s, avg_samples: 56.0, ips: 270.22312 samples/s, eta: 9:23:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:31:03] ppocr INFO: epoch: [436/500], global_step: 370700, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.945079, exp_rate<=1 : 0.978599, exp_rate<=2 : 0.995595, exp_rate<=3 : 0.998170, loss: 0.001266, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18013 s, avg_samples: 56.0, ips: 310.88972 samples/s, eta: 9:23:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:31:22] ppocr INFO: epoch: [436/500], global_step: 370800, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.945028, exp_rate<=1 : 0.978563, exp_rate<=2 : 0.995593, exp_rate<=3 : 0.998269, loss: 0.001169, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18694 s, avg_samples: 56.0, ips: 299.56876 samples/s, eta: 9:22:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:31:41] ppocr INFO: epoch: [436/500], global_step: 370900, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.944698, exp_rate<=1 : 0.978591, exp_rate<=2 : 0.995619, exp_rate<=3 : 0.998305, loss: 0.001325, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18921 s, avg_samples: 56.0, ips: 295.96415 samples/s, eta: 9:22:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:31:58] ppocr INFO: epoch: [436/500], global_step: 371000, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.945152, exp_rate<=1 : 0.978828, exp_rate<=2 : 0.995577, exp_rate<=3 : 0.998322, loss: 0.001660, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17393 s, avg_samples: 56.0, ips: 321.96174 samples/s, eta: 9:22:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:32:16] ppocr INFO: epoch: [436/500], global_step: 371100, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.944334, exp_rate<=1 : 0.978507, exp_rate<=2 : 0.995479, exp_rate<=3 : 0.998245, loss: 0.001520, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18356 s, avg_samples: 56.0, ips: 305.08409 samples/s, eta: 9:21:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:32:35] ppocr INFO: epoch: [436/500], global_step: 371200, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.943695, exp_rate<=1 : 0.978240, exp_rate<=2 : 0.995517, exp_rate<=3 : 0.998236, loss: 0.001891, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18806 s, avg_samples: 56.0, ips: 297.77944 samples/s, eta: 9:21:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:32:54] ppocr INFO: epoch: [436/500], global_step: 371300, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.943260, exp_rate<=1 : 0.978252, exp_rate<=2 : 0.995573, exp_rate<=3 : 0.998271, loss: 0.001249, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18959 s, avg_samples: 56.0, ips: 295.37417 samples/s, eta: 9:21:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:33:13] ppocr INFO: epoch: [436/500], global_step: 371400, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.943371, exp_rate<=1 : 0.978318, exp_rate<=2 : 0.995539, exp_rate<=3 : 0.998244, loss: 0.001734, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18978 s, avg_samples: 56.0, ips: 295.07961 samples/s, eta: 9:20:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:33:31] ppocr INFO: epoch: [436/500], global_step: 371500, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.943414, exp_rate<=1 : 0.978185, exp_rate<=2 : 0.995419, exp_rate<=3 : 0.998237, loss: 0.001000, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17872 s, avg_samples: 56.0, ips: 313.34275 samples/s, eta: 9:20:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:33:50] ppocr INFO: epoch: [436/500], global_step: 371600, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.943399, exp_rate<=1 : 0.978272, exp_rate<=2 : 0.995369, exp_rate<=3 : 0.998199, loss: 0.001106, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19082 s, avg_samples: 56.0, ips: 293.47561 samples/s, eta: 9:20:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:34:09] ppocr INFO: epoch: [436/500], global_step: 371700, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.942719, exp_rate<=1 : 0.978073, exp_rate<=2 : 0.995357, exp_rate<=3 : 0.998248, loss: 0.001360, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18766 s, avg_samples: 56.0, ips: 298.41527 samples/s, eta: 9:19:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:34:27] ppocr INFO: epoch: [436/500], global_step: 371800, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.942307, exp_rate<=1 : 0.977747, exp_rate<=2 : 0.995324, exp_rate<=3 : 0.998203, loss: 0.001576, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18242 s, avg_samples: 56.0, ips: 306.99212 samples/s, eta: 9:19:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:34:46] ppocr INFO: epoch: [436/500], global_step: 371900, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.942335, exp_rate<=1 : 0.977694, exp_rate<=2 : 0.995401, exp_rate<=3 : 0.998237, loss: 0.002362, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18892 s, avg_samples: 56.0, ips: 296.42764 samples/s, eta: 9:19:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:35:03] ppocr INFO: epoch: [436/500], global_step: 372000, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.942586, exp_rate<=1 : 0.977888, exp_rate<=2 : 0.995449, exp_rate<=3 : 0.998249, loss: 0.001401, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17051 s, avg_samples: 56.0, ips: 328.42657 samples/s, eta: 9:18:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:35:22] ppocr INFO: epoch: [436/500], global_step: 372100, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.942440, exp_rate<=1 : 0.977814, exp_rate<=2 : 0.995432, exp_rate<=3 : 0.998216, loss: 0.001775, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19011 s, avg_samples: 56.0, ips: 294.56490 samples/s, eta: 9:18:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:35:41] ppocr INFO: epoch: [436/500], global_step: 372200, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.942344, exp_rate<=1 : 0.977837, exp_rate<=2 : 0.995361, exp_rate<=3 : 0.998178, loss: 0.001836, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18862 s, avg_samples: 56.0, ips: 296.89700 samples/s, eta: 9:18:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:36:01] ppocr INFO: epoch: [436/500], global_step: 372300, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.941940, exp_rate<=1 : 0.977596, exp_rate<=2 : 0.995220, exp_rate<=3 : 0.998143, loss: 0.001158, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19435 s, avg_samples: 56.0, ips: 288.14619 samples/s, eta: 9:17:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:36:20] ppocr INFO: epoch: [436/500], global_step: 372400, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.941634, exp_rate<=1 : 0.977431, exp_rate<=2 : 0.995120, exp_rate<=3 : 0.998121, loss: 0.001152, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19679 s, avg_samples: 56.0, ips: 284.56032 samples/s, eta: 9:17:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:36:38] ppocr INFO: epoch: [436/500], global_step: 372500, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.941739, exp_rate<=1 : 0.977529, exp_rate<=2 : 0.995161, exp_rate<=3 : 0.998159, loss: 0.001319, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17564 s, avg_samples: 56.0, ips: 318.83695 samples/s, eta: 9:17:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:36:58] ppocr INFO: epoch: [436/500], global_step: 372600, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.941282, exp_rate<=1 : 0.977290, exp_rate<=2 : 0.995105, exp_rate<=3 : 0.998169, loss: 0.001450, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19842 s, avg_samples: 56.0, ips: 282.23036 samples/s, eta: 9:16:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:37:05] ppocr INFO: epoch: [436/500], global_step: 372640, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.941457, exp_rate<=1 : 0.977346, exp_rate<=2 : 0.995095, exp_rate<=3 : 0.998171, loss: 0.001146, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07245 s, avg_samples: 22.4, ips: 309.17249 samples/s, eta: 9:16:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:37:07] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 22:37:18] ppocr INFO: epoch: [437/500], global_step: 372700, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.946888, exp_rate<=1 : 0.978969, exp_rate<=2 : 0.994926, exp_rate<=3 : 0.998483, loss: 0.001674, avg_reader_cost: 0.02423 s, avg_batch_cost: 0.12716 s, avg_samples: 33.6, ips: 264.22908 samples/s, eta: 9:16:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:37:36] ppocr INFO: epoch: [437/500], global_step: 372800, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.943981, exp_rate<=1 : 0.976947, exp_rate<=2 : 0.994990, exp_rate<=3 : 0.998918, loss: 0.001551, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18392 s, avg_samples: 56.0, ips: 304.48417 samples/s, eta: 9:16:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:37:56] ppocr INFO: epoch: [437/500], global_step: 372900, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942698, exp_rate<=1 : 0.976286, exp_rate<=2 : 0.994636, exp_rate<=3 : 0.998656, loss: 0.001176, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19529 s, avg_samples: 56.0, ips: 286.75301 samples/s, eta: 9:16:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:38:15] ppocr INFO: epoch: [437/500], global_step: 373000, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.942979, exp_rate<=1 : 0.976807, exp_rate<=2 : 0.995091, exp_rate<=3 : 0.998624, loss: 0.000947, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19281 s, avg_samples: 56.0, ips: 290.44000 samples/s, eta: 9:15:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:38:36] ppocr INFO: epoch: [437/500], global_step: 373100, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.942806, exp_rate<=1 : 0.976811, exp_rate<=2 : 0.994847, exp_rate<=3 : 0.998414, loss: 0.001421, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20471 s, avg_samples: 56.0, ips: 273.56189 samples/s, eta: 9:15:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:38:53] ppocr INFO: epoch: [437/500], global_step: 373200, lr: 0.000100, edit distance: 0.001161, exp_rate: 0.944125, exp_rate<=1 : 0.977358, exp_rate<=2 : 0.994824, exp_rate<=3 : 0.998390, loss: 0.000630, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17665 s, avg_samples: 56.0, ips: 317.01187 samples/s, eta: 9:15:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:39:12] ppocr INFO: epoch: [437/500], global_step: 373300, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.944957, exp_rate<=1 : 0.978120, exp_rate<=2 : 0.994964, exp_rate<=3 : 0.998380, loss: 0.001363, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18973 s, avg_samples: 56.0, ips: 295.15088 samples/s, eta: 9:14:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:39:30] ppocr INFO: epoch: [437/500], global_step: 373400, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.945025, exp_rate<=1 : 0.978095, exp_rate<=2 : 0.994951, exp_rate<=3 : 0.998241, loss: 0.001192, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17446 s, avg_samples: 56.0, ips: 320.99440 samples/s, eta: 9:14:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:39:48] ppocr INFO: epoch: [437/500], global_step: 373500, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.945200, exp_rate<=1 : 0.978448, exp_rate<=2 : 0.995022, exp_rate<=3 : 0.998257, loss: 0.001322, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18601 s, avg_samples: 56.0, ips: 301.05510 samples/s, eta: 9:14:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:40:07] ppocr INFO: epoch: [437/500], global_step: 373600, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.944809, exp_rate<=1 : 0.978571, exp_rate<=2 : 0.994993, exp_rate<=3 : 0.998331, loss: 0.000737, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18661 s, avg_samples: 56.0, ips: 300.09228 samples/s, eta: 9:13:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:40:26] ppocr INFO: epoch: [437/500], global_step: 373700, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.945164, exp_rate<=1 : 0.978904, exp_rate<=2 : 0.995101, exp_rate<=3 : 0.998359, loss: 0.001181, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18510 s, avg_samples: 56.0, ips: 302.53659 samples/s, eta: 9:13:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:40:44] ppocr INFO: epoch: [437/500], global_step: 373800, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.945110, exp_rate<=1 : 0.978800, exp_rate<=2 : 0.995170, exp_rate<=3 : 0.998421, loss: 0.001443, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18018 s, avg_samples: 56.0, ips: 310.79773 samples/s, eta: 9:13:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:41:01] ppocr INFO: epoch: [437/500], global_step: 373900, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.945263, exp_rate<=1 : 0.978923, exp_rate<=2 : 0.995230, exp_rate<=3 : 0.998458, loss: 0.001700, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17881 s, avg_samples: 56.0, ips: 313.18752 samples/s, eta: 9:12:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:41:21] ppocr INFO: epoch: [437/500], global_step: 374000, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.945041, exp_rate<=1 : 0.978912, exp_rate<=2 : 0.995225, exp_rate<=3 : 0.998485, loss: 0.001094, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19027 s, avg_samples: 56.0, ips: 294.32172 samples/s, eta: 9:12:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:41:39] ppocr INFO: epoch: [437/500], global_step: 374100, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.944324, exp_rate<=1 : 0.978586, exp_rate<=2 : 0.995086, exp_rate<=3 : 0.998464, loss: 0.002175, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18812 s, avg_samples: 56.0, ips: 297.68588 samples/s, eta: 9:12:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:41:58] ppocr INFO: epoch: [437/500], global_step: 374200, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.943376, exp_rate<=1 : 0.978241, exp_rate<=2 : 0.995017, exp_rate<=3 : 0.998431, loss: 0.002397, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18759 s, avg_samples: 56.0, ips: 298.51725 samples/s, eta: 9:11:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:42:16] ppocr INFO: epoch: [437/500], global_step: 374300, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.943035, exp_rate<=1 : 0.978343, exp_rate<=2 : 0.995048, exp_rate<=3 : 0.998423, loss: 0.001230, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18104 s, avg_samples: 56.0, ips: 309.32745 samples/s, eta: 9:11:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:42:36] ppocr INFO: epoch: [437/500], global_step: 374400, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.942628, exp_rate<=1 : 0.978223, exp_rate<=2 : 0.995004, exp_rate<=3 : 0.998328, loss: 0.001396, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20044 s, avg_samples: 56.0, ips: 279.38895 samples/s, eta: 9:11:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:42:57] ppocr INFO: epoch: [437/500], global_step: 374500, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.941675, exp_rate<=1 : 0.977800, exp_rate<=2 : 0.994845, exp_rate<=3 : 0.998217, loss: 0.001415, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20849 s, avg_samples: 56.0, ips: 268.59977 samples/s, eta: 9:10:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:43:15] ppocr INFO: epoch: [437/500], global_step: 374600, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.941622, exp_rate<=1 : 0.977768, exp_rate<=2 : 0.994872, exp_rate<=3 : 0.998225, loss: 0.001848, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17835 s, avg_samples: 56.0, ips: 313.99103 samples/s, eta: 9:10:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:43:34] ppocr INFO: epoch: [437/500], global_step: 374700, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.941307, exp_rate<=1 : 0.977770, exp_rate<=2 : 0.994879, exp_rate<=3 : 0.998215, loss: 0.001538, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18690 s, avg_samples: 56.0, ips: 299.62329 samples/s, eta: 9:10:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:43:54] ppocr INFO: epoch: [437/500], global_step: 374800, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.941083, exp_rate<=1 : 0.977536, exp_rate<=2 : 0.994835, exp_rate<=3 : 0.998211, loss: 0.001162, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19892 s, avg_samples: 56.0, ips: 281.51596 samples/s, eta: 9:09:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:44:12] ppocr INFO: epoch: [437/500], global_step: 374900, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.941275, exp_rate<=1 : 0.977613, exp_rate<=2 : 0.994930, exp_rate<=3 : 0.998223, loss: 0.001519, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17995 s, avg_samples: 56.0, ips: 311.20563 samples/s, eta: 9:09:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:44:31] ppocr INFO: epoch: [437/500], global_step: 375000, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.940848, exp_rate<=1 : 0.977555, exp_rate<=2 : 0.994933, exp_rate<=3 : 0.998245, loss: 0.001869, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19368 s, avg_samples: 56.0, ips: 289.13704 samples/s, eta: 9:09:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:44:51] ppocr INFO: epoch: [437/500], global_step: 375100, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.940446, exp_rate<=1 : 0.977415, exp_rate<=2 : 0.994935, exp_rate<=3 : 0.998237, loss: 0.001396, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19570 s, avg_samples: 56.0, ips: 286.14802 samples/s, eta: 9:08:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:45:08] ppocr INFO: epoch: [437/500], global_step: 375200, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.940656, exp_rate<=1 : 0.977463, exp_rate<=2 : 0.995007, exp_rate<=3 : 0.998243, loss: 0.002112, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17572 s, avg_samples: 56.0, ips: 318.69167 samples/s, eta: 9:08:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:45:28] ppocr INFO: epoch: [437/500], global_step: 375300, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.940080, exp_rate<=1 : 0.977286, exp_rate<=2 : 0.994890, exp_rate<=3 : 0.998197, loss: 0.001564, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19825 s, avg_samples: 56.0, ips: 282.47173 samples/s, eta: 9:08:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:45:44] ppocr INFO: epoch: [437/500], global_step: 375380, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.939816, exp_rate<=1 : 0.977245, exp_rate<=2 : 0.994873, exp_rate<=3 : 0.998176, loss: 0.001784, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15731 s, avg_samples: 44.8, ips: 284.78259 samples/s, eta: 9:08:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:45:46] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 22:45:50] ppocr INFO: epoch: [438/500], global_step: 375400, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.932711, exp_rate<=1 : 0.965909, exp_rate<=2 : 0.995707, exp_rate<=3 : 0.997138, loss: 0.001250, avg_reader_cost: 0.02535 s, avg_batch_cost: 0.06564 s, avg_samples: 11.2, ips: 170.63596 samples/s, eta: 9:08:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:46:09] ppocr INFO: epoch: [438/500], global_step: 375500, lr: 0.000100, edit distance: 0.001029, exp_rate: 0.953241, exp_rate<=1 : 0.982624, exp_rate<=2 : 0.996477, exp_rate<=3 : 0.998532, loss: 0.001168, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18270 s, avg_samples: 56.0, ips: 306.50756 samples/s, eta: 9:07:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:46:27] ppocr INFO: epoch: [438/500], global_step: 375600, lr: 0.000100, edit distance: 0.001117, exp_rate: 0.951264, exp_rate<=1 : 0.981165, exp_rate<=2 : 0.995940, exp_rate<=3 : 0.998728, loss: 0.002060, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18070 s, avg_samples: 56.0, ips: 309.90261 samples/s, eta: 9:07:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:46:46] ppocr INFO: epoch: [438/500], global_step: 375700, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.946945, exp_rate<=1 : 0.980245, exp_rate<=2 : 0.995899, exp_rate<=3 : 0.998589, loss: 0.001188, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18961 s, avg_samples: 56.0, ips: 295.34823 samples/s, eta: 9:07:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:47:04] ppocr INFO: epoch: [438/500], global_step: 375800, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.947405, exp_rate<=1 : 0.979796, exp_rate<=2 : 0.995739, exp_rate<=3 : 0.998351, loss: 0.001310, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18609 s, avg_samples: 56.0, ips: 300.93059 samples/s, eta: 9:06:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:47:24] ppocr INFO: epoch: [438/500], global_step: 375900, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.945543, exp_rate<=1 : 0.978547, exp_rate<=2 : 0.995112, exp_rate<=3 : 0.997698, loss: 0.001408, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19219 s, avg_samples: 56.0, ips: 291.37434 samples/s, eta: 9:06:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:47:42] ppocr INFO: epoch: [438/500], global_step: 376000, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.946269, exp_rate<=1 : 0.978928, exp_rate<=2 : 0.995247, exp_rate<=3 : 0.997779, loss: 0.001419, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17854 s, avg_samples: 56.0, ips: 313.66280 samples/s, eta: 9:06:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:48:00] ppocr INFO: epoch: [438/500], global_step: 376100, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.946454, exp_rate<=1 : 0.978986, exp_rate<=2 : 0.995314, exp_rate<=3 : 0.997817, loss: 0.000786, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18277 s, avg_samples: 56.0, ips: 306.39916 samples/s, eta: 9:05:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:48:19] ppocr INFO: epoch: [438/500], global_step: 376200, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.945955, exp_rate<=1 : 0.978820, exp_rate<=2 : 0.995287, exp_rate<=3 : 0.997909, loss: 0.001037, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19634 s, avg_samples: 56.0, ips: 285.21578 samples/s, eta: 9:05:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:48:38] ppocr INFO: epoch: [438/500], global_step: 376300, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.945743, exp_rate<=1 : 0.978939, exp_rate<=2 : 0.995501, exp_rate<=3 : 0.998023, loss: 0.001243, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18643 s, avg_samples: 56.0, ips: 300.38415 samples/s, eta: 9:05:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:48:55] ppocr INFO: epoch: [438/500], global_step: 376400, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.946127, exp_rate<=1 : 0.979221, exp_rate<=2 : 0.995436, exp_rate<=3 : 0.998064, loss: 0.001042, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16956 s, avg_samples: 56.0, ips: 330.27035 samples/s, eta: 9:04:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:49:14] ppocr INFO: epoch: [438/500], global_step: 376500, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.946099, exp_rate<=1 : 0.979309, exp_rate<=2 : 0.995528, exp_rate<=3 : 0.998139, loss: 0.001636, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18498 s, avg_samples: 56.0, ips: 302.74038 samples/s, eta: 9:04:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:49:32] ppocr INFO: epoch: [438/500], global_step: 376600, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.945781, exp_rate<=1 : 0.979218, exp_rate<=2 : 0.995501, exp_rate<=3 : 0.998145, loss: 0.001256, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18488 s, avg_samples: 56.0, ips: 302.90003 samples/s, eta: 9:04:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:49:50] ppocr INFO: epoch: [438/500], global_step: 376700, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.946101, exp_rate<=1 : 0.979348, exp_rate<=2 : 0.995651, exp_rate<=3 : 0.998222, loss: 0.001301, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17997 s, avg_samples: 56.0, ips: 311.15876 samples/s, eta: 9:03:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:50:08] ppocr INFO: epoch: [438/500], global_step: 376800, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.945982, exp_rate<=1 : 0.979244, exp_rate<=2 : 0.995667, exp_rate<=3 : 0.998285, loss: 0.001074, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18293 s, avg_samples: 56.0, ips: 306.12113 samples/s, eta: 9:03:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:50:28] ppocr INFO: epoch: [438/500], global_step: 376900, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.945551, exp_rate<=1 : 0.979090, exp_rate<=2 : 0.995642, exp_rate<=3 : 0.998344, loss: 0.001217, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19405 s, avg_samples: 56.0, ips: 288.58717 samples/s, eta: 9:03:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:50:47] ppocr INFO: epoch: [438/500], global_step: 377000, lr: 0.000100, edit distance: 0.001214, exp_rate: 0.945169, exp_rate<=1 : 0.978959, exp_rate<=2 : 0.995526, exp_rate<=3 : 0.998331, loss: 0.002054, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19255 s, avg_samples: 56.0, ips: 290.83628 samples/s, eta: 9:02:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:51:08] ppocr INFO: epoch: [438/500], global_step: 377100, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.944160, exp_rate<=1 : 0.978493, exp_rate<=2 : 0.995491, exp_rate<=3 : 0.998334, loss: 0.002088, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20524 s, avg_samples: 56.0, ips: 272.84600 samples/s, eta: 9:02:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:51:26] ppocr INFO: epoch: [438/500], global_step: 377200, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.943824, exp_rate<=1 : 0.978474, exp_rate<=2 : 0.995433, exp_rate<=3 : 0.998313, loss: 0.001383, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18056 s, avg_samples: 56.0, ips: 310.14170 samples/s, eta: 9:02:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:51:44] ppocr INFO: epoch: [438/500], global_step: 377300, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.943730, exp_rate<=1 : 0.978556, exp_rate<=2 : 0.995474, exp_rate<=3 : 0.998363, loss: 0.001449, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18398 s, avg_samples: 56.0, ips: 304.38303 samples/s, eta: 9:01:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:52:03] ppocr INFO: epoch: [438/500], global_step: 377400, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.943652, exp_rate<=1 : 0.978358, exp_rate<=2 : 0.995410, exp_rate<=3 : 0.998364, loss: 0.001512, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18480 s, avg_samples: 56.0, ips: 303.03732 samples/s, eta: 9:01:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:52:22] ppocr INFO: epoch: [438/500], global_step: 377500, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.943569, exp_rate<=1 : 0.978369, exp_rate<=2 : 0.995453, exp_rate<=3 : 0.998395, loss: 0.001880, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19314 s, avg_samples: 56.0, ips: 289.94699 samples/s, eta: 9:01:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:52:42] ppocr INFO: epoch: [438/500], global_step: 377600, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.942704, exp_rate<=1 : 0.977932, exp_rate<=2 : 0.995325, exp_rate<=3 : 0.998365, loss: 0.001526, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20046 s, avg_samples: 56.0, ips: 279.36133 samples/s, eta: 9:01:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:53:01] ppocr INFO: epoch: [438/500], global_step: 377700, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.942518, exp_rate<=1 : 0.977866, exp_rate<=2 : 0.995305, exp_rate<=3 : 0.998362, loss: 0.001534, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19156 s, avg_samples: 56.0, ips: 292.33103 samples/s, eta: 9:00:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:53:21] ppocr INFO: epoch: [438/500], global_step: 377800, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.942036, exp_rate<=1 : 0.977608, exp_rate<=2 : 0.995241, exp_rate<=3 : 0.998337, loss: 0.001589, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19650 s, avg_samples: 56.0, ips: 284.98936 samples/s, eta: 9:00:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:53:39] ppocr INFO: epoch: [438/500], global_step: 377900, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.942230, exp_rate<=1 : 0.977623, exp_rate<=2 : 0.995158, exp_rate<=3 : 0.998286, loss: 0.001583, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17732 s, avg_samples: 56.0, ips: 315.81297 samples/s, eta: 9:00:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:53:57] ppocr INFO: epoch: [438/500], global_step: 378000, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.942113, exp_rate<=1 : 0.977561, exp_rate<=2 : 0.995179, exp_rate<=3 : 0.998308, loss: 0.001583, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18566 s, avg_samples: 56.0, ips: 301.61977 samples/s, eta: 8:59:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:54:15] ppocr INFO: epoch: [438/500], global_step: 378100, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.941635, exp_rate<=1 : 0.977368, exp_rate<=2 : 0.995047, exp_rate<=3 : 0.998259, loss: 0.001743, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17904 s, avg_samples: 56.0, ips: 312.77058 samples/s, eta: 8:59:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:54:19] ppocr INFO: epoch: [438/500], global_step: 378120, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.941570, exp_rate<=1 : 0.977379, exp_rate<=2 : 0.995042, exp_rate<=3 : 0.998251, loss: 0.001300, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03678 s, avg_samples: 11.2, ips: 304.53735 samples/s, eta: 8:59:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:54:21] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 22:54:36] ppocr INFO: epoch: [439/500], global_step: 378200, lr: 0.000100, edit distance: 0.000818, exp_rate: 0.953987, exp_rate<=1 : 0.982940, exp_rate<=2 : 0.997993, exp_rate<=3 : 0.999374, loss: 0.001144, avg_reader_cost: 0.02516 s, avg_batch_cost: 0.17175 s, avg_samples: 44.8, ips: 260.84773 samples/s, eta: 8:59:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:54:55] ppocr INFO: epoch: [439/500], global_step: 378300, lr: 0.000100, edit distance: 0.000914, exp_rate: 0.952708, exp_rate<=1 : 0.982244, exp_rate<=2 : 0.996586, exp_rate<=3 : 0.998806, loss: 0.001501, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18504 s, avg_samples: 56.0, ips: 302.63785 samples/s, eta: 8:58:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:55:14] ppocr INFO: epoch: [439/500], global_step: 378400, lr: 0.000100, edit distance: 0.000898, exp_rate: 0.952849, exp_rate<=1 : 0.982076, exp_rate<=2 : 0.995808, exp_rate<=3 : 0.998326, loss: 0.000981, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19307 s, avg_samples: 56.0, ips: 290.04902 samples/s, eta: 8:58:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:55:32] ppocr INFO: epoch: [439/500], global_step: 378500, lr: 0.000100, edit distance: 0.000943, exp_rate: 0.952679, exp_rate<=1 : 0.982481, exp_rate<=2 : 0.996182, exp_rate<=3 : 0.998699, loss: 0.001171, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18079 s, avg_samples: 56.0, ips: 309.74471 samples/s, eta: 8:58:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:55:49] ppocr INFO: epoch: [439/500], global_step: 378600, lr: 0.000100, edit distance: 0.000944, exp_rate: 0.953987, exp_rate<=1 : 0.982784, exp_rate<=2 : 0.996122, exp_rate<=3 : 0.998650, loss: 0.000995, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17182 s, avg_samples: 56.0, ips: 325.92736 samples/s, eta: 8:57:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:56:08] ppocr INFO: epoch: [439/500], global_step: 378700, lr: 0.000100, edit distance: 0.000969, exp_rate: 0.952733, exp_rate<=1 : 0.982316, exp_rate<=2 : 0.996011, exp_rate<=3 : 0.998674, loss: 0.001193, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18610 s, avg_samples: 56.0, ips: 300.90920 samples/s, eta: 8:57:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:56:26] ppocr INFO: epoch: [439/500], global_step: 378800, lr: 0.000100, edit distance: 0.001025, exp_rate: 0.951209, exp_rate<=1 : 0.981529, exp_rate<=2 : 0.995739, exp_rate<=3 : 0.998642, loss: 0.001256, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18347 s, avg_samples: 56.0, ips: 305.22218 samples/s, eta: 8:57:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:56:46] ppocr INFO: epoch: [439/500], global_step: 378900, lr: 0.000100, edit distance: 0.001047, exp_rate: 0.950300, exp_rate<=1 : 0.981114, exp_rate<=2 : 0.995634, exp_rate<=3 : 0.998586, loss: 0.001249, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20094 s, avg_samples: 56.0, ips: 278.68525 samples/s, eta: 8:56:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:57:06] ppocr INFO: epoch: [439/500], global_step: 379000, lr: 0.000100, edit distance: 0.001062, exp_rate: 0.949372, exp_rate<=1 : 0.980938, exp_rate<=2 : 0.995742, exp_rate<=3 : 0.998590, loss: 0.001650, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19694 s, avg_samples: 56.0, ips: 284.35012 samples/s, eta: 8:56:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:57:24] ppocr INFO: epoch: [439/500], global_step: 379100, lr: 0.000100, edit distance: 0.001079, exp_rate: 0.948673, exp_rate<=1 : 0.980698, exp_rate<=2 : 0.995789, exp_rate<=3 : 0.998628, loss: 0.001857, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18326 s, avg_samples: 56.0, ips: 305.57840 samples/s, eta: 8:56:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:57:44] ppocr INFO: epoch: [439/500], global_step: 379200, lr: 0.000100, edit distance: 0.001106, exp_rate: 0.947028, exp_rate<=1 : 0.979881, exp_rate<=2 : 0.995583, exp_rate<=3 : 0.998445, loss: 0.001274, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19966 s, avg_samples: 56.0, ips: 280.47248 samples/s, eta: 8:55:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:58:03] ppocr INFO: epoch: [439/500], global_step: 379300, lr: 0.000100, edit distance: 0.001098, exp_rate: 0.947147, exp_rate<=1 : 0.980105, exp_rate<=2 : 0.995695, exp_rate<=3 : 0.998516, loss: 0.001188, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18434 s, avg_samples: 56.0, ips: 303.79295 samples/s, eta: 8:55:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:58:22] ppocr INFO: epoch: [439/500], global_step: 379400, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.946204, exp_rate<=1 : 0.979655, exp_rate<=2 : 0.995615, exp_rate<=3 : 0.998454, loss: 0.001244, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18854 s, avg_samples: 56.0, ips: 297.02095 samples/s, eta: 8:55:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:58:41] ppocr INFO: epoch: [439/500], global_step: 379500, lr: 0.000100, edit distance: 0.001134, exp_rate: 0.945775, exp_rate<=1 : 0.979687, exp_rate<=2 : 0.995604, exp_rate<=3 : 0.998452, loss: 0.001381, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19445 s, avg_samples: 56.0, ips: 287.98777 samples/s, eta: 8:54:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:58:59] ppocr INFO: epoch: [439/500], global_step: 379600, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.945182, exp_rate<=1 : 0.979321, exp_rate<=2 : 0.995566, exp_rate<=3 : 0.998454, loss: 0.001876, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17652 s, avg_samples: 56.0, ips: 317.25299 samples/s, eta: 8:54:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:59:16] ppocr INFO: epoch: [439/500], global_step: 379700, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.944826, exp_rate<=1 : 0.979242, exp_rate<=2 : 0.995618, exp_rate<=3 : 0.998469, loss: 0.002071, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17512 s, avg_samples: 56.0, ips: 319.78887 samples/s, eta: 8:54:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:59:34] ppocr INFO: epoch: [439/500], global_step: 379800, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.944449, exp_rate<=1 : 0.979009, exp_rate<=2 : 0.995517, exp_rate<=3 : 0.998487, loss: 0.001163, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17994 s, avg_samples: 56.0, ips: 311.21635 samples/s, eta: 8:53:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 22:59:54] ppocr INFO: epoch: [439/500], global_step: 379900, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.943913, exp_rate<=1 : 0.978789, exp_rate<=2 : 0.995483, exp_rate<=3 : 0.998463, loss: 0.001991, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19474 s, avg_samples: 56.0, ips: 287.55963 samples/s, eta: 8:53:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:00:12] ppocr INFO: epoch: [439/500], global_step: 380000, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.943404, exp_rate<=1 : 0.978626, exp_rate<=2 : 0.995501, exp_rate<=3 : 0.998434, loss: 0.001149, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18295 s, avg_samples: 56.0, ips: 306.09837 samples/s, eta: 8:53:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:00:32] ppocr INFO: epoch: [439/500], global_step: 380100, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.943020, exp_rate<=1 : 0.978389, exp_rate<=2 : 0.995452, exp_rate<=3 : 0.998378, loss: 0.001386, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19325 s, avg_samples: 56.0, ips: 289.78522 samples/s, eta: 8:53:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:00:51] ppocr INFO: epoch: [439/500], global_step: 380200, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.943117, exp_rate<=1 : 0.978376, exp_rate<=2 : 0.995402, exp_rate<=3 : 0.998370, loss: 0.000979, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19440 s, avg_samples: 56.0, ips: 288.07185 samples/s, eta: 8:52:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:01:12] ppocr INFO: epoch: [439/500], global_step: 380300, lr: 0.000100, edit distance: 0.001214, exp_rate: 0.942968, exp_rate<=1 : 0.978243, exp_rate<=2 : 0.995409, exp_rate<=3 : 0.998373, loss: 0.001312, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20512 s, avg_samples: 56.0, ips: 273.00879 samples/s, eta: 8:52:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:01:30] ppocr INFO: epoch: [439/500], global_step: 380400, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.942731, exp_rate<=1 : 0.978201, exp_rate<=2 : 0.995396, exp_rate<=3 : 0.998412, loss: 0.001544, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18267 s, avg_samples: 56.0, ips: 306.56546 samples/s, eta: 8:52:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:01:47] ppocr INFO: epoch: [439/500], global_step: 380500, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.942740, exp_rate<=1 : 0.978220, exp_rate<=2 : 0.995406, exp_rate<=3 : 0.998408, loss: 0.002267, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17598 s, avg_samples: 56.0, ips: 318.21133 samples/s, eta: 8:51:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:02:07] ppocr INFO: epoch: [439/500], global_step: 380600, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.942423, exp_rate<=1 : 0.978014, exp_rate<=2 : 0.995330, exp_rate<=3 : 0.998390, loss: 0.001577, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19422 s, avg_samples: 56.0, ips: 288.32609 samples/s, eta: 8:51:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:02:26] ppocr INFO: epoch: [439/500], global_step: 380700, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.941812, exp_rate<=1 : 0.977784, exp_rate<=2 : 0.995301, exp_rate<=3 : 0.998376, loss: 0.001630, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18691 s, avg_samples: 56.0, ips: 299.60411 samples/s, eta: 8:51:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:02:45] ppocr INFO: epoch: [439/500], global_step: 380800, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.941594, exp_rate<=1 : 0.977691, exp_rate<=2 : 0.995288, exp_rate<=3 : 0.998370, loss: 0.001632, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19133 s, avg_samples: 56.0, ips: 292.69426 samples/s, eta: 8:50:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:02:55] ppocr INFO: epoch: [439/500], global_step: 380860, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.941675, exp_rate<=1 : 0.977773, exp_rate<=2 : 0.995309, exp_rate<=3 : 0.998369, loss: 0.001468, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.10392 s, avg_samples: 33.6, ips: 323.31668 samples/s, eta: 8:50:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:02:58] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 23:03:05] ppocr INFO: epoch: [440/500], global_step: 380900, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.944727, exp_rate<=1 : 0.973052, exp_rate<=2 : 0.996486, exp_rate<=3 : 0.997657, loss: 0.001409, avg_reader_cost: 0.03137 s, avg_batch_cost: 0.10300 s, avg_samples: 22.4, ips: 217.48378 samples/s, eta: 8:50:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:03:24] ppocr INFO: epoch: [440/500], global_step: 381000, lr: 0.000100, edit distance: 0.001135, exp_rate: 0.948745, exp_rate<=1 : 0.979640, exp_rate<=2 : 0.997058, exp_rate<=3 : 0.998787, loss: 0.001084, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18374 s, avg_samples: 56.0, ips: 304.77612 samples/s, eta: 8:50:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:03:43] ppocr INFO: epoch: [440/500], global_step: 381100, lr: 0.000100, edit distance: 0.001131, exp_rate: 0.949167, exp_rate<=1 : 0.980752, exp_rate<=2 : 0.996952, exp_rate<=3 : 0.999066, loss: 0.001446, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19003 s, avg_samples: 56.0, ips: 294.69606 samples/s, eta: 8:49:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:04:01] ppocr INFO: epoch: [440/500], global_step: 381200, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.948196, exp_rate<=1 : 0.979350, exp_rate<=2 : 0.996510, exp_rate<=3 : 0.998879, loss: 0.001648, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18105 s, avg_samples: 56.0, ips: 309.30075 samples/s, eta: 8:49:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:04:19] ppocr INFO: epoch: [440/500], global_step: 381300, lr: 0.000100, edit distance: 0.001160, exp_rate: 0.948689, exp_rate<=1 : 0.980120, exp_rate<=2 : 0.996665, exp_rate<=3 : 0.999037, loss: 0.001763, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18006 s, avg_samples: 56.0, ips: 311.01058 samples/s, eta: 8:49:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:04:37] ppocr INFO: epoch: [440/500], global_step: 381400, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.947075, exp_rate<=1 : 0.979480, exp_rate<=2 : 0.996278, exp_rate<=3 : 0.998925, loss: 0.001540, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17681 s, avg_samples: 56.0, ips: 316.72629 samples/s, eta: 8:48:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:04:57] ppocr INFO: epoch: [440/500], global_step: 381500, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.945315, exp_rate<=1 : 0.979250, exp_rate<=2 : 0.995865, exp_rate<=3 : 0.998822, loss: 0.001073, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20654 s, avg_samples: 56.0, ips: 271.14004 samples/s, eta: 8:48:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:05:15] ppocr INFO: epoch: [440/500], global_step: 381600, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.945108, exp_rate<=1 : 0.979522, exp_rate<=2 : 0.995949, exp_rate<=3 : 0.998812, loss: 0.001257, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17523 s, avg_samples: 56.0, ips: 319.58824 samples/s, eta: 8:48:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:05:33] ppocr INFO: epoch: [440/500], global_step: 381700, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.945686, exp_rate<=1 : 0.979710, exp_rate<=2 : 0.996065, exp_rate<=3 : 0.998796, loss: 0.000877, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18285 s, avg_samples: 56.0, ips: 306.25387 samples/s, eta: 8:47:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:05:52] ppocr INFO: epoch: [440/500], global_step: 381800, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.945442, exp_rate<=1 : 0.979623, exp_rate<=2 : 0.995929, exp_rate<=3 : 0.998745, loss: 0.001012, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18840 s, avg_samples: 56.0, ips: 297.23204 samples/s, eta: 8:47:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:06:12] ppocr INFO: epoch: [440/500], global_step: 381900, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.944909, exp_rate<=1 : 0.979320, exp_rate<=2 : 0.995925, exp_rate<=3 : 0.998693, loss: 0.001078, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19503 s, avg_samples: 56.0, ips: 287.13926 samples/s, eta: 8:47:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:06:30] ppocr INFO: epoch: [440/500], global_step: 382000, lr: 0.000100, edit distance: 0.001180, exp_rate: 0.944636, exp_rate<=1 : 0.979094, exp_rate<=2 : 0.995854, exp_rate<=3 : 0.998673, loss: 0.001818, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18533 s, avg_samples: 56.0, ips: 302.15663 samples/s, eta: 8:46:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:06:50] ppocr INFO: epoch: [440/500], global_step: 382100, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.944383, exp_rate<=1 : 0.978917, exp_rate<=2 : 0.995708, exp_rate<=3 : 0.998625, loss: 0.001712, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20245 s, avg_samples: 56.0, ips: 276.60927 samples/s, eta: 8:46:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:07:09] ppocr INFO: epoch: [440/500], global_step: 382200, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.943578, exp_rate<=1 : 0.978574, exp_rate<=2 : 0.995616, exp_rate<=3 : 0.998594, loss: 0.001721, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18371 s, avg_samples: 56.0, ips: 304.82890 samples/s, eta: 8:46:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:07:27] ppocr INFO: epoch: [440/500], global_step: 382300, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.943369, exp_rate<=1 : 0.978520, exp_rate<=2 : 0.995630, exp_rate<=3 : 0.998610, loss: 0.001562, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17963 s, avg_samples: 56.0, ips: 311.75768 samples/s, eta: 8:46:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:07:45] ppocr INFO: epoch: [440/500], global_step: 382400, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.943238, exp_rate<=1 : 0.978464, exp_rate<=2 : 0.995534, exp_rate<=3 : 0.998566, loss: 0.001398, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17822 s, avg_samples: 56.0, ips: 314.21527 samples/s, eta: 8:45:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:08:03] ppocr INFO: epoch: [440/500], global_step: 382500, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.943351, exp_rate<=1 : 0.978581, exp_rate<=2 : 0.995552, exp_rate<=3 : 0.998541, loss: 0.001568, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17915 s, avg_samples: 56.0, ips: 312.58679 samples/s, eta: 8:45:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:08:21] ppocr INFO: epoch: [440/500], global_step: 382600, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.942830, exp_rate<=1 : 0.978259, exp_rate<=2 : 0.995458, exp_rate<=3 : 0.998491, loss: 0.001560, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18661 s, avg_samples: 56.0, ips: 300.08629 samples/s, eta: 8:45:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:08:39] ppocr INFO: epoch: [440/500], global_step: 382700, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.943033, exp_rate<=1 : 0.978211, exp_rate<=2 : 0.995416, exp_rate<=3 : 0.998467, loss: 0.001161, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17531 s, avg_samples: 56.0, ips: 319.42722 samples/s, eta: 8:44:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:08:59] ppocr INFO: epoch: [440/500], global_step: 382800, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.942659, exp_rate<=1 : 0.978014, exp_rate<=2 : 0.995273, exp_rate<=3 : 0.998446, loss: 0.001797, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20047 s, avg_samples: 56.0, ips: 279.34024 samples/s, eta: 8:44:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:09:19] ppocr INFO: epoch: [440/500], global_step: 382900, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.941805, exp_rate<=1 : 0.977578, exp_rate<=2 : 0.995202, exp_rate<=3 : 0.998421, loss: 0.001661, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20450 s, avg_samples: 56.0, ips: 273.83309 samples/s, eta: 8:44:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:09:38] ppocr INFO: epoch: [440/500], global_step: 383000, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.941422, exp_rate<=1 : 0.977420, exp_rate<=2 : 0.995103, exp_rate<=3 : 0.998391, loss: 0.001007, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18895 s, avg_samples: 56.0, ips: 296.38235 samples/s, eta: 8:43:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:09:57] ppocr INFO: epoch: [440/500], global_step: 383100, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.941147, exp_rate<=1 : 0.977365, exp_rate<=2 : 0.995086, exp_rate<=3 : 0.998387, loss: 0.001963, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18938 s, avg_samples: 56.0, ips: 295.70201 samples/s, eta: 8:43:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:10:15] ppocr INFO: epoch: [440/500], global_step: 383200, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.940902, exp_rate<=1 : 0.977328, exp_rate<=2 : 0.995059, exp_rate<=3 : 0.998370, loss: 0.001669, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18147 s, avg_samples: 56.0, ips: 308.59822 samples/s, eta: 8:43:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:10:35] ppocr INFO: epoch: [440/500], global_step: 383300, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.940680, exp_rate<=1 : 0.977105, exp_rate<=2 : 0.994958, exp_rate<=3 : 0.998347, loss: 0.001102, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19484 s, avg_samples: 56.0, ips: 287.42201 samples/s, eta: 8:42:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:10:54] ppocr INFO: epoch: [440/500], global_step: 383400, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.940657, exp_rate<=1 : 0.977218, exp_rate<=2 : 0.994977, exp_rate<=3 : 0.998346, loss: 0.001111, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18737 s, avg_samples: 56.0, ips: 298.88012 samples/s, eta: 8:42:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:11:12] ppocr INFO: epoch: [440/500], global_step: 383500, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.940388, exp_rate<=1 : 0.977105, exp_rate<=2 : 0.994943, exp_rate<=3 : 0.998316, loss: 0.001408, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18823 s, avg_samples: 56.0, ips: 297.50210 samples/s, eta: 8:42:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:11:32] ppocr INFO: epoch: [440/500], global_step: 383600, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.940069, exp_rate<=1 : 0.976890, exp_rate<=2 : 0.994855, exp_rate<=3 : 0.998309, loss: 0.001790, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19613 s, avg_samples: 56.0, ips: 285.52703 samples/s, eta: 8:41:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:11:34] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 23:11:35] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_440 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 23:11:54] ppocr INFO: epoch: [441/500], global_step: 383700, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.950866, exp_rate<=1 : 0.979634, exp_rate<=2 : 0.995204, exp_rate<=3 : 0.998677, loss: 0.000956, avg_reader_cost: 0.03481 s, avg_batch_cost: 0.22203 s, avg_samples: 56.0, ips: 252.21795 samples/s, eta: 8:41:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:12:13] ppocr INFO: epoch: [441/500], global_step: 383800, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.947837, exp_rate<=1 : 0.979988, exp_rate<=2 : 0.995338, exp_rate<=3 : 0.999000, loss: 0.001403, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18746 s, avg_samples: 56.0, ips: 298.72740 samples/s, eta: 8:41:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:12:32] ppocr INFO: epoch: [441/500], global_step: 383900, lr: 0.000100, edit distance: 0.001101, exp_rate: 0.948702, exp_rate<=1 : 0.980657, exp_rate<=2 : 0.995956, exp_rate<=3 : 0.999052, loss: 0.001137, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18633 s, avg_samples: 56.0, ips: 300.53724 samples/s, eta: 8:40:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:12:52] ppocr INFO: epoch: [441/500], global_step: 384000, lr: 0.000100, edit distance: 0.001118, exp_rate: 0.947658, exp_rate<=1 : 0.980458, exp_rate<=2 : 0.995715, exp_rate<=3 : 0.998932, loss: 0.001440, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19897 s, avg_samples: 56.0, ips: 281.45085 samples/s, eta: 8:40:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:13:10] ppocr INFO: epoch: [441/500], global_step: 384100, lr: 0.000100, edit distance: 0.001121, exp_rate: 0.947654, exp_rate<=1 : 0.980800, exp_rate<=2 : 0.995746, exp_rate<=3 : 0.998689, loss: 0.001234, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18409 s, avg_samples: 56.0, ips: 304.20583 samples/s, eta: 8:40:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:13:30] ppocr INFO: epoch: [441/500], global_step: 384200, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.945604, exp_rate<=1 : 0.979194, exp_rate<=2 : 0.994952, exp_rate<=3 : 0.998154, loss: 0.001326, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19757 s, avg_samples: 56.0, ips: 283.44464 samples/s, eta: 8:39:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:13:50] ppocr INFO: epoch: [441/500], global_step: 384300, lr: 0.000100, edit distance: 0.001177, exp_rate: 0.943909, exp_rate<=1 : 0.978623, exp_rate<=2 : 0.994465, exp_rate<=3 : 0.997929, loss: 0.001247, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20510 s, avg_samples: 56.0, ips: 273.04169 samples/s, eta: 8:39:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:14:09] ppocr INFO: epoch: [441/500], global_step: 384400, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.944516, exp_rate<=1 : 0.979059, exp_rate<=2 : 0.994480, exp_rate<=3 : 0.997884, loss: 0.001356, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18232 s, avg_samples: 56.0, ips: 307.14941 samples/s, eta: 8:39:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:14:28] ppocr INFO: epoch: [441/500], global_step: 384500, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.944332, exp_rate<=1 : 0.978921, exp_rate<=2 : 0.994552, exp_rate<=3 : 0.997814, loss: 0.001425, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19159 s, avg_samples: 56.0, ips: 292.28893 samples/s, eta: 8:39:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:14:47] ppocr INFO: epoch: [441/500], global_step: 384600, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.944390, exp_rate<=1 : 0.978754, exp_rate<=2 : 0.994591, exp_rate<=3 : 0.997819, loss: 0.001176, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19416 s, avg_samples: 56.0, ips: 288.41713 samples/s, eta: 8:38:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:15:06] ppocr INFO: epoch: [441/500], global_step: 384700, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.943621, exp_rate<=1 : 0.978333, exp_rate<=2 : 0.994622, exp_rate<=3 : 0.997798, loss: 0.001425, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18685 s, avg_samples: 56.0, ips: 299.70281 samples/s, eta: 8:38:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:15:25] ppocr INFO: epoch: [441/500], global_step: 384800, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.942908, exp_rate<=1 : 0.978125, exp_rate<=2 : 0.994499, exp_rate<=3 : 0.997669, loss: 0.001337, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19300 s, avg_samples: 56.0, ips: 290.15280 samples/s, eta: 8:38:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:15:43] ppocr INFO: epoch: [441/500], global_step: 384900, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.942390, exp_rate<=1 : 0.977860, exp_rate<=2 : 0.994548, exp_rate<=3 : 0.997731, loss: 0.001440, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17331 s, avg_samples: 56.0, ips: 323.11864 samples/s, eta: 8:37:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:16:02] ppocr INFO: epoch: [441/500], global_step: 385000, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.942046, exp_rate<=1 : 0.977794, exp_rate<=2 : 0.994538, exp_rate<=3 : 0.997778, loss: 0.001307, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18827 s, avg_samples: 56.0, ips: 297.44376 samples/s, eta: 8:37:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:16:20] ppocr INFO: epoch: [441/500], global_step: 385100, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.941522, exp_rate<=1 : 0.977638, exp_rate<=2 : 0.994558, exp_rate<=3 : 0.997810, loss: 0.002108, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18774 s, avg_samples: 56.0, ips: 298.27914 samples/s, eta: 8:37:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:16:39] ppocr INFO: epoch: [441/500], global_step: 385200, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.941536, exp_rate<=1 : 0.977586, exp_rate<=2 : 0.994554, exp_rate<=3 : 0.997823, loss: 0.001220, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18954 s, avg_samples: 56.0, ips: 295.44680 samples/s, eta: 8:36:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:16:59] ppocr INFO: epoch: [441/500], global_step: 385300, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.941502, exp_rate<=1 : 0.977371, exp_rate<=2 : 0.994581, exp_rate<=3 : 0.997870, loss: 0.001367, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19683 s, avg_samples: 56.0, ips: 284.51132 samples/s, eta: 8:36:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:17:17] ppocr INFO: epoch: [441/500], global_step: 385400, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.941534, exp_rate<=1 : 0.977510, exp_rate<=2 : 0.994602, exp_rate<=3 : 0.997891, loss: 0.001388, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18470 s, avg_samples: 56.0, ips: 303.20145 samples/s, eta: 8:36:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:17:36] ppocr INFO: epoch: [441/500], global_step: 385500, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.941045, exp_rate<=1 : 0.977221, exp_rate<=2 : 0.994551, exp_rate<=3 : 0.997908, loss: 0.001463, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18883 s, avg_samples: 56.0, ips: 296.56155 samples/s, eta: 8:35:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:17:54] ppocr INFO: epoch: [441/500], global_step: 385600, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.941260, exp_rate<=1 : 0.977421, exp_rate<=2 : 0.994659, exp_rate<=3 : 0.997946, loss: 0.001818, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17509 s, avg_samples: 56.0, ips: 319.83739 samples/s, eta: 8:35:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:18:14] ppocr INFO: epoch: [441/500], global_step: 385700, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.940790, exp_rate<=1 : 0.977262, exp_rate<=2 : 0.994686, exp_rate<=3 : 0.997976, loss: 0.001736, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20492 s, avg_samples: 56.0, ips: 273.27098 samples/s, eta: 8:35:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:18:32] ppocr INFO: epoch: [441/500], global_step: 385800, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.941033, exp_rate<=1 : 0.977470, exp_rate<=2 : 0.994701, exp_rate<=3 : 0.998011, loss: 0.001354, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17336 s, avg_samples: 56.0, ips: 323.03563 samples/s, eta: 8:34:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:18:51] ppocr INFO: epoch: [441/500], global_step: 385900, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.940857, exp_rate<=1 : 0.977391, exp_rate<=2 : 0.994706, exp_rate<=3 : 0.998043, loss: 0.001038, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19679 s, avg_samples: 56.0, ips: 284.57242 samples/s, eta: 8:34:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:19:11] ppocr INFO: epoch: [441/500], global_step: 386000, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.940346, exp_rate<=1 : 0.977054, exp_rate<=2 : 0.994604, exp_rate<=3 : 0.998032, loss: 0.001911, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19344 s, avg_samples: 56.0, ips: 289.50068 samples/s, eta: 8:34:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:19:29] ppocr INFO: epoch: [441/500], global_step: 386100, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.940159, exp_rate<=1 : 0.976985, exp_rate<=2 : 0.994635, exp_rate<=3 : 0.998021, loss: 0.001440, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18168 s, avg_samples: 56.0, ips: 308.23834 samples/s, eta: 8:33:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:19:47] ppocr INFO: epoch: [441/500], global_step: 386200, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.940010, exp_rate<=1 : 0.976906, exp_rate<=2 : 0.994584, exp_rate<=3 : 0.997996, loss: 0.001295, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17458 s, avg_samples: 56.0, ips: 320.77833 samples/s, eta: 8:33:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:20:04] ppocr INFO: epoch: [441/500], global_step: 386300, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.940020, exp_rate<=1 : 0.976911, exp_rate<=2 : 0.994575, exp_rate<=3 : 0.998020, loss: 0.001421, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17912 s, avg_samples: 56.0, ips: 312.64521 samples/s, eta: 8:33:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:20:12] ppocr INFO: epoch: [441/500], global_step: 386340, lr: 0.000100, edit distance: 0.001284, exp_rate: 0.940238, exp_rate<=1 : 0.976981, exp_rate<=2 : 0.994605, exp_rate<=3 : 0.998034, loss: 0.001885, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07272 s, avg_samples: 22.4, ips: 308.04218 samples/s, eta: 8:33:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:20:14] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 23:20:25] ppocr INFO: epoch: [442/500], global_step: 386400, lr: 0.000100, edit distance: 0.000882, exp_rate: 0.953150, exp_rate<=1 : 0.983801, exp_rate<=2 : 0.996837, exp_rate<=3 : 0.998232, loss: 0.001267, avg_reader_cost: 0.02481 s, avg_batch_cost: 0.13049 s, avg_samples: 33.6, ips: 257.49458 samples/s, eta: 8:32:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:20:43] ppocr INFO: epoch: [442/500], global_step: 386500, lr: 0.000100, edit distance: 0.001079, exp_rate: 0.946546, exp_rate<=1 : 0.980228, exp_rate<=2 : 0.995759, exp_rate<=3 : 0.998318, loss: 0.000867, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18314 s, avg_samples: 56.0, ips: 305.77871 samples/s, eta: 8:32:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:21:02] ppocr INFO: epoch: [442/500], global_step: 386600, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.944537, exp_rate<=1 : 0.977525, exp_rate<=2 : 0.994675, exp_rate<=3 : 0.997580, loss: 0.002443, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18854 s, avg_samples: 56.0, ips: 297.02091 samples/s, eta: 8:32:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:21:21] ppocr INFO: epoch: [442/500], global_step: 386700, lr: 0.000100, edit distance: 0.001306, exp_rate: 0.938910, exp_rate<=1 : 0.975356, exp_rate<=2 : 0.994305, exp_rate<=3 : 0.997262, loss: 0.002007, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18730 s, avg_samples: 56.0, ips: 298.99299 samples/s, eta: 8:31:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:21:38] ppocr INFO: epoch: [442/500], global_step: 386800, lr: 0.000100, edit distance: 0.001354, exp_rate: 0.938256, exp_rate<=1 : 0.975267, exp_rate<=2 : 0.994578, exp_rate<=3 : 0.997655, loss: 0.002168, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17501 s, avg_samples: 56.0, ips: 319.97428 samples/s, eta: 8:31:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:21:56] ppocr INFO: epoch: [442/500], global_step: 386900, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.939479, exp_rate<=1 : 0.975621, exp_rate<=2 : 0.994540, exp_rate<=3 : 0.997758, loss: 0.001945, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17699 s, avg_samples: 56.0, ips: 316.39753 samples/s, eta: 8:31:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:22:14] ppocr INFO: epoch: [442/500], global_step: 387000, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.940279, exp_rate<=1 : 0.975802, exp_rate<=2 : 0.994592, exp_rate<=3 : 0.997722, loss: 0.001366, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18149 s, avg_samples: 56.0, ips: 308.54955 samples/s, eta: 8:31:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:22:33] ppocr INFO: epoch: [442/500], global_step: 387100, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.940685, exp_rate<=1 : 0.976051, exp_rate<=2 : 0.994808, exp_rate<=3 : 0.997932, loss: 0.001496, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19270 s, avg_samples: 56.0, ips: 290.60335 samples/s, eta: 8:30:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:22:54] ppocr INFO: epoch: [442/500], global_step: 387200, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.940103, exp_rate<=1 : 0.975814, exp_rate<=2 : 0.994705, exp_rate<=3 : 0.997923, loss: 0.001189, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20440 s, avg_samples: 56.0, ips: 273.97505 samples/s, eta: 8:30:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:23:12] ppocr INFO: epoch: [442/500], global_step: 387300, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.940685, exp_rate<=1 : 0.976234, exp_rate<=2 : 0.994958, exp_rate<=3 : 0.998094, loss: 0.001281, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18168 s, avg_samples: 56.0, ips: 308.23250 samples/s, eta: 8:30:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:23:33] ppocr INFO: epoch: [442/500], global_step: 387400, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.941371, exp_rate<=1 : 0.976523, exp_rate<=2 : 0.995030, exp_rate<=3 : 0.998055, loss: 0.001009, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20703 s, avg_samples: 56.0, ips: 270.49417 samples/s, eta: 8:29:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:23:52] ppocr INFO: epoch: [442/500], global_step: 387500, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.941760, exp_rate<=1 : 0.976682, exp_rate<=2 : 0.995006, exp_rate<=3 : 0.998029, loss: 0.001348, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19121 s, avg_samples: 56.0, ips: 292.87276 samples/s, eta: 8:29:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:24:11] ppocr INFO: epoch: [442/500], global_step: 387600, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.941904, exp_rate<=1 : 0.977006, exp_rate<=2 : 0.995116, exp_rate<=3 : 0.998059, loss: 0.001032, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19121 s, avg_samples: 56.0, ips: 292.86450 samples/s, eta: 8:29:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:24:31] ppocr INFO: epoch: [442/500], global_step: 387700, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.941933, exp_rate<=1 : 0.976976, exp_rate<=2 : 0.994998, exp_rate<=3 : 0.998018, loss: 0.001215, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19666 s, avg_samples: 56.0, ips: 284.74984 samples/s, eta: 8:28:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:24:49] ppocr INFO: epoch: [442/500], global_step: 387800, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.942279, exp_rate<=1 : 0.977263, exp_rate<=2 : 0.995099, exp_rate<=3 : 0.998058, loss: 0.001350, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18528 s, avg_samples: 56.0, ips: 302.24799 samples/s, eta: 8:28:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:25:08] ppocr INFO: epoch: [442/500], global_step: 387900, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.942081, exp_rate<=1 : 0.977384, exp_rate<=2 : 0.995152, exp_rate<=3 : 0.998101, loss: 0.002062, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18441 s, avg_samples: 56.0, ips: 303.67162 samples/s, eta: 8:28:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:25:26] ppocr INFO: epoch: [442/500], global_step: 388000, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.942098, exp_rate<=1 : 0.977519, exp_rate<=2 : 0.995240, exp_rate<=3 : 0.998169, loss: 0.001246, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17850 s, avg_samples: 56.0, ips: 313.71959 samples/s, eta: 8:27:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:25:45] ppocr INFO: epoch: [442/500], global_step: 388100, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.941555, exp_rate<=1 : 0.977354, exp_rate<=2 : 0.995140, exp_rate<=3 : 0.998129, loss: 0.001569, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19364 s, avg_samples: 56.0, ips: 289.19628 samples/s, eta: 8:27:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:26:04] ppocr INFO: epoch: [442/500], global_step: 388200, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.941198, exp_rate<=1 : 0.977413, exp_rate<=2 : 0.995114, exp_rate<=3 : 0.998140, loss: 0.001368, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19102 s, avg_samples: 56.0, ips: 293.15583 samples/s, eta: 8:27:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:26:22] ppocr INFO: epoch: [442/500], global_step: 388300, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.941150, exp_rate<=1 : 0.977405, exp_rate<=2 : 0.995118, exp_rate<=3 : 0.998174, loss: 0.001785, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18272 s, avg_samples: 56.0, ips: 306.47446 samples/s, eta: 8:26:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:26:42] ppocr INFO: epoch: [442/500], global_step: 388400, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.940949, exp_rate<=1 : 0.977271, exp_rate<=2 : 0.995019, exp_rate<=3 : 0.998161, loss: 0.001798, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19735 s, avg_samples: 56.0, ips: 283.75621 samples/s, eta: 8:26:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:27:01] ppocr INFO: epoch: [442/500], global_step: 388500, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.941182, exp_rate<=1 : 0.977329, exp_rate<=2 : 0.995029, exp_rate<=3 : 0.998176, loss: 0.001638, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18570 s, avg_samples: 56.0, ips: 301.56252 samples/s, eta: 8:26:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:27:20] ppocr INFO: epoch: [442/500], global_step: 388600, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.940819, exp_rate<=1 : 0.977225, exp_rate<=2 : 0.994957, exp_rate<=3 : 0.998158, loss: 0.001598, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18922 s, avg_samples: 56.0, ips: 295.95925 samples/s, eta: 8:25:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:27:39] ppocr INFO: epoch: [442/500], global_step: 388700, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.940763, exp_rate<=1 : 0.977193, exp_rate<=2 : 0.994897, exp_rate<=3 : 0.998138, loss: 0.001317, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19573 s, avg_samples: 56.0, ips: 286.10541 samples/s, eta: 8:25:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:27:59] ppocr INFO: epoch: [442/500], global_step: 388800, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.940547, exp_rate<=1 : 0.977073, exp_rate<=2 : 0.994885, exp_rate<=3 : 0.998164, loss: 0.001517, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19501 s, avg_samples: 56.0, ips: 287.16889 samples/s, eta: 8:25:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:28:17] ppocr INFO: epoch: [442/500], global_step: 388900, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.940753, exp_rate<=1 : 0.977235, exp_rate<=2 : 0.994955, exp_rate<=3 : 0.998195, loss: 0.001547, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17692 s, avg_samples: 56.0, ips: 316.52378 samples/s, eta: 8:24:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:28:36] ppocr INFO: epoch: [442/500], global_step: 389000, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.940714, exp_rate<=1 : 0.977176, exp_rate<=2 : 0.994934, exp_rate<=3 : 0.998189, loss: 0.001783, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18934 s, avg_samples: 56.0, ips: 295.76699 samples/s, eta: 8:24:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:28:51] ppocr INFO: epoch: [442/500], global_step: 389080, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.940451, exp_rate<=1 : 0.977027, exp_rate<=2 : 0.994919, exp_rate<=3 : 0.998176, loss: 0.001842, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.15080 s, avg_samples: 44.8, ips: 297.09075 samples/s, eta: 8:24:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:28:53] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 23:28:57] ppocr INFO: epoch: [443/500], global_step: 389100, lr: 0.000100, edit distance: 0.001086, exp_rate: 0.935863, exp_rate<=1 : 0.981548, exp_rate<=2 : 0.994702, exp_rate<=3 : 0.998917, loss: 0.002033, avg_reader_cost: 0.02520 s, avg_batch_cost: 0.05956 s, avg_samples: 11.2, ips: 188.04844 samples/s, eta: 8:24:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:29:16] ppocr INFO: epoch: [443/500], global_step: 389200, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.937697, exp_rate<=1 : 0.976908, exp_rate<=2 : 0.995233, exp_rate<=3 : 0.998443, loss: 0.001791, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19291 s, avg_samples: 56.0, ips: 290.29390 samples/s, eta: 8:24:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:29:34] ppocr INFO: epoch: [443/500], global_step: 389300, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.942983, exp_rate<=1 : 0.978188, exp_rate<=2 : 0.995957, exp_rate<=3 : 0.998643, loss: 0.001402, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17847 s, avg_samples: 56.0, ips: 313.78409 samples/s, eta: 8:23:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:29:52] ppocr INFO: epoch: [443/500], global_step: 389400, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.943984, exp_rate<=1 : 0.979181, exp_rate<=2 : 0.996113, exp_rate<=3 : 0.998624, loss: 0.000624, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17779 s, avg_samples: 56.0, ips: 314.98527 samples/s, eta: 8:23:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:30:09] ppocr INFO: epoch: [443/500], global_step: 389500, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.946364, exp_rate<=1 : 0.980254, exp_rate<=2 : 0.996115, exp_rate<=3 : 0.998651, loss: 0.001253, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17074 s, avg_samples: 56.0, ips: 327.97470 samples/s, eta: 8:23:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:30:25] ppocr INFO: epoch: [443/500], global_step: 389600, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.946953, exp_rate<=1 : 0.980291, exp_rate<=2 : 0.996047, exp_rate<=3 : 0.998671, loss: 0.000961, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16500 s, avg_samples: 56.0, ips: 339.39066 samples/s, eta: 8:22:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:30:43] ppocr INFO: epoch: [443/500], global_step: 389700, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.947232, exp_rate<=1 : 0.980520, exp_rate<=2 : 0.996175, exp_rate<=3 : 0.998805, loss: 0.001361, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17478 s, avg_samples: 56.0, ips: 320.40325 samples/s, eta: 8:22:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:31:02] ppocr INFO: epoch: [443/500], global_step: 389800, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.946214, exp_rate<=1 : 0.980051, exp_rate<=2 : 0.996082, exp_rate<=3 : 0.998711, loss: 0.001626, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18916 s, avg_samples: 56.0, ips: 296.05097 samples/s, eta: 8:22:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:31:20] ppocr INFO: epoch: [443/500], global_step: 389900, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.944786, exp_rate<=1 : 0.979498, exp_rate<=2 : 0.995813, exp_rate<=3 : 0.998553, loss: 0.001132, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18785 s, avg_samples: 56.0, ips: 298.10305 samples/s, eta: 8:21:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:31:41] ppocr INFO: epoch: [443/500], global_step: 390000, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.944581, exp_rate<=1 : 0.979158, exp_rate<=2 : 0.995516, exp_rate<=3 : 0.998360, loss: 0.001465, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20479 s, avg_samples: 56.0, ips: 273.44614 samples/s, eta: 8:21:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:31:59] ppocr INFO: epoch: [443/500], global_step: 390100, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.944680, exp_rate<=1 : 0.979052, exp_rate<=2 : 0.995460, exp_rate<=3 : 0.998340, loss: 0.001267, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18412 s, avg_samples: 56.0, ips: 304.14203 samples/s, eta: 8:21:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:32:20] ppocr INFO: epoch: [443/500], global_step: 390200, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.943856, exp_rate<=1 : 0.978751, exp_rate<=2 : 0.995413, exp_rate<=3 : 0.998360, loss: 0.001529, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20168 s, avg_samples: 56.0, ips: 277.66123 samples/s, eta: 8:20:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:32:39] ppocr INFO: epoch: [443/500], global_step: 390300, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.943423, exp_rate<=1 : 0.978752, exp_rate<=2 : 0.995421, exp_rate<=3 : 0.998327, loss: 0.001267, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19732 s, avg_samples: 56.0, ips: 283.80455 samples/s, eta: 8:20:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:32:59] ppocr INFO: epoch: [443/500], global_step: 390400, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.943092, exp_rate<=1 : 0.978862, exp_rate<=2 : 0.995408, exp_rate<=3 : 0.998317, loss: 0.001734, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19613 s, avg_samples: 56.0, ips: 285.52240 samples/s, eta: 8:20:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:33:18] ppocr INFO: epoch: [443/500], global_step: 390500, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.943002, exp_rate<=1 : 0.978746, exp_rate<=2 : 0.995371, exp_rate<=3 : 0.998295, loss: 0.000964, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19197 s, avg_samples: 56.0, ips: 291.71163 samples/s, eta: 8:19:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:33:38] ppocr INFO: epoch: [443/500], global_step: 390600, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.942856, exp_rate<=1 : 0.978723, exp_rate<=2 : 0.995297, exp_rate<=3 : 0.998298, loss: 0.001294, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19943 s, avg_samples: 56.0, ips: 280.80135 samples/s, eta: 8:19:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:33:57] ppocr INFO: epoch: [443/500], global_step: 390700, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.942479, exp_rate<=1 : 0.978577, exp_rate<=2 : 0.995232, exp_rate<=3 : 0.998314, loss: 0.001766, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19155 s, avg_samples: 56.0, ips: 292.35545 samples/s, eta: 8:19:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:34:17] ppocr INFO: epoch: [443/500], global_step: 390800, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.942294, exp_rate<=1 : 0.978579, exp_rate<=2 : 0.995172, exp_rate<=3 : 0.998257, loss: 0.001767, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20067 s, avg_samples: 56.0, ips: 279.07145 samples/s, eta: 8:18:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:34:36] ppocr INFO: epoch: [443/500], global_step: 390900, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.942359, exp_rate<=1 : 0.978702, exp_rate<=2 : 0.995149, exp_rate<=3 : 0.998233, loss: 0.000889, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18925 s, avg_samples: 56.0, ips: 295.89891 samples/s, eta: 8:18:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:34:55] ppocr INFO: epoch: [443/500], global_step: 391000, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.942606, exp_rate<=1 : 0.978865, exp_rate<=2 : 0.995168, exp_rate<=3 : 0.998252, loss: 0.001488, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19039 s, avg_samples: 56.0, ips: 294.12617 samples/s, eta: 8:18:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:35:15] ppocr INFO: epoch: [443/500], global_step: 391100, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.942717, exp_rate<=1 : 0.978903, exp_rate<=2 : 0.995133, exp_rate<=3 : 0.998228, loss: 0.001484, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19863 s, avg_samples: 56.0, ips: 281.93440 samples/s, eta: 8:18:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:35:35] ppocr INFO: epoch: [443/500], global_step: 391200, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.942625, exp_rate<=1 : 0.978860, exp_rate<=2 : 0.995163, exp_rate<=3 : 0.998232, loss: 0.001023, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19512 s, avg_samples: 56.0, ips: 287.00852 samples/s, eta: 8:17:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:35:53] ppocr INFO: epoch: [443/500], global_step: 391300, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.942622, exp_rate<=1 : 0.978777, exp_rate<=2 : 0.995192, exp_rate<=3 : 0.998255, loss: 0.001513, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17694 s, avg_samples: 56.0, ips: 316.48647 samples/s, eta: 8:17:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:36:11] ppocr INFO: epoch: [443/500], global_step: 391400, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.942309, exp_rate<=1 : 0.978669, exp_rate<=2 : 0.995197, exp_rate<=3 : 0.998239, loss: 0.002237, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18156 s, avg_samples: 56.0, ips: 308.44283 samples/s, eta: 8:17:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:36:28] ppocr INFO: epoch: [443/500], global_step: 391500, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.942281, exp_rate<=1 : 0.978640, exp_rate<=2 : 0.995252, exp_rate<=3 : 0.998311, loss: 0.001215, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17356 s, avg_samples: 56.0, ips: 322.66202 samples/s, eta: 8:16:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:36:46] ppocr INFO: epoch: [443/500], global_step: 391600, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.942388, exp_rate<=1 : 0.978812, exp_rate<=2 : 0.995270, exp_rate<=3 : 0.998338, loss: 0.001705, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18152 s, avg_samples: 56.0, ips: 308.50920 samples/s, eta: 8:16:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:37:05] ppocr INFO: epoch: [443/500], global_step: 391700, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.941698, exp_rate<=1 : 0.978455, exp_rate<=2 : 0.995154, exp_rate<=3 : 0.998322, loss: 0.001509, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19099 s, avg_samples: 56.0, ips: 293.21143 samples/s, eta: 8:16:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:37:24] ppocr INFO: epoch: [443/500], global_step: 391800, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.941786, exp_rate<=1 : 0.978483, exp_rate<=2 : 0.995163, exp_rate<=3 : 0.998325, loss: 0.001210, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18413 s, avg_samples: 56.0, ips: 304.13796 samples/s, eta: 8:15:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:37:27] ppocr INFO: epoch: [443/500], global_step: 391820, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.941800, exp_rate<=1 : 0.978468, exp_rate<=2 : 0.995174, exp_rate<=3 : 0.998332, loss: 0.001699, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03586 s, avg_samples: 11.2, ips: 312.32532 samples/s, eta: 8:15:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:37:29] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 23:37:45] ppocr INFO: epoch: [444/500], global_step: 391900, lr: 0.000100, edit distance: 0.001411, exp_rate: 0.948693, exp_rate<=1 : 0.983395, exp_rate<=2 : 0.996505, exp_rate<=3 : 0.998645, loss: 0.001637, avg_reader_cost: 0.02468 s, avg_batch_cost: 0.17197 s, avg_samples: 44.8, ips: 260.50441 samples/s, eta: 8:15:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:38:03] ppocr INFO: epoch: [444/500], global_step: 392000, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.947640, exp_rate<=1 : 0.982792, exp_rate<=2 : 0.995842, exp_rate<=3 : 0.998721, loss: 0.001599, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18275 s, avg_samples: 56.0, ips: 306.43384 samples/s, eta: 8:15:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:38:23] ppocr INFO: epoch: [444/500], global_step: 392100, lr: 0.000100, edit distance: 0.001443, exp_rate: 0.942877, exp_rate<=1 : 0.979178, exp_rate<=2 : 0.995154, exp_rate<=3 : 0.998241, loss: 0.001650, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19672 s, avg_samples: 56.0, ips: 284.67041 samples/s, eta: 8:14:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:38:42] ppocr INFO: epoch: [444/500], global_step: 392200, lr: 0.000100, edit distance: 0.001388, exp_rate: 0.942330, exp_rate<=1 : 0.979059, exp_rate<=2 : 0.995217, exp_rate<=3 : 0.998409, loss: 0.001633, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19094 s, avg_samples: 56.0, ips: 293.28744 samples/s, eta: 8:14:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:39:00] ppocr INFO: epoch: [444/500], global_step: 392300, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.943394, exp_rate<=1 : 0.979688, exp_rate<=2 : 0.995622, exp_rate<=3 : 0.998710, loss: 0.001439, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18292 s, avg_samples: 56.0, ips: 306.14106 samples/s, eta: 8:14:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:39:20] ppocr INFO: epoch: [444/500], global_step: 392400, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.944083, exp_rate<=1 : 0.979797, exp_rate<=2 : 0.995579, exp_rate<=3 : 0.998591, loss: 0.000986, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19961 s, avg_samples: 56.0, ips: 280.55067 samples/s, eta: 8:13:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:39:39] ppocr INFO: epoch: [444/500], global_step: 392500, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.943673, exp_rate<=1 : 0.979423, exp_rate<=2 : 0.995259, exp_rate<=3 : 0.998528, loss: 0.001903, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19370 s, avg_samples: 56.0, ips: 289.10908 samples/s, eta: 8:13:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:39:59] ppocr INFO: epoch: [444/500], global_step: 392600, lr: 0.000100, edit distance: 0.001329, exp_rate: 0.942751, exp_rate<=1 : 0.978977, exp_rate<=2 : 0.994954, exp_rate<=3 : 0.998334, loss: 0.001448, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19358 s, avg_samples: 56.0, ips: 289.28565 samples/s, eta: 8:13:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:40:16] ppocr INFO: epoch: [444/500], global_step: 392700, lr: 0.000100, edit distance: 0.001324, exp_rate: 0.943353, exp_rate<=1 : 0.979126, exp_rate<=2 : 0.995028, exp_rate<=3 : 0.998377, loss: 0.001206, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16952 s, avg_samples: 56.0, ips: 330.35057 samples/s, eta: 8:12:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:40:35] ppocr INFO: epoch: [444/500], global_step: 392800, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.943527, exp_rate<=1 : 0.979282, exp_rate<=2 : 0.995198, exp_rate<=3 : 0.998441, loss: 0.001583, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18950 s, avg_samples: 56.0, ips: 295.51266 samples/s, eta: 8:12:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:40:54] ppocr INFO: epoch: [444/500], global_step: 392900, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.943206, exp_rate<=1 : 0.979220, exp_rate<=2 : 0.995229, exp_rate<=3 : 0.998399, loss: 0.001389, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19172 s, avg_samples: 56.0, ips: 292.08712 samples/s, eta: 8:12:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:41:13] ppocr INFO: epoch: [444/500], global_step: 393000, lr: 0.000100, edit distance: 0.001304, exp_rate: 0.942898, exp_rate<=1 : 0.978954, exp_rate<=2 : 0.995147, exp_rate<=3 : 0.998354, loss: 0.001669, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18611 s, avg_samples: 56.0, ips: 300.90135 samples/s, eta: 8:11:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:41:31] ppocr INFO: epoch: [444/500], global_step: 393100, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.942606, exp_rate<=1 : 0.978756, exp_rate<=2 : 0.995125, exp_rate<=3 : 0.998334, loss: 0.001092, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18894 s, avg_samples: 56.0, ips: 296.38509 samples/s, eta: 8:11:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:41:50] ppocr INFO: epoch: [444/500], global_step: 393200, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.943451, exp_rate<=1 : 0.978969, exp_rate<=2 : 0.995159, exp_rate<=3 : 0.998343, loss: 0.001356, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18329 s, avg_samples: 56.0, ips: 305.53440 samples/s, eta: 8:11:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:42:10] ppocr INFO: epoch: [444/500], global_step: 393300, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.942348, exp_rate<=1 : 0.978431, exp_rate<=2 : 0.994970, exp_rate<=3 : 0.998361, loss: 0.001043, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20299 s, avg_samples: 56.0, ips: 275.87913 samples/s, eta: 8:10:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:42:30] ppocr INFO: epoch: [444/500], global_step: 393400, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.941557, exp_rate<=1 : 0.977856, exp_rate<=2 : 0.994866, exp_rate<=3 : 0.998289, loss: 0.001672, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20195 s, avg_samples: 56.0, ips: 277.29210 samples/s, eta: 8:10:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:42:48] ppocr INFO: epoch: [444/500], global_step: 393500, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.941686, exp_rate<=1 : 0.977907, exp_rate<=2 : 0.994987, exp_rate<=3 : 0.998332, loss: 0.001256, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17646 s, avg_samples: 56.0, ips: 317.35431 samples/s, eta: 8:10:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:43:07] ppocr INFO: epoch: [444/500], global_step: 393600, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.942106, exp_rate<=1 : 0.977978, exp_rate<=2 : 0.995002, exp_rate<=3 : 0.998345, loss: 0.001302, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19209 s, avg_samples: 56.0, ips: 291.53548 samples/s, eta: 8:10:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:43:27] ppocr INFO: epoch: [444/500], global_step: 393700, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.941653, exp_rate<=1 : 0.977883, exp_rate<=2 : 0.995016, exp_rate<=3 : 0.998387, loss: 0.001137, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19586 s, avg_samples: 56.0, ips: 285.92393 samples/s, eta: 8:09:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:43:45] ppocr INFO: epoch: [444/500], global_step: 393800, lr: 0.000100, edit distance: 0.001299, exp_rate: 0.941868, exp_rate<=1 : 0.978022, exp_rate<=2 : 0.995049, exp_rate<=3 : 0.998423, loss: 0.000983, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17990 s, avg_samples: 56.0, ips: 311.27677 samples/s, eta: 8:09:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:44:04] ppocr INFO: epoch: [444/500], global_step: 393900, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.942202, exp_rate<=1 : 0.978208, exp_rate<=2 : 0.995157, exp_rate<=3 : 0.998447, loss: 0.001623, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19068 s, avg_samples: 56.0, ips: 293.68259 samples/s, eta: 8:09:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:44:22] ppocr INFO: epoch: [444/500], global_step: 394000, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.941951, exp_rate<=1 : 0.978042, exp_rate<=2 : 0.995075, exp_rate<=3 : 0.998394, loss: 0.002155, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17822 s, avg_samples: 56.0, ips: 314.20971 samples/s, eta: 8:08:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:44:40] ppocr INFO: epoch: [444/500], global_step: 394100, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.941624, exp_rate<=1 : 0.977900, exp_rate<=2 : 0.994974, exp_rate<=3 : 0.998327, loss: 0.001542, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18431 s, avg_samples: 56.0, ips: 303.84163 samples/s, eta: 8:08:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:44:57] ppocr INFO: epoch: [444/500], global_step: 394200, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941784, exp_rate<=1 : 0.977990, exp_rate<=2 : 0.995006, exp_rate<=3 : 0.998321, loss: 0.001613, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16470 s, avg_samples: 56.0, ips: 340.02181 samples/s, eta: 8:08:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:45:16] ppocr INFO: epoch: [444/500], global_step: 394300, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.941631, exp_rate<=1 : 0.977913, exp_rate<=2 : 0.995020, exp_rate<=3 : 0.998323, loss: 0.001469, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19763 s, avg_samples: 56.0, ips: 283.35465 samples/s, eta: 8:07:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:45:35] ppocr INFO: epoch: [444/500], global_step: 394400, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.941249, exp_rate<=1 : 0.977688, exp_rate<=2 : 0.995020, exp_rate<=3 : 0.998309, loss: 0.001718, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18604 s, avg_samples: 56.0, ips: 301.00998 samples/s, eta: 8:07:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:45:54] ppocr INFO: epoch: [444/500], global_step: 394500, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.941230, exp_rate<=1 : 0.977670, exp_rate<=2 : 0.995048, exp_rate<=3 : 0.998319, loss: 0.001482, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19100 s, avg_samples: 56.0, ips: 293.19207 samples/s, eta: 8:07:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:46:05] ppocr INFO: epoch: [444/500], global_step: 394560, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.941320, exp_rate<=1 : 0.977676, exp_rate<=2 : 0.995044, exp_rate<=3 : 0.998309, loss: 0.001873, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10747 s, avg_samples: 33.6, ips: 312.65419 samples/s, eta: 8:06:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:46:07] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 23:46:16] ppocr INFO: epoch: [445/500], global_step: 394600, lr: 0.000100, edit distance: 0.001511, exp_rate: 0.942796, exp_rate<=1 : 0.977746, exp_rate<=2 : 0.996782, exp_rate<=3 : 0.997493, loss: 0.001640, avg_reader_cost: 0.02618 s, avg_batch_cost: 0.10605 s, avg_samples: 22.4, ips: 211.21519 samples/s, eta: 8:06:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:46:34] ppocr INFO: epoch: [445/500], global_step: 394700, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.947524, exp_rate<=1 : 0.980384, exp_rate<=2 : 0.994849, exp_rate<=3 : 0.997605, loss: 0.001658, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18263 s, avg_samples: 56.0, ips: 306.62394 samples/s, eta: 8:06:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:46:52] ppocr INFO: epoch: [445/500], global_step: 394800, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.946820, exp_rate<=1 : 0.979925, exp_rate<=2 : 0.995147, exp_rate<=3 : 0.997986, loss: 0.001281, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17701 s, avg_samples: 56.0, ips: 316.37364 samples/s, eta: 8:06:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:47:10] ppocr INFO: epoch: [445/500], global_step: 394900, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.947961, exp_rate<=1 : 0.980680, exp_rate<=2 : 0.995353, exp_rate<=3 : 0.998217, loss: 0.001278, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17990 s, avg_samples: 56.0, ips: 311.27957 samples/s, eta: 8:05:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:47:27] ppocr INFO: epoch: [445/500], global_step: 395000, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.948754, exp_rate<=1 : 0.980763, exp_rate<=2 : 0.995478, exp_rate<=3 : 0.998341, loss: 0.001251, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17316 s, avg_samples: 56.0, ips: 323.40058 samples/s, eta: 8:05:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:47:47] ppocr INFO: epoch: [445/500], global_step: 395100, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.948030, exp_rate<=1 : 0.980421, exp_rate<=2 : 0.995451, exp_rate<=3 : 0.998404, loss: 0.001687, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19657 s, avg_samples: 56.0, ips: 284.88699 samples/s, eta: 8:05:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:48:05] ppocr INFO: epoch: [445/500], global_step: 395200, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.947052, exp_rate<=1 : 0.979701, exp_rate<=2 : 0.995365, exp_rate<=3 : 0.998409, loss: 0.001782, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18788 s, avg_samples: 56.0, ips: 298.05729 samples/s, eta: 8:04:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:48:24] ppocr INFO: epoch: [445/500], global_step: 395300, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.946173, exp_rate<=1 : 0.979081, exp_rate<=2 : 0.995270, exp_rate<=3 : 0.998403, loss: 0.000933, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18693 s, avg_samples: 56.0, ips: 299.57977 samples/s, eta: 8:04:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:48:42] ppocr INFO: epoch: [445/500], global_step: 395400, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.946471, exp_rate<=1 : 0.979102, exp_rate<=2 : 0.995291, exp_rate<=3 : 0.998456, loss: 0.001417, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18181 s, avg_samples: 56.0, ips: 308.01186 samples/s, eta: 8:04:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:49:02] ppocr INFO: epoch: [445/500], global_step: 395500, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.946505, exp_rate<=1 : 0.979100, exp_rate<=2 : 0.995285, exp_rate<=3 : 0.998463, loss: 0.001436, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19258 s, avg_samples: 56.0, ips: 290.78625 samples/s, eta: 8:03:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:49:21] ppocr INFO: epoch: [445/500], global_step: 395600, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.946316, exp_rate<=1 : 0.979099, exp_rate<=2 : 0.995413, exp_rate<=3 : 0.998535, loss: 0.000947, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19330 s, avg_samples: 56.0, ips: 289.70826 samples/s, eta: 8:03:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:49:40] ppocr INFO: epoch: [445/500], global_step: 395700, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.946381, exp_rate<=1 : 0.979112, exp_rate<=2 : 0.995319, exp_rate<=3 : 0.998443, loss: 0.002048, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18648 s, avg_samples: 56.0, ips: 300.29418 samples/s, eta: 8:03:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:50:01] ppocr INFO: epoch: [445/500], global_step: 395800, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.943940, exp_rate<=1 : 0.978152, exp_rate<=2 : 0.994937, exp_rate<=3 : 0.998309, loss: 0.002058, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21763 s, avg_samples: 56.0, ips: 257.31683 samples/s, eta: 8:03:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:50:20] ppocr INFO: epoch: [445/500], global_step: 395900, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.943438, exp_rate<=1 : 0.978060, exp_rate<=2 : 0.995012, exp_rate<=3 : 0.998336, loss: 0.001000, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18692 s, avg_samples: 56.0, ips: 299.58940 samples/s, eta: 8:02:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:50:39] ppocr INFO: epoch: [445/500], global_step: 396000, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.943151, exp_rate<=1 : 0.977991, exp_rate<=2 : 0.995006, exp_rate<=3 : 0.998340, loss: 0.001388, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18529 s, avg_samples: 56.0, ips: 302.22543 samples/s, eta: 8:02:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:50:57] ppocr INFO: epoch: [445/500], global_step: 396100, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.942936, exp_rate<=1 : 0.977896, exp_rate<=2 : 0.994977, exp_rate<=3 : 0.998346, loss: 0.001411, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18438 s, avg_samples: 56.0, ips: 303.72522 samples/s, eta: 8:02:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:51:17] ppocr INFO: epoch: [445/500], global_step: 396200, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.943027, exp_rate<=1 : 0.978068, exp_rate<=2 : 0.994910, exp_rate<=3 : 0.998288, loss: 0.000911, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19506 s, avg_samples: 56.0, ips: 287.08702 samples/s, eta: 8:01:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:51:34] ppocr INFO: epoch: [445/500], global_step: 396300, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.943526, exp_rate<=1 : 0.978258, exp_rate<=2 : 0.994956, exp_rate<=3 : 0.998299, loss: 0.001481, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17306 s, avg_samples: 56.0, ips: 323.58185 samples/s, eta: 8:01:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:51:53] ppocr INFO: epoch: [445/500], global_step: 396400, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.943242, exp_rate<=1 : 0.978187, exp_rate<=2 : 0.995010, exp_rate<=3 : 0.998276, loss: 0.001839, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18588 s, avg_samples: 56.0, ips: 301.27407 samples/s, eta: 8:01:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:52:11] ppocr INFO: epoch: [445/500], global_step: 396500, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.942888, exp_rate<=1 : 0.978124, exp_rate<=2 : 0.994963, exp_rate<=3 : 0.998299, loss: 0.001313, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18271 s, avg_samples: 56.0, ips: 306.50449 samples/s, eta: 8:00:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:52:29] ppocr INFO: epoch: [445/500], global_step: 396600, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.943095, exp_rate<=1 : 0.978274, exp_rate<=2 : 0.995004, exp_rate<=3 : 0.998304, loss: 0.001427, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18116 s, avg_samples: 56.0, ips: 309.12585 samples/s, eta: 8:00:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:52:48] ppocr INFO: epoch: [445/500], global_step: 396700, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.942866, exp_rate<=1 : 0.978086, exp_rate<=2 : 0.994939, exp_rate<=3 : 0.998249, loss: 0.001502, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18780 s, avg_samples: 56.0, ips: 298.19408 samples/s, eta: 8:00:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:53:07] ppocr INFO: epoch: [445/500], global_step: 396800, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.942381, exp_rate<=1 : 0.977837, exp_rate<=2 : 0.994950, exp_rate<=3 : 0.998287, loss: 0.001595, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19373 s, avg_samples: 56.0, ips: 289.06229 samples/s, eta: 7:59:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:53:26] ppocr INFO: epoch: [445/500], global_step: 396900, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.942187, exp_rate<=1 : 0.977680, exp_rate<=2 : 0.994894, exp_rate<=3 : 0.998267, loss: 0.001568, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18962 s, avg_samples: 56.0, ips: 295.32118 samples/s, eta: 7:59:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:53:44] ppocr INFO: epoch: [445/500], global_step: 397000, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.942251, exp_rate<=1 : 0.977818, exp_rate<=2 : 0.994977, exp_rate<=3 : 0.998299, loss: 0.001478, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17511 s, avg_samples: 56.0, ips: 319.79632 samples/s, eta: 7:59:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:54:03] ppocr INFO: epoch: [445/500], global_step: 397100, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.941672, exp_rate<=1 : 0.977587, exp_rate<=2 : 0.994932, exp_rate<=3 : 0.998282, loss: 0.001520, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19333 s, avg_samples: 56.0, ips: 289.65932 samples/s, eta: 7:58:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:54:21] ppocr INFO: epoch: [445/500], global_step: 397200, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.941584, exp_rate<=1 : 0.977482, exp_rate<=2 : 0.994834, exp_rate<=3 : 0.998235, loss: 0.001179, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17803 s, avg_samples: 56.0, ips: 314.56074 samples/s, eta: 7:58:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:54:40] ppocr INFO: epoch: [445/500], global_step: 397300, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.941531, exp_rate<=1 : 0.977376, exp_rate<=2 : 0.994845, exp_rate<=3 : 0.998248, loss: 0.001519, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19369 s, avg_samples: 56.0, ips: 289.11468 samples/s, eta: 7:58:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:54:42] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/10 23:54:43] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_445 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/10 23:55:03] ppocr INFO: epoch: [446/500], global_step: 397400, lr: 0.000100, edit distance: 0.001399, exp_rate: 0.940575, exp_rate<=1 : 0.978393, exp_rate<=2 : 0.995081, exp_rate<=3 : 0.998914, loss: 0.001425, avg_reader_cost: 0.03167 s, avg_batch_cost: 0.22419 s, avg_samples: 56.0, ips: 249.78707 samples/s, eta: 7:57:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:55:21] ppocr INFO: epoch: [446/500], global_step: 397500, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.941615, exp_rate<=1 : 0.977503, exp_rate<=2 : 0.994844, exp_rate<=3 : 0.998775, loss: 0.001177, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18642 s, avg_samples: 56.0, ips: 300.39906 samples/s, eta: 7:57:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:55:40] ppocr INFO: epoch: [446/500], global_step: 397600, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.944101, exp_rate<=1 : 0.978029, exp_rate<=2 : 0.995234, exp_rate<=3 : 0.998652, loss: 0.001436, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19048 s, avg_samples: 56.0, ips: 293.99579 samples/s, eta: 7:57:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:55:58] ppocr INFO: epoch: [446/500], global_step: 397700, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.946518, exp_rate<=1 : 0.979358, exp_rate<=2 : 0.995507, exp_rate<=3 : 0.998811, loss: 0.001152, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17987 s, avg_samples: 56.0, ips: 311.33854 samples/s, eta: 7:56:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:56:16] ppocr INFO: epoch: [446/500], global_step: 397800, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.947077, exp_rate<=1 : 0.979956, exp_rate<=2 : 0.995810, exp_rate<=3 : 0.998818, loss: 0.001337, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17274 s, avg_samples: 56.0, ips: 324.18979 samples/s, eta: 7:56:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:56:35] ppocr INFO: epoch: [446/500], global_step: 397900, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.946264, exp_rate<=1 : 0.979676, exp_rate<=2 : 0.995566, exp_rate<=3 : 0.998635, loss: 0.001124, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19030 s, avg_samples: 56.0, ips: 294.27829 samples/s, eta: 7:56:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:56:54] ppocr INFO: epoch: [446/500], global_step: 398000, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.946258, exp_rate<=1 : 0.979642, exp_rate<=2 : 0.995568, exp_rate<=3 : 0.998669, loss: 0.001415, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18699 s, avg_samples: 56.0, ips: 299.47766 samples/s, eta: 7:55:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:57:13] ppocr INFO: epoch: [446/500], global_step: 398100, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.945943, exp_rate<=1 : 0.979227, exp_rate<=2 : 0.995423, exp_rate<=3 : 0.998641, loss: 0.001248, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19479 s, avg_samples: 56.0, ips: 287.48406 samples/s, eta: 7:55:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:57:31] ppocr INFO: epoch: [446/500], global_step: 398200, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.944868, exp_rate<=1 : 0.978723, exp_rate<=2 : 0.995272, exp_rate<=3 : 0.998524, loss: 0.001425, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17828 s, avg_samples: 56.0, ips: 314.11011 samples/s, eta: 7:55:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:57:48] ppocr INFO: epoch: [446/500], global_step: 398300, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.944893, exp_rate<=1 : 0.978821, exp_rate<=2 : 0.995316, exp_rate<=3 : 0.998508, loss: 0.001600, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17280 s, avg_samples: 56.0, ips: 324.06686 samples/s, eta: 7:55:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:58:08] ppocr INFO: epoch: [446/500], global_step: 398400, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.944377, exp_rate<=1 : 0.978653, exp_rate<=2 : 0.995322, exp_rate<=3 : 0.998517, loss: 0.000909, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20154 s, avg_samples: 56.0, ips: 277.86394 samples/s, eta: 7:54:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:58:26] ppocr INFO: epoch: [446/500], global_step: 398500, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.943983, exp_rate<=1 : 0.978658, exp_rate<=2 : 0.995317, exp_rate<=3 : 0.998426, loss: 0.001392, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17781 s, avg_samples: 56.0, ips: 314.94459 samples/s, eta: 7:54:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:58:45] ppocr INFO: epoch: [446/500], global_step: 398600, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.943645, exp_rate<=1 : 0.978487, exp_rate<=2 : 0.995340, exp_rate<=3 : 0.998395, loss: 0.001285, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18825 s, avg_samples: 56.0, ips: 297.48338 samples/s, eta: 7:54:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:59:05] ppocr INFO: epoch: [446/500], global_step: 398700, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.943457, exp_rate<=1 : 0.978326, exp_rate<=2 : 0.995292, exp_rate<=3 : 0.998405, loss: 0.001609, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19681 s, avg_samples: 56.0, ips: 284.53632 samples/s, eta: 7:53:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:59:22] ppocr INFO: epoch: [446/500], global_step: 398800, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.943711, exp_rate<=1 : 0.978442, exp_rate<=2 : 0.995310, exp_rate<=3 : 0.998395, loss: 0.001962, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17713 s, avg_samples: 56.0, ips: 316.15651 samples/s, eta: 7:53:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/10 23:59:41] ppocr INFO: epoch: [446/500], global_step: 398900, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.942937, exp_rate<=1 : 0.978405, exp_rate<=2 : 0.995286, exp_rate<=3 : 0.998428, loss: 0.001575, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18692 s, avg_samples: 56.0, ips: 299.59177 samples/s, eta: 7:53:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:00:00] ppocr INFO: epoch: [446/500], global_step: 399000, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.942475, exp_rate<=1 : 0.978291, exp_rate<=2 : 0.995304, exp_rate<=3 : 0.998439, loss: 0.001417, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19069 s, avg_samples: 56.0, ips: 293.67286 samples/s, eta: 7:52:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:00:20] ppocr INFO: epoch: [446/500], global_step: 399100, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.941465, exp_rate<=1 : 0.977827, exp_rate<=2 : 0.995097, exp_rate<=3 : 0.998424, loss: 0.001469, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20176 s, avg_samples: 56.0, ips: 277.55636 samples/s, eta: 7:52:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:00:39] ppocr INFO: epoch: [446/500], global_step: 399200, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.941890, exp_rate<=1 : 0.977897, exp_rate<=2 : 0.995087, exp_rate<=3 : 0.998381, loss: 0.001736, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18443 s, avg_samples: 56.0, ips: 303.63081 samples/s, eta: 7:52:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:00:58] ppocr INFO: epoch: [446/500], global_step: 399300, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.941573, exp_rate<=1 : 0.977719, exp_rate<=2 : 0.995028, exp_rate<=3 : 0.998355, loss: 0.001364, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19251 s, avg_samples: 56.0, ips: 290.89531 samples/s, eta: 7:51:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:01:16] ppocr INFO: epoch: [446/500], global_step: 399400, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.941621, exp_rate<=1 : 0.977743, exp_rate<=2 : 0.995007, exp_rate<=3 : 0.998330, loss: 0.001554, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17546 s, avg_samples: 56.0, ips: 319.16076 samples/s, eta: 7:51:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:01:34] ppocr INFO: epoch: [446/500], global_step: 399500, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.941629, exp_rate<=1 : 0.977573, exp_rate<=2 : 0.994970, exp_rate<=3 : 0.998339, loss: 0.001844, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17870 s, avg_samples: 56.0, ips: 313.37102 samples/s, eta: 7:51:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:01:53] ppocr INFO: epoch: [446/500], global_step: 399600, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941254, exp_rate<=1 : 0.977360, exp_rate<=2 : 0.994979, exp_rate<=3 : 0.998369, loss: 0.001509, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19086 s, avg_samples: 56.0, ips: 293.40370 samples/s, eta: 7:50:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:02:12] ppocr INFO: epoch: [446/500], global_step: 399700, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.940974, exp_rate<=1 : 0.977142, exp_rate<=2 : 0.994892, exp_rate<=3 : 0.998357, loss: 0.001310, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19055 s, avg_samples: 56.0, ips: 293.89049 samples/s, eta: 7:50:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:02:30] ppocr INFO: epoch: [446/500], global_step: 399800, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.940686, exp_rate<=1 : 0.977104, exp_rate<=2 : 0.994883, exp_rate<=3 : 0.998377, loss: 0.002032, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18561 s, avg_samples: 56.0, ips: 301.71014 samples/s, eta: 7:50:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:02:50] ppocr INFO: epoch: [446/500], global_step: 399900, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.940263, exp_rate<=1 : 0.976904, exp_rate<=2 : 0.994810, exp_rate<=3 : 0.998368, loss: 0.001743, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19619 s, avg_samples: 56.0, ips: 285.43558 samples/s, eta: 7:49:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:03:09] ppocr INFO: epoch: [446/500], global_step: 400000, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.940303, exp_rate<=1 : 0.976798, exp_rate<=2 : 0.994753, exp_rate<=3 : 0.998348, loss: 0.001708, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18890 s, avg_samples: 56.0, ips: 296.44547 samples/s, eta: 7:49:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:03:17] ppocr INFO: epoch: [446/500], global_step: 400040, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.940319, exp_rate<=1 : 0.976777, exp_rate<=2 : 0.994735, exp_rate<=3 : 0.998330, loss: 0.001522, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.08136 s, avg_samples: 22.4, ips: 275.32862 samples/s, eta: 7:49:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:03:19] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 00:03:31] ppocr INFO: epoch: [447/500], global_step: 400100, lr: 0.000100, edit distance: 0.000769, exp_rate: 0.957497, exp_rate<=1 : 0.985326, exp_rate<=2 : 0.996817, exp_rate<=3 : 0.998232, loss: 0.000982, avg_reader_cost: 0.02522 s, avg_batch_cost: 0.13483 s, avg_samples: 33.6, ips: 249.20401 samples/s, eta: 7:49:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:03:48] ppocr INFO: epoch: [447/500], global_step: 400200, lr: 0.000100, edit distance: 0.000862, exp_rate: 0.955661, exp_rate<=1 : 0.984101, exp_rate<=2 : 0.996463, exp_rate<=3 : 0.998745, loss: 0.001056, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17222 s, avg_samples: 56.0, ips: 325.16936 samples/s, eta: 7:48:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:04:07] ppocr INFO: epoch: [447/500], global_step: 400300, lr: 0.000100, edit distance: 0.000979, exp_rate: 0.950389, exp_rate<=1 : 0.981164, exp_rate<=2 : 0.995876, exp_rate<=3 : 0.998318, loss: 0.001322, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18987 s, avg_samples: 56.0, ips: 294.93667 samples/s, eta: 7:48:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:04:26] ppocr INFO: epoch: [447/500], global_step: 400400, lr: 0.000100, edit distance: 0.001024, exp_rate: 0.948976, exp_rate<=1 : 0.979939, exp_rate<=2 : 0.995294, exp_rate<=3 : 0.998013, loss: 0.001114, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19381 s, avg_samples: 56.0, ips: 288.94317 samples/s, eta: 7:48:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:04:46] ppocr INFO: epoch: [447/500], global_step: 400500, lr: 0.000100, edit distance: 0.001020, exp_rate: 0.948001, exp_rate<=1 : 0.980104, exp_rate<=2 : 0.995278, exp_rate<=3 : 0.998097, loss: 0.001117, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20149 s, avg_samples: 56.0, ips: 277.92936 samples/s, eta: 7:48:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:05:06] ppocr INFO: epoch: [447/500], global_step: 400600, lr: 0.000100, edit distance: 0.001027, exp_rate: 0.947958, exp_rate<=1 : 0.980322, exp_rate<=2 : 0.995149, exp_rate<=3 : 0.998214, loss: 0.001307, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19170 s, avg_samples: 56.0, ips: 292.11918 samples/s, eta: 7:47:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:05:23] ppocr INFO: epoch: [447/500], global_step: 400700, lr: 0.000100, edit distance: 0.001068, exp_rate: 0.948413, exp_rate<=1 : 0.980563, exp_rate<=2 : 0.995364, exp_rate<=3 : 0.998353, loss: 0.001188, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17453 s, avg_samples: 56.0, ips: 320.87035 samples/s, eta: 7:47:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:05:42] ppocr INFO: epoch: [447/500], global_step: 400800, lr: 0.000100, edit distance: 0.001061, exp_rate: 0.948289, exp_rate<=1 : 0.980855, exp_rate<=2 : 0.995331, exp_rate<=3 : 0.998349, loss: 0.001391, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18918 s, avg_samples: 56.0, ips: 296.01825 samples/s, eta: 7:47:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:06:01] ppocr INFO: epoch: [447/500], global_step: 400900, lr: 0.000100, edit distance: 0.001093, exp_rate: 0.947307, exp_rate<=1 : 0.980233, exp_rate<=2 : 0.995348, exp_rate<=3 : 0.998305, loss: 0.001240, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18735 s, avg_samples: 56.0, ips: 298.89820 samples/s, eta: 7:46:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:06:19] ppocr INFO: epoch: [447/500], global_step: 401000, lr: 0.000100, edit distance: 0.001101, exp_rate: 0.947368, exp_rate<=1 : 0.980515, exp_rate<=2 : 0.995475, exp_rate<=3 : 0.998345, loss: 0.001092, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17970 s, avg_samples: 56.0, ips: 311.62788 samples/s, eta: 7:46:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:06:37] ppocr INFO: epoch: [447/500], global_step: 401100, lr: 0.000100, edit distance: 0.001114, exp_rate: 0.946649, exp_rate<=1 : 0.980415, exp_rate<=2 : 0.995495, exp_rate<=3 : 0.998317, loss: 0.000996, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18671 s, avg_samples: 56.0, ips: 299.92377 samples/s, eta: 7:46:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:06:58] ppocr INFO: epoch: [447/500], global_step: 401200, lr: 0.000100, edit distance: 0.001124, exp_rate: 0.945923, exp_rate<=1 : 0.980036, exp_rate<=2 : 0.995361, exp_rate<=3 : 0.998266, loss: 0.001347, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20145 s, avg_samples: 56.0, ips: 277.98528 samples/s, eta: 7:45:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:07:16] ppocr INFO: epoch: [447/500], global_step: 401300, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.945252, exp_rate<=1 : 0.979580, exp_rate<=2 : 0.995264, exp_rate<=3 : 0.998254, loss: 0.001244, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18061 s, avg_samples: 56.0, ips: 310.05178 samples/s, eta: 7:45:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:07:35] ppocr INFO: epoch: [447/500], global_step: 401400, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.945331, exp_rate<=1 : 0.979581, exp_rate<=2 : 0.995303, exp_rate<=3 : 0.998265, loss: 0.001239, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19755 s, avg_samples: 56.0, ips: 283.46763 samples/s, eta: 7:45:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:07:56] ppocr INFO: epoch: [447/500], global_step: 401500, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.945223, exp_rate<=1 : 0.979496, exp_rate<=2 : 0.995218, exp_rate<=3 : 0.998223, loss: 0.001077, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20389 s, avg_samples: 56.0, ips: 274.66332 samples/s, eta: 7:44:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:08:14] ppocr INFO: epoch: [447/500], global_step: 401600, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.945049, exp_rate<=1 : 0.979404, exp_rate<=2 : 0.995156, exp_rate<=3 : 0.998231, loss: 0.000949, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18575 s, avg_samples: 56.0, ips: 301.47762 samples/s, eta: 7:44:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:08:34] ppocr INFO: epoch: [447/500], global_step: 401700, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.945254, exp_rate<=1 : 0.979510, exp_rate<=2 : 0.995197, exp_rate<=3 : 0.998219, loss: 0.001364, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19646 s, avg_samples: 56.0, ips: 285.05227 samples/s, eta: 7:44:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:08:53] ppocr INFO: epoch: [447/500], global_step: 401800, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.945000, exp_rate<=1 : 0.979308, exp_rate<=2 : 0.995209, exp_rate<=3 : 0.998225, loss: 0.000844, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18571 s, avg_samples: 56.0, ips: 301.53888 samples/s, eta: 7:43:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:09:12] ppocr INFO: epoch: [447/500], global_step: 401900, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.944946, exp_rate<=1 : 0.979132, exp_rate<=2 : 0.995171, exp_rate<=3 : 0.998207, loss: 0.001851, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19397 s, avg_samples: 56.0, ips: 288.70649 samples/s, eta: 7:43:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:09:30] ppocr INFO: epoch: [447/500], global_step: 402000, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.944911, exp_rate<=1 : 0.979100, exp_rate<=2 : 0.995238, exp_rate<=3 : 0.998255, loss: 0.001792, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17808 s, avg_samples: 56.0, ips: 314.47107 samples/s, eta: 7:43:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:09:49] ppocr INFO: epoch: [447/500], global_step: 402100, lr: 0.000100, edit distance: 0.001180, exp_rate: 0.944471, exp_rate<=1 : 0.978879, exp_rate<=2 : 0.995126, exp_rate<=3 : 0.998230, loss: 0.001837, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19144 s, avg_samples: 56.0, ips: 292.51878 samples/s, eta: 7:42:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:10:08] ppocr INFO: epoch: [447/500], global_step: 402200, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.943593, exp_rate<=1 : 0.978581, exp_rate<=2 : 0.995051, exp_rate<=3 : 0.998197, loss: 0.001992, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18679 s, avg_samples: 56.0, ips: 299.80542 samples/s, eta: 7:42:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:10:25] ppocr INFO: epoch: [447/500], global_step: 402300, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.943353, exp_rate<=1 : 0.978523, exp_rate<=2 : 0.995091, exp_rate<=3 : 0.998209, loss: 0.001245, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17556 s, avg_samples: 56.0, ips: 318.98514 samples/s, eta: 7:42:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:10:44] ppocr INFO: epoch: [447/500], global_step: 402400, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.942881, exp_rate<=1 : 0.978311, exp_rate<=2 : 0.995030, exp_rate<=3 : 0.998189, loss: 0.001018, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18411 s, avg_samples: 56.0, ips: 304.16083 samples/s, eta: 7:41:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:11:03] ppocr INFO: epoch: [447/500], global_step: 402500, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.942645, exp_rate<=1 : 0.978246, exp_rate<=2 : 0.995041, exp_rate<=3 : 0.998210, loss: 0.001248, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18683 s, avg_samples: 56.0, ips: 299.73522 samples/s, eta: 7:41:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:11:22] ppocr INFO: epoch: [447/500], global_step: 402600, lr: 0.000100, edit distance: 0.001256, exp_rate: 0.942327, exp_rate<=1 : 0.978227, exp_rate<=2 : 0.995016, exp_rate<=3 : 0.998249, loss: 0.001793, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19013 s, avg_samples: 56.0, ips: 294.54123 samples/s, eta: 7:41:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:11:41] ppocr INFO: epoch: [447/500], global_step: 402700, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.941796, exp_rate<=1 : 0.978025, exp_rate<=2 : 0.995011, exp_rate<=3 : 0.998269, loss: 0.001753, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19126 s, avg_samples: 56.0, ips: 292.79314 samples/s, eta: 7:41:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:11:54] ppocr INFO: epoch: [447/500], global_step: 402780, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.941873, exp_rate<=1 : 0.977950, exp_rate<=2 : 0.995028, exp_rate<=3 : 0.998271, loss: 0.001172, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.13665 s, avg_samples: 44.8, ips: 327.84481 samples/s, eta: 7:40:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:11:56] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 00:12:00] ppocr INFO: epoch: [448/500], global_step: 402800, lr: 0.000100, edit distance: 0.000853, exp_rate: 0.948052, exp_rate<=1 : 0.978869, exp_rate<=2 : 1.000000, exp_rate<=3 : 1.000000, loss: 0.001310, avg_reader_cost: 0.02578 s, avg_batch_cost: 0.06085 s, avg_samples: 11.2, ips: 184.04461 samples/s, eta: 7:40:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:12:19] ppocr INFO: epoch: [448/500], global_step: 402900, lr: 0.000100, edit distance: 0.001092, exp_rate: 0.945713, exp_rate<=1 : 0.979767, exp_rate<=2 : 0.996124, exp_rate<=3 : 0.998467, loss: 0.001379, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18654 s, avg_samples: 56.0, ips: 300.20191 samples/s, eta: 7:40:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:12:38] ppocr INFO: epoch: [448/500], global_step: 403000, lr: 0.000100, edit distance: 0.001116, exp_rate: 0.945462, exp_rate<=1 : 0.981246, exp_rate<=2 : 0.996692, exp_rate<=3 : 0.999001, loss: 0.001360, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18369 s, avg_samples: 56.0, ips: 304.86781 samples/s, eta: 7:40:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:12:57] ppocr INFO: epoch: [448/500], global_step: 403100, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.945869, exp_rate<=1 : 0.981416, exp_rate<=2 : 0.996338, exp_rate<=3 : 0.999008, loss: 0.001144, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18994 s, avg_samples: 56.0, ips: 294.83538 samples/s, eta: 7:39:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:13:15] ppocr INFO: epoch: [448/500], global_step: 403200, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.943760, exp_rate<=1 : 0.980381, exp_rate<=2 : 0.995851, exp_rate<=3 : 0.998760, loss: 0.002312, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18686 s, avg_samples: 56.0, ips: 299.68623 samples/s, eta: 7:39:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:13:35] ppocr INFO: epoch: [448/500], global_step: 403300, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.941903, exp_rate<=1 : 0.979632, exp_rate<=2 : 0.995824, exp_rate<=3 : 0.998827, loss: 0.001488, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19481 s, avg_samples: 56.0, ips: 287.46315 samples/s, eta: 7:39:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:13:53] ppocr INFO: epoch: [448/500], global_step: 403400, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.943599, exp_rate<=1 : 0.980347, exp_rate<=2 : 0.996108, exp_rate<=3 : 0.998843, loss: 0.001023, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18160 s, avg_samples: 56.0, ips: 308.36178 samples/s, eta: 7:38:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:14:13] ppocr INFO: epoch: [448/500], global_step: 403500, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.944053, exp_rate<=1 : 0.979969, exp_rate<=2 : 0.995824, exp_rate<=3 : 0.998668, loss: 0.001125, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19583 s, avg_samples: 56.0, ips: 285.95712 samples/s, eta: 7:38:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:14:31] ppocr INFO: epoch: [448/500], global_step: 403600, lr: 0.000100, edit distance: 0.001126, exp_rate: 0.944537, exp_rate<=1 : 0.980132, exp_rate<=2 : 0.995779, exp_rate<=3 : 0.998656, loss: 0.000995, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18840 s, avg_samples: 56.0, ips: 297.24417 samples/s, eta: 7:38:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:14:50] ppocr INFO: epoch: [448/500], global_step: 403700, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.944849, exp_rate<=1 : 0.980379, exp_rate<=2 : 0.995893, exp_rate<=3 : 0.998653, loss: 0.001749, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18820 s, avg_samples: 56.0, ips: 297.54811 samples/s, eta: 7:37:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:15:09] ppocr INFO: epoch: [448/500], global_step: 403800, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.944361, exp_rate<=1 : 0.980113, exp_rate<=2 : 0.995979, exp_rate<=3 : 0.998686, loss: 0.001545, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18912 s, avg_samples: 56.0, ips: 296.10373 samples/s, eta: 7:37:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:15:29] ppocr INFO: epoch: [448/500], global_step: 403900, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.943683, exp_rate<=1 : 0.979729, exp_rate<=2 : 0.995797, exp_rate<=3 : 0.998681, loss: 0.001100, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19527 s, avg_samples: 56.0, ips: 286.77674 samples/s, eta: 7:37:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:15:49] ppocr INFO: epoch: [448/500], global_step: 404000, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.943766, exp_rate<=1 : 0.979775, exp_rate<=2 : 0.995791, exp_rate<=3 : 0.998594, loss: 0.001213, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19797 s, avg_samples: 56.0, ips: 282.87661 samples/s, eta: 7:36:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:16:06] ppocr INFO: epoch: [448/500], global_step: 404100, lr: 0.000100, edit distance: 0.001132, exp_rate: 0.943661, exp_rate<=1 : 0.979714, exp_rate<=2 : 0.995768, exp_rate<=3 : 0.998581, loss: 0.001048, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17434 s, avg_samples: 56.0, ips: 321.20583 samples/s, eta: 7:36:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:16:25] ppocr INFO: epoch: [448/500], global_step: 404200, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.943637, exp_rate<=1 : 0.979535, exp_rate<=2 : 0.995744, exp_rate<=3 : 0.998569, loss: 0.000840, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18508 s, avg_samples: 56.0, ips: 302.56377 samples/s, eta: 7:36:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:16:44] ppocr INFO: epoch: [448/500], global_step: 404300, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.943886, exp_rate<=1 : 0.979471, exp_rate<=2 : 0.995799, exp_rate<=3 : 0.998581, loss: 0.001480, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18989 s, avg_samples: 56.0, ips: 294.90509 samples/s, eta: 7:35:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:17:03] ppocr INFO: epoch: [448/500], global_step: 404400, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.943082, exp_rate<=1 : 0.979124, exp_rate<=2 : 0.995688, exp_rate<=3 : 0.998532, loss: 0.002019, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19351 s, avg_samples: 56.0, ips: 289.39457 samples/s, eta: 7:35:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:17:22] ppocr INFO: epoch: [448/500], global_step: 404500, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.942047, exp_rate<=1 : 0.978757, exp_rate<=2 : 0.995645, exp_rate<=3 : 0.998487, loss: 0.001335, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18907 s, avg_samples: 56.0, ips: 296.18104 samples/s, eta: 7:35:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:17:41] ppocr INFO: epoch: [448/500], global_step: 404600, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.942074, exp_rate<=1 : 0.978688, exp_rate<=2 : 0.995581, exp_rate<=3 : 0.998426, loss: 0.001721, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18811 s, avg_samples: 56.0, ips: 297.70326 samples/s, eta: 7:34:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:17:59] ppocr INFO: epoch: [448/500], global_step: 404700, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.942368, exp_rate<=1 : 0.978640, exp_rate<=2 : 0.995522, exp_rate<=3 : 0.998393, loss: 0.001017, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17812 s, avg_samples: 56.0, ips: 314.40109 samples/s, eta: 7:34:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:18:17] ppocr INFO: epoch: [448/500], global_step: 404800, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.942525, exp_rate<=1 : 0.978626, exp_rate<=2 : 0.995526, exp_rate<=3 : 0.998366, loss: 0.001015, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18444 s, avg_samples: 56.0, ips: 303.62591 samples/s, eta: 7:34:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:18:36] ppocr INFO: epoch: [448/500], global_step: 404900, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.942259, exp_rate<=1 : 0.978439, exp_rate<=2 : 0.995495, exp_rate<=3 : 0.998359, loss: 0.001832, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19041 s, avg_samples: 56.0, ips: 294.10445 samples/s, eta: 7:34:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:18:53] ppocr INFO: epoch: [448/500], global_step: 405000, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.942490, exp_rate<=1 : 0.978444, exp_rate<=2 : 0.995511, exp_rate<=3 : 0.998354, loss: 0.001898, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17360 s, avg_samples: 56.0, ips: 322.57397 samples/s, eta: 7:33:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:19:11] ppocr INFO: epoch: [448/500], global_step: 405100, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.942096, exp_rate<=1 : 0.978334, exp_rate<=2 : 0.995474, exp_rate<=3 : 0.998336, loss: 0.001865, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18009 s, avg_samples: 56.0, ips: 310.95862 samples/s, eta: 7:33:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:19:31] ppocr INFO: epoch: [448/500], global_step: 405200, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.941684, exp_rate<=1 : 0.978196, exp_rate<=2 : 0.995377, exp_rate<=3 : 0.998315, loss: 0.001752, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19489 s, avg_samples: 56.0, ips: 287.34543 samples/s, eta: 7:33:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:19:51] ppocr INFO: epoch: [448/500], global_step: 405300, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.941259, exp_rate<=1 : 0.978092, exp_rate<=2 : 0.995371, exp_rate<=3 : 0.998331, loss: 0.001679, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19749 s, avg_samples: 56.0, ips: 283.56514 samples/s, eta: 7:32:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:20:09] ppocr INFO: epoch: [448/500], global_step: 405400, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.940746, exp_rate<=1 : 0.977833, exp_rate<=2 : 0.995305, exp_rate<=3 : 0.998282, loss: 0.001982, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18379 s, avg_samples: 56.0, ips: 304.68899 samples/s, eta: 7:32:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:20:28] ppocr INFO: epoch: [448/500], global_step: 405500, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.940671, exp_rate<=1 : 0.977764, exp_rate<=2 : 0.995303, exp_rate<=3 : 0.998280, loss: 0.001461, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19250 s, avg_samples: 56.0, ips: 290.91587 samples/s, eta: 7:32:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:20:32] ppocr INFO: epoch: [448/500], global_step: 405520, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.940526, exp_rate<=1 : 0.977709, exp_rate<=2 : 0.995319, exp_rate<=3 : 0.998276, loss: 0.001902, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03479 s, avg_samples: 11.2, ips: 321.92128 samples/s, eta: 7:32:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:20:34] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 00:20:50] ppocr INFO: epoch: [449/500], global_step: 405600, lr: 0.000100, edit distance: 0.000834, exp_rate: 0.954345, exp_rate<=1 : 0.981130, exp_rate<=2 : 0.995694, exp_rate<=3 : 0.998733, loss: 0.000822, avg_reader_cost: 0.02724 s, avg_batch_cost: 0.17777 s, avg_samples: 44.8, ips: 252.00436 samples/s, eta: 7:31:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:21:09] ppocr INFO: epoch: [449/500], global_step: 405700, lr: 0.000100, edit distance: 0.000853, exp_rate: 0.954654, exp_rate<=1 : 0.982969, exp_rate<=2 : 0.996512, exp_rate<=3 : 0.999132, loss: 0.001295, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19543 s, avg_samples: 56.0, ips: 286.54233 samples/s, eta: 7:31:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:21:28] ppocr INFO: epoch: [449/500], global_step: 405800, lr: 0.000100, edit distance: 0.000934, exp_rate: 0.952934, exp_rate<=1 : 0.981918, exp_rate<=2 : 0.995958, exp_rate<=3 : 0.998471, loss: 0.001108, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18929 s, avg_samples: 56.0, ips: 295.84196 samples/s, eta: 7:31:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:21:47] ppocr INFO: epoch: [449/500], global_step: 405900, lr: 0.000100, edit distance: 0.001010, exp_rate: 0.951091, exp_rate<=1 : 0.981948, exp_rate<=2 : 0.995910, exp_rate<=3 : 0.998586, loss: 0.001237, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18936 s, avg_samples: 56.0, ips: 295.72647 samples/s, eta: 7:30:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:22:07] ppocr INFO: epoch: [449/500], global_step: 406000, lr: 0.000100, edit distance: 0.001062, exp_rate: 0.946791, exp_rate<=1 : 0.979000, exp_rate<=2 : 0.994654, exp_rate<=3 : 0.997897, loss: 0.001552, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19590 s, avg_samples: 56.0, ips: 285.86539 samples/s, eta: 7:30:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:22:25] ppocr INFO: epoch: [449/500], global_step: 406100, lr: 0.000100, edit distance: 0.001097, exp_rate: 0.945774, exp_rate<=1 : 0.979206, exp_rate<=2 : 0.994655, exp_rate<=3 : 0.997885, loss: 0.001463, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17792 s, avg_samples: 56.0, ips: 314.75519 samples/s, eta: 7:30:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:22:46] ppocr INFO: epoch: [449/500], global_step: 406200, lr: 0.000100, edit distance: 0.001099, exp_rate: 0.945146, exp_rate<=1 : 0.978660, exp_rate<=2 : 0.994342, exp_rate<=3 : 0.997881, loss: 0.000988, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21043 s, avg_samples: 56.0, ips: 266.12676 samples/s, eta: 7:29:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:23:04] ppocr INFO: epoch: [449/500], global_step: 406300, lr: 0.000100, edit distance: 0.001103, exp_rate: 0.945262, exp_rate<=1 : 0.979014, exp_rate<=2 : 0.994690, exp_rate<=3 : 0.998050, loss: 0.001580, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18170 s, avg_samples: 56.0, ips: 308.20127 samples/s, eta: 7:29:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:23:22] ppocr INFO: epoch: [449/500], global_step: 406400, lr: 0.000100, edit distance: 0.001104, exp_rate: 0.945346, exp_rate<=1 : 0.978990, exp_rate<=2 : 0.994782, exp_rate<=3 : 0.998114, loss: 0.001645, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18349 s, avg_samples: 56.0, ips: 305.19570 samples/s, eta: 7:29:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:23:41] ppocr INFO: epoch: [449/500], global_step: 406500, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.944782, exp_rate<=1 : 0.978518, exp_rate<=2 : 0.994608, exp_rate<=3 : 0.998160, loss: 0.001208, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18620 s, avg_samples: 56.0, ips: 300.75775 samples/s, eta: 7:28:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:23:59] ppocr INFO: epoch: [449/500], global_step: 406600, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.945119, exp_rate<=1 : 0.978798, exp_rate<=2 : 0.994762, exp_rate<=3 : 0.998232, loss: 0.001351, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18248 s, avg_samples: 56.0, ips: 306.88575 samples/s, eta: 7:28:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:24:17] ppocr INFO: epoch: [449/500], global_step: 406700, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.945054, exp_rate<=1 : 0.978639, exp_rate<=2 : 0.994655, exp_rate<=3 : 0.998188, loss: 0.001340, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17481 s, avg_samples: 56.0, ips: 320.34899 samples/s, eta: 7:28:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:24:34] ppocr INFO: epoch: [449/500], global_step: 406800, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.945709, exp_rate<=1 : 0.978894, exp_rate<=2 : 0.994799, exp_rate<=3 : 0.998245, loss: 0.002147, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17434 s, avg_samples: 56.0, ips: 321.21009 samples/s, eta: 7:27:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:24:52] ppocr INFO: epoch: [449/500], global_step: 406900, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.945298, exp_rate<=1 : 0.978854, exp_rate<=2 : 0.994903, exp_rate<=3 : 0.998242, loss: 0.001593, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18394 s, avg_samples: 56.0, ips: 304.45290 samples/s, eta: 7:27:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:25:12] ppocr INFO: epoch: [449/500], global_step: 407000, lr: 0.000100, edit distance: 0.001183, exp_rate: 0.944090, exp_rate<=1 : 0.978442, exp_rate<=2 : 0.994831, exp_rate<=3 : 0.998251, loss: 0.001140, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19340 s, avg_samples: 56.0, ips: 289.55822 samples/s, eta: 7:27:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:25:31] ppocr INFO: epoch: [449/500], global_step: 407100, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.943939, exp_rate<=1 : 0.978362, exp_rate<=2 : 0.994849, exp_rate<=3 : 0.998269, loss: 0.001368, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19366 s, avg_samples: 56.0, ips: 289.16712 samples/s, eta: 7:27:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:25:49] ppocr INFO: epoch: [449/500], global_step: 407200, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.943958, exp_rate<=1 : 0.978419, exp_rate<=2 : 0.994904, exp_rate<=3 : 0.998286, loss: 0.001121, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18186 s, avg_samples: 56.0, ips: 307.92431 samples/s, eta: 7:26:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:26:08] ppocr INFO: epoch: [449/500], global_step: 407300, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.943677, exp_rate<=1 : 0.978357, exp_rate<=2 : 0.994754, exp_rate<=3 : 0.998164, loss: 0.001460, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18386 s, avg_samples: 56.0, ips: 304.57802 samples/s, eta: 7:26:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:26:25] ppocr INFO: epoch: [449/500], global_step: 407400, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.944260, exp_rate<=1 : 0.978685, exp_rate<=2 : 0.994871, exp_rate<=3 : 0.998239, loss: 0.000870, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16831 s, avg_samples: 56.0, ips: 332.72201 samples/s, eta: 7:26:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:26:42] ppocr INFO: epoch: [449/500], global_step: 407500, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.944255, exp_rate<=1 : 0.978684, exp_rate<=2 : 0.994866, exp_rate<=3 : 0.998220, loss: 0.001495, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17437 s, avg_samples: 56.0, ips: 321.16053 samples/s, eta: 7:25:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:27:02] ppocr INFO: epoch: [449/500], global_step: 407600, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.944084, exp_rate<=1 : 0.978522, exp_rate<=2 : 0.994827, exp_rate<=3 : 0.998201, loss: 0.001302, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19409 s, avg_samples: 56.0, ips: 288.52313 samples/s, eta: 7:25:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:27:21] ppocr INFO: epoch: [449/500], global_step: 407700, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.943702, exp_rate<=1 : 0.978289, exp_rate<=2 : 0.994826, exp_rate<=3 : 0.998198, loss: 0.001503, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18980 s, avg_samples: 56.0, ips: 295.04835 samples/s, eta: 7:25:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:27:40] ppocr INFO: epoch: [449/500], global_step: 407800, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.943439, exp_rate<=1 : 0.978241, exp_rate<=2 : 0.994841, exp_rate<=3 : 0.998204, loss: 0.001473, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19787 s, avg_samples: 56.0, ips: 283.02017 samples/s, eta: 7:24:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:27:58] ppocr INFO: epoch: [449/500], global_step: 407900, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.943154, exp_rate<=1 : 0.978221, exp_rate<=2 : 0.994850, exp_rate<=3 : 0.998234, loss: 0.001997, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17991 s, avg_samples: 56.0, ips: 311.26611 samples/s, eta: 7:24:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:28:19] ppocr INFO: epoch: [449/500], global_step: 408000, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.942615, exp_rate<=1 : 0.977936, exp_rate<=2 : 0.994729, exp_rate<=3 : 0.998218, loss: 0.001593, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21084 s, avg_samples: 56.0, ips: 265.60783 samples/s, eta: 7:24:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:28:38] ppocr INFO: epoch: [449/500], global_step: 408100, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.942764, exp_rate<=1 : 0.978000, exp_rate<=2 : 0.994804, exp_rate<=3 : 0.998230, loss: 0.000952, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18099 s, avg_samples: 56.0, ips: 309.41415 samples/s, eta: 7:23:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:28:58] ppocr INFO: epoch: [449/500], global_step: 408200, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.942387, exp_rate<=1 : 0.977963, exp_rate<=2 : 0.994753, exp_rate<=3 : 0.998208, loss: 0.001363, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20685 s, avg_samples: 56.0, ips: 270.72775 samples/s, eta: 7:23:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:29:10] ppocr INFO: epoch: [449/500], global_step: 408260, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.942203, exp_rate<=1 : 0.977958, exp_rate<=2 : 0.994770, exp_rate<=3 : 0.998214, loss: 0.001604, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11427 s, avg_samples: 33.6, ips: 294.04303 samples/s, eta: 7:23:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:29:12] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 00:29:20] ppocr INFO: epoch: [450/500], global_step: 408300, lr: 0.000100, edit distance: 0.000911, exp_rate: 0.956466, exp_rate<=1 : 0.984216, exp_rate<=2 : 0.997072, exp_rate<=3 : 1.000000, loss: 0.001101, avg_reader_cost: 0.02631 s, avg_batch_cost: 0.09819 s, avg_samples: 22.4, ips: 228.12069 samples/s, eta: 7:23:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:29:40] ppocr INFO: epoch: [450/500], global_step: 408400, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.946227, exp_rate<=1 : 0.981133, exp_rate<=2 : 0.996305, exp_rate<=3 : 0.998221, loss: 0.001404, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.20853 s, avg_samples: 56.0, ips: 268.55199 samples/s, eta: 7:22:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:30:01] ppocr INFO: epoch: [450/500], global_step: 408500, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.949658, exp_rate<=1 : 0.982030, exp_rate<=2 : 0.996524, exp_rate<=3 : 0.998585, loss: 0.001268, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20397 s, avg_samples: 56.0, ips: 274.54822 samples/s, eta: 7:22:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:30:20] ppocr INFO: epoch: [450/500], global_step: 408600, lr: 0.000100, edit distance: 0.001091, exp_rate: 0.948830, exp_rate<=1 : 0.981927, exp_rate<=2 : 0.996586, exp_rate<=3 : 0.998703, loss: 0.002147, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19314 s, avg_samples: 56.0, ips: 289.94445 samples/s, eta: 7:22:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:30:38] ppocr INFO: epoch: [450/500], global_step: 408700, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.947939, exp_rate<=1 : 0.981160, exp_rate<=2 : 0.996069, exp_rate<=3 : 0.998526, loss: 0.000983, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17802 s, avg_samples: 56.0, ips: 314.56854 samples/s, eta: 7:21:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:30:57] ppocr INFO: epoch: [450/500], global_step: 408800, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.946090, exp_rate<=1 : 0.979837, exp_rate<=2 : 0.995498, exp_rate<=3 : 0.998130, loss: 0.001217, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19084 s, avg_samples: 56.0, ips: 293.44329 samples/s, eta: 7:21:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:31:15] ppocr INFO: epoch: [450/500], global_step: 408900, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.946232, exp_rate<=1 : 0.979941, exp_rate<=2 : 0.995405, exp_rate<=3 : 0.998190, loss: 0.000971, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17786 s, avg_samples: 56.0, ips: 314.85781 samples/s, eta: 7:21:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:31:32] ppocr INFO: epoch: [450/500], global_step: 409000, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.946857, exp_rate<=1 : 0.979915, exp_rate<=2 : 0.995517, exp_rate<=3 : 0.998289, loss: 0.001300, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17097 s, avg_samples: 56.0, ips: 327.54088 samples/s, eta: 7:20:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:31:50] ppocr INFO: epoch: [450/500], global_step: 409100, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.946515, exp_rate<=1 : 0.979509, exp_rate<=2 : 0.995474, exp_rate<=3 : 0.998337, loss: 0.001320, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17457 s, avg_samples: 56.0, ips: 320.79330 samples/s, eta: 7:20:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:32:07] ppocr INFO: epoch: [450/500], global_step: 409200, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.947235, exp_rate<=1 : 0.979721, exp_rate<=2 : 0.995533, exp_rate<=3 : 0.998410, loss: 0.001177, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17551 s, avg_samples: 56.0, ips: 319.06744 samples/s, eta: 7:20:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:32:25] ppocr INFO: epoch: [450/500], global_step: 409300, lr: 0.000100, edit distance: 0.001169, exp_rate: 0.947562, exp_rate<=1 : 0.980027, exp_rate<=2 : 0.995549, exp_rate<=3 : 0.998428, loss: 0.000994, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17834 s, avg_samples: 56.0, ips: 313.99957 samples/s, eta: 7:20:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:32:43] ppocr INFO: epoch: [450/500], global_step: 409400, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.947892, exp_rate<=1 : 0.980126, exp_rate<=2 : 0.995530, exp_rate<=3 : 0.998476, loss: 0.001946, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18056 s, avg_samples: 56.0, ips: 310.14978 samples/s, eta: 7:19:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:33:02] ppocr INFO: epoch: [450/500], global_step: 409500, lr: 0.000100, edit distance: 0.001167, exp_rate: 0.947361, exp_rate<=1 : 0.979659, exp_rate<=2 : 0.995409, exp_rate<=3 : 0.998416, loss: 0.001208, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19450 s, avg_samples: 56.0, ips: 287.91967 samples/s, eta: 7:19:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:33:20] ppocr INFO: epoch: [450/500], global_step: 409600, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.946758, exp_rate<=1 : 0.979451, exp_rate<=2 : 0.995445, exp_rate<=3 : 0.998396, loss: 0.001313, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17189 s, avg_samples: 56.0, ips: 325.79262 samples/s, eta: 7:19:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:33:40] ppocr INFO: epoch: [450/500], global_step: 409700, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.946141, exp_rate<=1 : 0.979484, exp_rate<=2 : 0.995269, exp_rate<=3 : 0.998352, loss: 0.001232, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19924 s, avg_samples: 56.0, ips: 281.07349 samples/s, eta: 7:18:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25019 MB [2024/07/11 00:34:00] ppocr INFO: epoch: [450/500], global_step: 409800, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.945168, exp_rate<=1 : 0.979182, exp_rate<=2 : 0.995105, exp_rate<=3 : 0.998285, loss: 0.001018, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20586 s, avg_samples: 56.0, ips: 272.03267 samples/s, eta: 7:18:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:34:19] ppocr INFO: epoch: [450/500], global_step: 409900, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.945011, exp_rate<=1 : 0.979062, exp_rate<=2 : 0.995123, exp_rate<=3 : 0.998267, loss: 0.001423, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18395 s, avg_samples: 56.0, ips: 304.43611 samples/s, eta: 7:18:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:34:37] ppocr INFO: epoch: [450/500], global_step: 410000, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.944706, exp_rate<=1 : 0.978954, exp_rate<=2 : 0.995126, exp_rate<=3 : 0.998255, loss: 0.002020, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18780 s, avg_samples: 56.0, ips: 298.19424 samples/s, eta: 7:17:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:34:57] ppocr INFO: epoch: [450/500], global_step: 410100, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.944260, exp_rate<=1 : 0.978637, exp_rate<=2 : 0.995091, exp_rate<=3 : 0.998253, loss: 0.001563, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19343 s, avg_samples: 56.0, ips: 289.50567 samples/s, eta: 7:17:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:35:17] ppocr INFO: epoch: [450/500], global_step: 410200, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.943492, exp_rate<=1 : 0.978380, exp_rate<=2 : 0.995081, exp_rate<=3 : 0.998254, loss: 0.000977, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20224 s, avg_samples: 56.0, ips: 276.90143 samples/s, eta: 7:17:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:35:36] ppocr INFO: epoch: [450/500], global_step: 410300, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.943232, exp_rate<=1 : 0.978316, exp_rate<=2 : 0.995105, exp_rate<=3 : 0.998259, loss: 0.001499, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19213 s, avg_samples: 56.0, ips: 291.46231 samples/s, eta: 7:16:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:35:55] ppocr INFO: epoch: [450/500], global_step: 410400, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942804, exp_rate<=1 : 0.978136, exp_rate<=2 : 0.995041, exp_rate<=3 : 0.998175, loss: 0.001519, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19117 s, avg_samples: 56.0, ips: 292.92815 samples/s, eta: 7:16:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:36:13] ppocr INFO: epoch: [450/500], global_step: 410500, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.942794, exp_rate<=1 : 0.978075, exp_rate<=2 : 0.995042, exp_rate<=3 : 0.998214, loss: 0.002040, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17711 s, avg_samples: 56.0, ips: 316.18482 samples/s, eta: 7:16:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:36:31] ppocr INFO: epoch: [450/500], global_step: 410600, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.942840, exp_rate<=1 : 0.978126, exp_rate<=2 : 0.995106, exp_rate<=3 : 0.998245, loss: 0.000905, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17841 s, avg_samples: 56.0, ips: 313.87928 samples/s, eta: 7:15:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:36:50] ppocr INFO: epoch: [450/500], global_step: 410700, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.942646, exp_rate<=1 : 0.978108, exp_rate<=2 : 0.995098, exp_rate<=3 : 0.998221, loss: 0.001796, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18560 s, avg_samples: 56.0, ips: 301.72375 samples/s, eta: 7:15:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:37:09] ppocr INFO: epoch: [450/500], global_step: 410800, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.941793, exp_rate<=1 : 0.977775, exp_rate<=2 : 0.994998, exp_rate<=3 : 0.998198, loss: 0.001695, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18908 s, avg_samples: 56.0, ips: 296.16993 samples/s, eta: 7:15:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:37:27] ppocr INFO: epoch: [450/500], global_step: 410900, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.941648, exp_rate<=1 : 0.977642, exp_rate<=2 : 0.994956, exp_rate<=3 : 0.998191, loss: 0.001643, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17969 s, avg_samples: 56.0, ips: 311.65513 samples/s, eta: 7:14:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:37:46] ppocr INFO: epoch: [450/500], global_step: 411000, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.941310, exp_rate<=1 : 0.977429, exp_rate<=2 : 0.994937, exp_rate<=3 : 0.998184, loss: 0.001814, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19558 s, avg_samples: 56.0, ips: 286.32811 samples/s, eta: 7:14:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:37:48] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 00:37:49] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_450 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 00:38:09] ppocr INFO: epoch: [451/500], global_step: 411100, lr: 0.000100, edit distance: 0.001099, exp_rate: 0.946903, exp_rate<=1 : 0.981850, exp_rate<=2 : 0.994912, exp_rate<=3 : 0.998304, loss: 0.001375, avg_reader_cost: 0.03234 s, avg_batch_cost: 0.23077 s, avg_samples: 56.0, ips: 242.66074 samples/s, eta: 7:14:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:38:27] ppocr INFO: epoch: [451/500], global_step: 411200, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.950987, exp_rate<=1 : 0.982237, exp_rate<=2 : 0.995630, exp_rate<=3 : 0.998088, loss: 0.000862, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17846 s, avg_samples: 56.0, ips: 313.78938 samples/s, eta: 7:13:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:38:46] ppocr INFO: epoch: [451/500], global_step: 411300, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.947035, exp_rate<=1 : 0.980104, exp_rate<=2 : 0.995191, exp_rate<=3 : 0.998161, loss: 0.001598, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18985 s, avg_samples: 56.0, ips: 294.96692 samples/s, eta: 7:13:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:39:04] ppocr INFO: epoch: [451/500], global_step: 411400, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.948957, exp_rate<=1 : 0.980350, exp_rate<=2 : 0.995103, exp_rate<=3 : 0.998162, loss: 0.000935, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17731 s, avg_samples: 56.0, ips: 315.83051 samples/s, eta: 7:13:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:39:23] ppocr INFO: epoch: [451/500], global_step: 411500, lr: 0.000100, edit distance: 0.001126, exp_rate: 0.948521, exp_rate<=1 : 0.979960, exp_rate<=2 : 0.995304, exp_rate<=3 : 0.998253, loss: 0.001163, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18727 s, avg_samples: 56.0, ips: 299.03144 samples/s, eta: 7:13:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:39:41] ppocr INFO: epoch: [451/500], global_step: 411600, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.947834, exp_rate<=1 : 0.979384, exp_rate<=2 : 0.994969, exp_rate<=3 : 0.998051, loss: 0.001164, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18068 s, avg_samples: 56.0, ips: 309.94359 samples/s, eta: 7:12:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:40:00] ppocr INFO: epoch: [451/500], global_step: 411700, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.945670, exp_rate<=1 : 0.979039, exp_rate<=2 : 0.994944, exp_rate<=3 : 0.997957, loss: 0.001611, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18942 s, avg_samples: 56.0, ips: 295.64511 samples/s, eta: 7:12:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:40:19] ppocr INFO: epoch: [451/500], global_step: 411800, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.944653, exp_rate<=1 : 0.978983, exp_rate<=2 : 0.994990, exp_rate<=3 : 0.997998, loss: 0.001259, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18911 s, avg_samples: 56.0, ips: 296.12033 samples/s, eta: 7:12:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:40:39] ppocr INFO: epoch: [451/500], global_step: 411900, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.944580, exp_rate<=1 : 0.978894, exp_rate<=2 : 0.995045, exp_rate<=3 : 0.998066, loss: 0.001720, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20226 s, avg_samples: 56.0, ips: 276.87649 samples/s, eta: 7:11:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:40:58] ppocr INFO: epoch: [451/500], global_step: 412000, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.943439, exp_rate<=1 : 0.978360, exp_rate<=2 : 0.994950, exp_rate<=3 : 0.998046, loss: 0.001169, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19506 s, avg_samples: 56.0, ips: 287.09249 samples/s, eta: 7:11:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:41:17] ppocr INFO: epoch: [451/500], global_step: 412100, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.944316, exp_rate<=1 : 0.978758, exp_rate<=2 : 0.995127, exp_rate<=3 : 0.998181, loss: 0.001731, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18332 s, avg_samples: 56.0, ips: 305.48225 samples/s, eta: 7:11:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:41:35] ppocr INFO: epoch: [451/500], global_step: 412200, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.945133, exp_rate<=1 : 0.979383, exp_rate<=2 : 0.995282, exp_rate<=3 : 0.998226, loss: 0.000914, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18194 s, avg_samples: 56.0, ips: 307.79486 samples/s, eta: 7:10:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:41:54] ppocr INFO: epoch: [451/500], global_step: 412300, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.944447, exp_rate<=1 : 0.979129, exp_rate<=2 : 0.995173, exp_rate<=3 : 0.998133, loss: 0.001879, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19381 s, avg_samples: 56.0, ips: 288.93647 samples/s, eta: 7:10:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:42:12] ppocr INFO: epoch: [451/500], global_step: 412400, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.944633, exp_rate<=1 : 0.979142, exp_rate<=2 : 0.995285, exp_rate<=3 : 0.998215, loss: 0.001141, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18163 s, avg_samples: 56.0, ips: 308.32018 samples/s, eta: 7:10:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:42:30] ppocr INFO: epoch: [451/500], global_step: 412500, lr: 0.000100, edit distance: 0.001183, exp_rate: 0.944743, exp_rate<=1 : 0.979316, exp_rate<=2 : 0.995246, exp_rate<=3 : 0.998284, loss: 0.001639, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17204 s, avg_samples: 56.0, ips: 325.50838 samples/s, eta: 7:09:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:42:49] ppocr INFO: epoch: [451/500], global_step: 412600, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.944643, exp_rate<=1 : 0.979329, exp_rate<=2 : 0.995207, exp_rate<=3 : 0.998295, loss: 0.001850, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19406 s, avg_samples: 56.0, ips: 288.57734 samples/s, eta: 7:09:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:43:08] ppocr INFO: epoch: [451/500], global_step: 412700, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.943820, exp_rate<=1 : 0.978982, exp_rate<=2 : 0.995076, exp_rate<=3 : 0.998255, loss: 0.001179, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18682 s, avg_samples: 56.0, ips: 299.76124 samples/s, eta: 7:09:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:43:26] ppocr INFO: epoch: [451/500], global_step: 412800, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.943999, exp_rate<=1 : 0.979141, exp_rate<=2 : 0.995149, exp_rate<=3 : 0.998275, loss: 0.001928, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18518 s, avg_samples: 56.0, ips: 302.41582 samples/s, eta: 7:08:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:43:45] ppocr INFO: epoch: [451/500], global_step: 412900, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.943149, exp_rate<=1 : 0.978784, exp_rate<=2 : 0.995023, exp_rate<=3 : 0.998200, loss: 0.002140, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18703 s, avg_samples: 56.0, ips: 299.42127 samples/s, eta: 7:08:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:44:03] ppocr INFO: epoch: [451/500], global_step: 413000, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.942865, exp_rate<=1 : 0.978695, exp_rate<=2 : 0.995014, exp_rate<=3 : 0.998179, loss: 0.001437, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18205 s, avg_samples: 56.0, ips: 307.60669 samples/s, eta: 7:08:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:44:22] ppocr INFO: epoch: [451/500], global_step: 413100, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.942376, exp_rate<=1 : 0.978338, exp_rate<=2 : 0.994889, exp_rate<=3 : 0.998086, loss: 0.001916, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18722 s, avg_samples: 56.0, ips: 299.12124 samples/s, eta: 7:07:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:44:42] ppocr INFO: epoch: [451/500], global_step: 413200, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.941903, exp_rate<=1 : 0.978215, exp_rate<=2 : 0.994880, exp_rate<=3 : 0.998094, loss: 0.001380, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19944 s, avg_samples: 56.0, ips: 280.78363 samples/s, eta: 7:07:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:45:01] ppocr INFO: epoch: [451/500], global_step: 413300, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.941683, exp_rate<=1 : 0.978046, exp_rate<=2 : 0.994833, exp_rate<=3 : 0.998106, loss: 0.001881, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18639 s, avg_samples: 56.0, ips: 300.44150 samples/s, eta: 7:07:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:45:20] ppocr INFO: epoch: [451/500], global_step: 413400, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.941284, exp_rate<=1 : 0.977836, exp_rate<=2 : 0.994745, exp_rate<=3 : 0.998055, loss: 0.001375, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19077 s, avg_samples: 56.0, ips: 293.54003 samples/s, eta: 7:06:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:45:39] ppocr INFO: epoch: [451/500], global_step: 413500, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.940890, exp_rate<=1 : 0.977692, exp_rate<=2 : 0.994746, exp_rate<=3 : 0.998056, loss: 0.001732, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19332 s, avg_samples: 56.0, ips: 289.67191 samples/s, eta: 7:06:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:45:59] ppocr INFO: epoch: [451/500], global_step: 413600, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.940769, exp_rate<=1 : 0.977696, exp_rate<=2 : 0.994820, exp_rate<=3 : 0.998070, loss: 0.001719, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19546 s, avg_samples: 56.0, ips: 286.49860 samples/s, eta: 7:06:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:46:17] ppocr INFO: epoch: [451/500], global_step: 413700, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.940677, exp_rate<=1 : 0.977671, exp_rate<=2 : 0.994852, exp_rate<=3 : 0.998089, loss: 0.001532, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17811 s, avg_samples: 56.0, ips: 314.40884 samples/s, eta: 7:06:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:46:24] ppocr INFO: epoch: [451/500], global_step: 413740, lr: 0.000100, edit distance: 0.001315, exp_rate: 0.940949, exp_rate<=1 : 0.977796, exp_rate<=2 : 0.994869, exp_rate<=3 : 0.998110, loss: 0.001323, avg_reader_cost: 0.00005 s, avg_batch_cost: 0.07130 s, avg_samples: 22.4, ips: 314.16121 samples/s, eta: 7:05:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:46:26] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 00:46:36] ppocr INFO: epoch: [452/500], global_step: 413800, lr: 0.000100, edit distance: 0.000808, exp_rate: 0.965033, exp_rate<=1 : 0.988685, exp_rate<=2 : 0.998639, exp_rate<=3 : 0.999197, loss: 0.000621, avg_reader_cost: 0.02506 s, avg_batch_cost: 0.12615 s, avg_samples: 33.6, ips: 266.34487 samples/s, eta: 7:05:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:46:56] ppocr INFO: epoch: [452/500], global_step: 413900, lr: 0.000100, edit distance: 0.000934, exp_rate: 0.953641, exp_rate<=1 : 0.984953, exp_rate<=2 : 0.996663, exp_rate<=3 : 0.998829, loss: 0.001431, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19949 s, avg_samples: 56.0, ips: 280.71865 samples/s, eta: 7:05:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:47:14] ppocr INFO: epoch: [452/500], global_step: 414000, lr: 0.000100, edit distance: 0.000999, exp_rate: 0.953401, exp_rate<=1 : 0.983166, exp_rate<=2 : 0.996337, exp_rate<=3 : 0.998730, loss: 0.001260, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17678 s, avg_samples: 56.0, ips: 316.77474 samples/s, eta: 7:05:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:47:32] ppocr INFO: epoch: [452/500], global_step: 414100, lr: 0.000100, edit distance: 0.001065, exp_rate: 0.950906, exp_rate<=1 : 0.982401, exp_rate<=2 : 0.995746, exp_rate<=3 : 0.998675, loss: 0.000964, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18402 s, avg_samples: 56.0, ips: 304.30831 samples/s, eta: 7:04:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:47:51] ppocr INFO: epoch: [452/500], global_step: 414200, lr: 0.000100, edit distance: 0.001068, exp_rate: 0.950670, exp_rate<=1 : 0.982341, exp_rate<=2 : 0.995988, exp_rate<=3 : 0.998806, loss: 0.001643, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19072 s, avg_samples: 56.0, ips: 293.62025 samples/s, eta: 7:04:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:48:11] ppocr INFO: epoch: [452/500], global_step: 414300, lr: 0.000100, edit distance: 0.001060, exp_rate: 0.950243, exp_rate<=1 : 0.981982, exp_rate<=2 : 0.995879, exp_rate<=3 : 0.998713, loss: 0.000810, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19053 s, avg_samples: 56.0, ips: 293.91571 samples/s, eta: 7:04:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:48:29] ppocr INFO: epoch: [452/500], global_step: 414400, lr: 0.000100, edit distance: 0.001062, exp_rate: 0.950585, exp_rate<=1 : 0.981924, exp_rate<=2 : 0.995914, exp_rate<=3 : 0.998733, loss: 0.001526, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18524 s, avg_samples: 56.0, ips: 302.31287 samples/s, eta: 7:03:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:48:48] ppocr INFO: epoch: [452/500], global_step: 414500, lr: 0.000100, edit distance: 0.001079, exp_rate: 0.950026, exp_rate<=1 : 0.981738, exp_rate<=2 : 0.995955, exp_rate<=3 : 0.998640, loss: 0.001309, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18989 s, avg_samples: 56.0, ips: 294.90839 samples/s, eta: 7:03:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:49:07] ppocr INFO: epoch: [452/500], global_step: 414600, lr: 0.000100, edit distance: 0.001035, exp_rate: 0.951286, exp_rate<=1 : 0.982058, exp_rate<=2 : 0.996183, exp_rate<=3 : 0.998779, loss: 0.000978, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18580 s, avg_samples: 56.0, ips: 301.39813 samples/s, eta: 7:03:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:49:26] ppocr INFO: epoch: [452/500], global_step: 414700, lr: 0.000100, edit distance: 0.001032, exp_rate: 0.951141, exp_rate<=1 : 0.981918, exp_rate<=2 : 0.996143, exp_rate<=3 : 0.998741, loss: 0.000905, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18994 s, avg_samples: 56.0, ips: 294.83769 samples/s, eta: 7:02:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:49:45] ppocr INFO: epoch: [452/500], global_step: 414800, lr: 0.000100, edit distance: 0.001020, exp_rate: 0.950863, exp_rate<=1 : 0.981939, exp_rate<=2 : 0.996137, exp_rate<=3 : 0.998700, loss: 0.001098, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19207 s, avg_samples: 56.0, ips: 291.55717 samples/s, eta: 7:02:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:50:04] ppocr INFO: epoch: [452/500], global_step: 414900, lr: 0.000100, edit distance: 0.001037, exp_rate: 0.949812, exp_rate<=1 : 0.981300, exp_rate<=2 : 0.995929, exp_rate<=3 : 0.998702, loss: 0.001313, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18818 s, avg_samples: 56.0, ips: 297.58496 samples/s, eta: 7:02:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:50:22] ppocr INFO: epoch: [452/500], global_step: 415000, lr: 0.000100, edit distance: 0.001072, exp_rate: 0.948882, exp_rate<=1 : 0.980824, exp_rate<=2 : 0.995773, exp_rate<=3 : 0.998621, loss: 0.001578, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18164 s, avg_samples: 56.0, ips: 308.29436 samples/s, eta: 7:01:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:50:40] ppocr INFO: epoch: [452/500], global_step: 415100, lr: 0.000100, edit distance: 0.001089, exp_rate: 0.948159, exp_rate<=1 : 0.980607, exp_rate<=2 : 0.995666, exp_rate<=3 : 0.998592, loss: 0.000972, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18060 s, avg_samples: 56.0, ips: 310.07211 samples/s, eta: 7:01:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:50:59] ppocr INFO: epoch: [452/500], global_step: 415200, lr: 0.000100, edit distance: 0.001102, exp_rate: 0.947986, exp_rate<=1 : 0.980455, exp_rate<=2 : 0.995599, exp_rate<=3 : 0.998622, loss: 0.002283, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18683 s, avg_samples: 56.0, ips: 299.74399 samples/s, eta: 7:01:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:51:18] ppocr INFO: epoch: [452/500], global_step: 415300, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.947083, exp_rate<=1 : 0.980168, exp_rate<=2 : 0.995550, exp_rate<=3 : 0.998605, loss: 0.001763, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19210 s, avg_samples: 56.0, ips: 291.51005 samples/s, eta: 7:00:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:51:38] ppocr INFO: epoch: [452/500], global_step: 415400, lr: 0.000100, edit distance: 0.001134, exp_rate: 0.946288, exp_rate<=1 : 0.979764, exp_rate<=2 : 0.995430, exp_rate<=3 : 0.998556, loss: 0.001917, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19886 s, avg_samples: 56.0, ips: 281.60511 samples/s, eta: 7:00:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:51:57] ppocr INFO: epoch: [452/500], global_step: 415500, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.945556, exp_rate<=1 : 0.979645, exp_rate<=2 : 0.995411, exp_rate<=3 : 0.998541, loss: 0.001604, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18804 s, avg_samples: 56.0, ips: 297.81612 samples/s, eta: 7:00:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:52:17] ppocr INFO: epoch: [452/500], global_step: 415600, lr: 0.000100, edit distance: 0.001161, exp_rate: 0.944964, exp_rate<=1 : 0.979390, exp_rate<=2 : 0.995384, exp_rate<=3 : 0.998559, loss: 0.001539, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19805 s, avg_samples: 56.0, ips: 282.75444 samples/s, eta: 6:59:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:52:36] ppocr INFO: epoch: [452/500], global_step: 415700, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.944335, exp_rate<=1 : 0.978975, exp_rate<=2 : 0.995235, exp_rate<=3 : 0.998482, loss: 0.001461, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19538 s, avg_samples: 56.0, ips: 286.62124 samples/s, eta: 6:59:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:52:53] ppocr INFO: epoch: [452/500], global_step: 415800, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.944253, exp_rate<=1 : 0.978946, exp_rate<=2 : 0.995236, exp_rate<=3 : 0.998510, loss: 0.002009, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17348 s, avg_samples: 56.0, ips: 322.80208 samples/s, eta: 6:59:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:53:13] ppocr INFO: epoch: [452/500], global_step: 415900, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.943515, exp_rate<=1 : 0.978722, exp_rate<=2 : 0.995104, exp_rate<=3 : 0.998459, loss: 0.001760, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19340 s, avg_samples: 56.0, ips: 289.55096 samples/s, eta: 6:59:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:53:32] ppocr INFO: epoch: [452/500], global_step: 416000, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.943173, exp_rate<=1 : 0.978503, exp_rate<=2 : 0.995057, exp_rate<=3 : 0.998419, loss: 0.001492, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19255 s, avg_samples: 56.0, ips: 290.83906 samples/s, eta: 6:58:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:53:51] ppocr INFO: epoch: [452/500], global_step: 416100, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.943100, exp_rate<=1 : 0.978413, exp_rate<=2 : 0.995041, exp_rate<=3 : 0.998405, loss: 0.001500, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18864 s, avg_samples: 56.0, ips: 296.85541 samples/s, eta: 6:58:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:54:10] ppocr INFO: epoch: [452/500], global_step: 416200, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.943208, exp_rate<=1 : 0.978368, exp_rate<=2 : 0.995057, exp_rate<=3 : 0.998368, loss: 0.001779, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18776 s, avg_samples: 56.0, ips: 298.25603 samples/s, eta: 6:58:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:54:28] ppocr INFO: epoch: [452/500], global_step: 416300, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.943205, exp_rate<=1 : 0.978336, exp_rate<=2 : 0.995095, exp_rate<=3 : 0.998390, loss: 0.001285, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17959 s, avg_samples: 56.0, ips: 311.83003 samples/s, eta: 6:57:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:54:47] ppocr INFO: epoch: [452/500], global_step: 416400, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.943053, exp_rate<=1 : 0.978217, exp_rate<=2 : 0.995012, exp_rate<=3 : 0.998397, loss: 0.000987, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19271 s, avg_samples: 56.0, ips: 290.59294 samples/s, eta: 6:57:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:55:02] ppocr INFO: epoch: [452/500], global_step: 416480, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.943023, exp_rate<=1 : 0.978253, exp_rate<=2 : 0.995022, exp_rate<=3 : 0.998402, loss: 0.000749, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14996 s, avg_samples: 44.8, ips: 298.74235 samples/s, eta: 6:57:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:55:04] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 00:55:09] ppocr INFO: epoch: [453/500], global_step: 416500, lr: 0.000100, edit distance: 0.000762, exp_rate: 0.955357, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.996301, exp_rate<=3 : 0.998847, loss: 0.001017, avg_reader_cost: 0.02602 s, avg_batch_cost: 0.06554 s, avg_samples: 11.2, ips: 170.89335 samples/s, eta: 6:57:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:55:27] ppocr INFO: epoch: [453/500], global_step: 416600, lr: 0.000100, edit distance: 0.000797, exp_rate: 0.955357, exp_rate<=1 : 0.984991, exp_rate<=2 : 0.996929, exp_rate<=3 : 0.999354, loss: 0.001124, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18208 s, avg_samples: 56.0, ips: 307.55394 samples/s, eta: 6:56:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:55:46] ppocr INFO: epoch: [453/500], global_step: 416700, lr: 0.000100, edit distance: 0.000872, exp_rate: 0.953397, exp_rate<=1 : 0.982658, exp_rate<=2 : 0.996216, exp_rate<=3 : 0.999067, loss: 0.000951, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19094 s, avg_samples: 56.0, ips: 293.28854 samples/s, eta: 6:56:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:56:05] ppocr INFO: epoch: [453/500], global_step: 416800, lr: 0.000100, edit distance: 0.000923, exp_rate: 0.952503, exp_rate<=1 : 0.981971, exp_rate<=2 : 0.996377, exp_rate<=3 : 0.999022, loss: 0.001301, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18749 s, avg_samples: 56.0, ips: 298.68449 samples/s, eta: 6:56:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:56:25] ppocr INFO: epoch: [453/500], global_step: 416900, lr: 0.000100, edit distance: 0.000981, exp_rate: 0.949169, exp_rate<=1 : 0.980325, exp_rate<=2 : 0.995979, exp_rate<=3 : 0.998771, loss: 0.001918, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20157 s, avg_samples: 56.0, ips: 277.81909 samples/s, eta: 6:55:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:56:43] ppocr INFO: epoch: [453/500], global_step: 417000, lr: 0.000100, edit distance: 0.001029, exp_rate: 0.947679, exp_rate<=1 : 0.980031, exp_rate<=2 : 0.995925, exp_rate<=3 : 0.998679, loss: 0.001636, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18576 s, avg_samples: 56.0, ips: 301.45639 samples/s, eta: 6:55:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:57:01] ppocr INFO: epoch: [453/500], global_step: 417100, lr: 0.000100, edit distance: 0.001048, exp_rate: 0.947896, exp_rate<=1 : 0.980177, exp_rate<=2 : 0.996142, exp_rate<=3 : 0.998812, loss: 0.001338, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17882 s, avg_samples: 56.0, ips: 313.17107 samples/s, eta: 6:55:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:57:19] ppocr INFO: epoch: [453/500], global_step: 417200, lr: 0.000100, edit distance: 0.001073, exp_rate: 0.947119, exp_rate<=1 : 0.980358, exp_rate<=2 : 0.996004, exp_rate<=3 : 0.998585, loss: 0.001739, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17786 s, avg_samples: 56.0, ips: 314.85007 samples/s, eta: 6:54:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:57:38] ppocr INFO: epoch: [453/500], global_step: 417300, lr: 0.000100, edit distance: 0.001072, exp_rate: 0.947387, exp_rate<=1 : 0.980555, exp_rate<=2 : 0.996100, exp_rate<=3 : 0.998678, loss: 0.001410, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18377 s, avg_samples: 56.0, ips: 304.73618 samples/s, eta: 6:54:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:57:57] ppocr INFO: epoch: [453/500], global_step: 417400, lr: 0.000100, edit distance: 0.001073, exp_rate: 0.946799, exp_rate<=1 : 0.980201, exp_rate<=2 : 0.995927, exp_rate<=3 : 0.998653, loss: 0.000872, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18995 s, avg_samples: 56.0, ips: 294.81093 samples/s, eta: 6:54:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:58:15] ppocr INFO: epoch: [453/500], global_step: 417500, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.946790, exp_rate<=1 : 0.980127, exp_rate<=2 : 0.995793, exp_rate<=3 : 0.998675, loss: 0.000955, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18479 s, avg_samples: 56.0, ips: 303.05079 samples/s, eta: 6:53:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:58:34] ppocr INFO: epoch: [453/500], global_step: 417600, lr: 0.000100, edit distance: 0.001109, exp_rate: 0.946300, exp_rate<=1 : 0.979835, exp_rate<=2 : 0.995612, exp_rate<=3 : 0.998622, loss: 0.001765, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18910 s, avg_samples: 56.0, ips: 296.13799 samples/s, eta: 6:53:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:58:53] ppocr INFO: epoch: [453/500], global_step: 417700, lr: 0.000100, edit distance: 0.001133, exp_rate: 0.944394, exp_rate<=1 : 0.979085, exp_rate<=2 : 0.995510, exp_rate<=3 : 0.998578, loss: 0.002166, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19202 s, avg_samples: 56.0, ips: 291.63298 samples/s, eta: 6:53:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:59:13] ppocr INFO: epoch: [453/500], global_step: 417800, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.943697, exp_rate<=1 : 0.978639, exp_rate<=2 : 0.995278, exp_rate<=3 : 0.998423, loss: 0.001433, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19719 s, avg_samples: 56.0, ips: 283.99544 samples/s, eta: 6:52:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:59:31] ppocr INFO: epoch: [453/500], global_step: 417900, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.943702, exp_rate<=1 : 0.978609, exp_rate<=2 : 0.995155, exp_rate<=3 : 0.998374, loss: 0.001261, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18109 s, avg_samples: 56.0, ips: 309.23599 samples/s, eta: 6:52:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 00:59:51] ppocr INFO: epoch: [453/500], global_step: 418000, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.943464, exp_rate<=1 : 0.978488, exp_rate<=2 : 0.995036, exp_rate<=3 : 0.998327, loss: 0.001615, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20260 s, avg_samples: 56.0, ips: 276.40927 samples/s, eta: 6:52:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:00:10] ppocr INFO: epoch: [453/500], global_step: 418100, lr: 0.000100, edit distance: 0.001144, exp_rate: 0.943252, exp_rate<=1 : 0.978546, exp_rate<=2 : 0.995085, exp_rate<=3 : 0.998361, loss: 0.001354, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18537 s, avg_samples: 56.0, ips: 302.10008 samples/s, eta: 6:52:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:00:28] ppocr INFO: epoch: [453/500], global_step: 418200, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.943147, exp_rate<=1 : 0.978533, exp_rate<=2 : 0.995103, exp_rate<=3 : 0.998373, loss: 0.001405, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18325 s, avg_samples: 56.0, ips: 305.58729 samples/s, eta: 6:51:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:00:48] ppocr INFO: epoch: [453/500], global_step: 418300, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.942622, exp_rate<=1 : 0.978294, exp_rate<=2 : 0.995030, exp_rate<=3 : 0.998370, loss: 0.001620, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19948 s, avg_samples: 56.0, ips: 280.73630 samples/s, eta: 6:51:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:01:08] ppocr INFO: epoch: [453/500], global_step: 418400, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.942233, exp_rate<=1 : 0.978161, exp_rate<=2 : 0.994958, exp_rate<=3 : 0.998371, loss: 0.001712, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19347 s, avg_samples: 56.0, ips: 289.45532 samples/s, eta: 6:51:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:01:26] ppocr INFO: epoch: [453/500], global_step: 418500, lr: 0.000100, edit distance: 0.001214, exp_rate: 0.941801, exp_rate<=1 : 0.977959, exp_rate<=2 : 0.994892, exp_rate<=3 : 0.998323, loss: 0.001611, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18690 s, avg_samples: 56.0, ips: 299.61994 samples/s, eta: 6:50:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:01:44] ppocr INFO: epoch: [453/500], global_step: 418600, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.941793, exp_rate<=1 : 0.977816, exp_rate<=2 : 0.994827, exp_rate<=3 : 0.998228, loss: 0.000749, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17972 s, avg_samples: 56.0, ips: 311.59037 samples/s, eta: 6:50:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:02:03] ppocr INFO: epoch: [453/500], global_step: 418700, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.941808, exp_rate<=1 : 0.977931, exp_rate<=2 : 0.994808, exp_rate<=3 : 0.998206, loss: 0.001343, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18864 s, avg_samples: 56.0, ips: 296.85940 samples/s, eta: 6:50:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:02:21] ppocr INFO: epoch: [453/500], global_step: 418800, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.941672, exp_rate<=1 : 0.977970, exp_rate<=2 : 0.994805, exp_rate<=3 : 0.998238, loss: 0.000721, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17779 s, avg_samples: 56.0, ips: 314.98019 samples/s, eta: 6:49:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:02:40] ppocr INFO: epoch: [453/500], global_step: 418900, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.941615, exp_rate<=1 : 0.978009, exp_rate<=2 : 0.994812, exp_rate<=3 : 0.998231, loss: 0.001633, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18527 s, avg_samples: 56.0, ips: 302.25543 samples/s, eta: 6:49:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:02:57] ppocr INFO: epoch: [453/500], global_step: 419000, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.941816, exp_rate<=1 : 0.978022, exp_rate<=2 : 0.994889, exp_rate<=3 : 0.998248, loss: 0.001274, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17275 s, avg_samples: 56.0, ips: 324.16938 samples/s, eta: 6:49:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:03:16] ppocr INFO: epoch: [453/500], global_step: 419100, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.941340, exp_rate<=1 : 0.977767, exp_rate<=2 : 0.994828, exp_rate<=3 : 0.998230, loss: 0.001531, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19374 s, avg_samples: 56.0, ips: 289.04726 samples/s, eta: 6:48:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:03:35] ppocr INFO: epoch: [453/500], global_step: 419200, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.940558, exp_rate<=1 : 0.977440, exp_rate<=2 : 0.994768, exp_rate<=3 : 0.998212, loss: 0.002084, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18363 s, avg_samples: 56.0, ips: 304.95747 samples/s, eta: 6:48:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:03:38] ppocr INFO: epoch: [453/500], global_step: 419220, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.940416, exp_rate<=1 : 0.977362, exp_rate<=2 : 0.994738, exp_rate<=3 : 0.998206, loss: 0.002183, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03639 s, avg_samples: 11.2, ips: 307.79050 samples/s, eta: 6:48:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:03:40] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 01:03:55] ppocr INFO: epoch: [454/500], global_step: 419300, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.946558, exp_rate<=1 : 0.978389, exp_rate<=2 : 0.996454, exp_rate<=3 : 0.998480, loss: 0.001397, avg_reader_cost: 0.02614 s, avg_batch_cost: 0.17063 s, avg_samples: 44.8, ips: 262.55096 samples/s, eta: 6:48:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:04:14] ppocr INFO: epoch: [454/500], global_step: 419400, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.947640, exp_rate<=1 : 0.979681, exp_rate<=2 : 0.996439, exp_rate<=3 : 0.998953, loss: 0.001224, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18548 s, avg_samples: 56.0, ips: 301.91132 samples/s, eta: 6:47:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:04:32] ppocr INFO: epoch: [454/500], global_step: 419500, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.948179, exp_rate<=1 : 0.980924, exp_rate<=2 : 0.996933, exp_rate<=3 : 0.999010, loss: 0.001661, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17949 s, avg_samples: 56.0, ips: 311.99822 samples/s, eta: 6:47:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:04:50] ppocr INFO: epoch: [454/500], global_step: 419600, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.946740, exp_rate<=1 : 0.981162, exp_rate<=2 : 0.996788, exp_rate<=3 : 0.998940, loss: 0.001223, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18241 s, avg_samples: 56.0, ips: 306.99288 samples/s, eta: 6:47:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:05:08] ppocr INFO: epoch: [454/500], global_step: 419700, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.947477, exp_rate<=1 : 0.981520, exp_rate<=2 : 0.996795, exp_rate<=3 : 0.998785, loss: 0.001486, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17468 s, avg_samples: 56.0, ips: 320.58902 samples/s, eta: 6:46:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:05:27] ppocr INFO: epoch: [454/500], global_step: 419800, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.946725, exp_rate<=1 : 0.980830, exp_rate<=2 : 0.996306, exp_rate<=3 : 0.998579, loss: 0.001610, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19137 s, avg_samples: 56.0, ips: 292.62006 samples/s, eta: 6:46:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:05:46] ppocr INFO: epoch: [454/500], global_step: 419900, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.944784, exp_rate<=1 : 0.980184, exp_rate<=2 : 0.996008, exp_rate<=3 : 0.998450, loss: 0.001778, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18689 s, avg_samples: 56.0, ips: 299.64867 samples/s, eta: 6:46:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:06:04] ppocr INFO: epoch: [454/500], global_step: 420000, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.945025, exp_rate<=1 : 0.980301, exp_rate<=2 : 0.995977, exp_rate<=3 : 0.998477, loss: 0.001479, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18280 s, avg_samples: 56.0, ips: 306.35252 samples/s, eta: 6:45:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:06:22] ppocr INFO: epoch: [454/500], global_step: 420100, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.944167, exp_rate<=1 : 0.980108, exp_rate<=2 : 0.995734, exp_rate<=3 : 0.998482, loss: 0.001649, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18300 s, avg_samples: 56.0, ips: 306.00901 samples/s, eta: 6:45:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:06:41] ppocr INFO: epoch: [454/500], global_step: 420200, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.944209, exp_rate<=1 : 0.979986, exp_rate<=2 : 0.995864, exp_rate<=3 : 0.998540, loss: 0.001295, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18727 s, avg_samples: 56.0, ips: 299.03977 samples/s, eta: 6:45:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:07:00] ppocr INFO: epoch: [454/500], global_step: 420300, lr: 0.000100, edit distance: 0.001281, exp_rate: 0.943588, exp_rate<=1 : 0.979635, exp_rate<=2 : 0.995534, exp_rate<=3 : 0.998417, loss: 0.001174, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19174 s, avg_samples: 56.0, ips: 292.06609 samples/s, eta: 6:45:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:07:21] ppocr INFO: epoch: [454/500], global_step: 420400, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.942404, exp_rate<=1 : 0.979048, exp_rate<=2 : 0.995426, exp_rate<=3 : 0.998329, loss: 0.001257, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21293 s, avg_samples: 56.0, ips: 262.99579 samples/s, eta: 6:44:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:07:40] ppocr INFO: epoch: [454/500], global_step: 420500, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.942413, exp_rate<=1 : 0.979099, exp_rate<=2 : 0.995481, exp_rate<=3 : 0.998318, loss: 0.001579, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19078 s, avg_samples: 56.0, ips: 293.52813 samples/s, eta: 6:44:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:07:59] ppocr INFO: epoch: [454/500], global_step: 420600, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.942346, exp_rate<=1 : 0.979031, exp_rate<=2 : 0.995348, exp_rate<=3 : 0.998228, loss: 0.001094, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18268 s, avg_samples: 56.0, ips: 306.55069 samples/s, eta: 6:44:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:08:19] ppocr INFO: epoch: [454/500], global_step: 420700, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.942106, exp_rate<=1 : 0.978859, exp_rate<=2 : 0.995266, exp_rate<=3 : 0.998194, loss: 0.001507, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20686 s, avg_samples: 56.0, ips: 270.71908 samples/s, eta: 6:43:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:08:39] ppocr INFO: epoch: [454/500], global_step: 420800, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.941894, exp_rate<=1 : 0.978758, exp_rate<=2 : 0.995285, exp_rate<=3 : 0.998215, loss: 0.001727, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19797 s, avg_samples: 56.0, ips: 282.87273 samples/s, eta: 6:43:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:08:57] ppocr INFO: epoch: [454/500], global_step: 420900, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.942199, exp_rate<=1 : 0.978978, exp_rate<=2 : 0.995362, exp_rate<=3 : 0.998279, loss: 0.001755, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17571 s, avg_samples: 56.0, ips: 318.71358 samples/s, eta: 6:43:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:09:16] ppocr INFO: epoch: [454/500], global_step: 421000, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.941854, exp_rate<=1 : 0.978838, exp_rate<=2 : 0.995367, exp_rate<=3 : 0.998261, loss: 0.001543, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19400 s, avg_samples: 56.0, ips: 288.65567 samples/s, eta: 6:42:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:09:37] ppocr INFO: epoch: [454/500], global_step: 421100, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.941434, exp_rate<=1 : 0.978589, exp_rate<=2 : 0.995366, exp_rate<=3 : 0.998282, loss: 0.002000, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20587 s, avg_samples: 56.0, ips: 272.01622 samples/s, eta: 6:42:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:09:56] ppocr INFO: epoch: [454/500], global_step: 421200, lr: 0.000100, edit distance: 0.001293, exp_rate: 0.941425, exp_rate<=1 : 0.978486, exp_rate<=2 : 0.995354, exp_rate<=3 : 0.998268, loss: 0.001259, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18760 s, avg_samples: 56.0, ips: 298.50253 samples/s, eta: 6:42:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:10:14] ppocr INFO: epoch: [454/500], global_step: 421300, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.941609, exp_rate<=1 : 0.978482, exp_rate<=2 : 0.995331, exp_rate<=3 : 0.998254, loss: 0.001532, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18549 s, avg_samples: 56.0, ips: 301.89680 samples/s, eta: 6:41:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:10:32] ppocr INFO: epoch: [454/500], global_step: 421400, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941573, exp_rate<=1 : 0.978535, exp_rate<=2 : 0.995368, exp_rate<=3 : 0.998289, loss: 0.001364, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17789 s, avg_samples: 56.0, ips: 314.80458 samples/s, eta: 6:41:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:10:51] ppocr INFO: epoch: [454/500], global_step: 421500, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.941912, exp_rate<=1 : 0.978659, exp_rate<=2 : 0.995358, exp_rate<=3 : 0.998293, loss: 0.001556, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18507 s, avg_samples: 56.0, ips: 302.58639 samples/s, eta: 6:41:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:11:10] ppocr INFO: epoch: [454/500], global_step: 421600, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.941713, exp_rate<=1 : 0.978455, exp_rate<=2 : 0.995277, exp_rate<=3 : 0.998258, loss: 0.001422, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19815 s, avg_samples: 56.0, ips: 282.61766 samples/s, eta: 6:40:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:11:28] ppocr INFO: epoch: [454/500], global_step: 421700, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.941888, exp_rate<=1 : 0.978483, exp_rate<=2 : 0.995261, exp_rate<=3 : 0.998232, loss: 0.001417, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17340 s, avg_samples: 56.0, ips: 322.96158 samples/s, eta: 6:40:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:11:45] ppocr INFO: epoch: [454/500], global_step: 421800, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.942017, exp_rate<=1 : 0.978489, exp_rate<=2 : 0.995293, exp_rate<=3 : 0.998256, loss: 0.001144, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17436 s, avg_samples: 56.0, ips: 321.17728 samples/s, eta: 6:40:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:12:03] ppocr INFO: epoch: [454/500], global_step: 421900, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.942400, exp_rate<=1 : 0.978606, exp_rate<=2 : 0.995347, exp_rate<=3 : 0.998278, loss: 0.001732, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17658 s, avg_samples: 56.0, ips: 317.13521 samples/s, eta: 6:39:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:12:14] ppocr INFO: epoch: [454/500], global_step: 421960, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.942197, exp_rate<=1 : 0.978499, exp_rate<=2 : 0.995310, exp_rate<=3 : 0.998269, loss: 0.001480, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11063 s, avg_samples: 33.6, ips: 303.71843 samples/s, eta: 6:39:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:12:16] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 01:12:24] ppocr INFO: epoch: [455/500], global_step: 422000, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.952381, exp_rate<=1 : 0.983947, exp_rate<=2 : 0.996898, exp_rate<=3 : 0.998746, loss: 0.000961, avg_reader_cost: 0.02579 s, avg_batch_cost: 0.09785 s, avg_samples: 22.4, ips: 228.92679 samples/s, eta: 6:39:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:12:43] ppocr INFO: epoch: [455/500], global_step: 422100, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.943706, exp_rate<=1 : 0.979661, exp_rate<=2 : 0.995751, exp_rate<=3 : 0.998221, loss: 0.001019, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18993 s, avg_samples: 56.0, ips: 294.84884 samples/s, eta: 6:39:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:13:02] ppocr INFO: epoch: [455/500], global_step: 422200, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.945900, exp_rate<=1 : 0.980554, exp_rate<=2 : 0.995794, exp_rate<=3 : 0.998195, loss: 0.001719, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18854 s, avg_samples: 56.0, ips: 297.01694 samples/s, eta: 6:38:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:13:20] ppocr INFO: epoch: [455/500], global_step: 422300, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.946913, exp_rate<=1 : 0.980927, exp_rate<=2 : 0.996353, exp_rate<=3 : 0.998406, loss: 0.001034, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18742 s, avg_samples: 56.0, ips: 298.79869 samples/s, eta: 6:38:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:13:40] ppocr INFO: epoch: [455/500], global_step: 422400, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.946510, exp_rate<=1 : 0.980550, exp_rate<=2 : 0.996140, exp_rate<=3 : 0.998360, loss: 0.001374, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19080 s, avg_samples: 56.0, ips: 293.50024 samples/s, eta: 6:38:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:13:58] ppocr INFO: epoch: [455/500], global_step: 422500, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.946579, exp_rate<=1 : 0.980909, exp_rate<=2 : 0.996080, exp_rate<=3 : 0.998399, loss: 0.001117, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18789 s, avg_samples: 56.0, ips: 298.05371 samples/s, eta: 6:38:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:14:17] ppocr INFO: epoch: [455/500], global_step: 422600, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.946316, exp_rate<=1 : 0.980498, exp_rate<=2 : 0.995746, exp_rate<=3 : 0.998325, loss: 0.001483, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18962 s, avg_samples: 56.0, ips: 295.33111 samples/s, eta: 6:37:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:14:36] ppocr INFO: epoch: [455/500], global_step: 422700, lr: 0.000100, edit distance: 0.001206, exp_rate: 0.945769, exp_rate<=1 : 0.980027, exp_rate<=2 : 0.995569, exp_rate<=3 : 0.998454, loss: 0.001300, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18836 s, avg_samples: 56.0, ips: 297.29602 samples/s, eta: 6:37:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:14:56] ppocr INFO: epoch: [455/500], global_step: 422800, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.945280, exp_rate<=1 : 0.979725, exp_rate<=2 : 0.995318, exp_rate<=3 : 0.998303, loss: 0.001336, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19383 s, avg_samples: 56.0, ips: 288.91515 samples/s, eta: 6:37:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:15:15] ppocr INFO: epoch: [455/500], global_step: 422900, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.944822, exp_rate<=1 : 0.979253, exp_rate<=2 : 0.995320, exp_rate<=3 : 0.998367, loss: 0.001253, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18893 s, avg_samples: 56.0, ips: 296.40469 samples/s, eta: 6:36:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:15:34] ppocr INFO: epoch: [455/500], global_step: 423000, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.944782, exp_rate<=1 : 0.979151, exp_rate<=2 : 0.995339, exp_rate<=3 : 0.998458, loss: 0.001852, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19268 s, avg_samples: 56.0, ips: 290.64260 samples/s, eta: 6:36:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:15:53] ppocr INFO: epoch: [455/500], global_step: 423100, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.944396, exp_rate<=1 : 0.979114, exp_rate<=2 : 0.995301, exp_rate<=3 : 0.998405, loss: 0.001297, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19673 s, avg_samples: 56.0, ips: 284.64823 samples/s, eta: 6:36:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:16:12] ppocr INFO: epoch: [455/500], global_step: 423200, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.944738, exp_rate<=1 : 0.979344, exp_rate<=2 : 0.995402, exp_rate<=3 : 0.998442, loss: 0.001330, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18134 s, avg_samples: 56.0, ips: 308.80572 samples/s, eta: 6:35:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:16:30] ppocr INFO: epoch: [455/500], global_step: 423300, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.944512, exp_rate<=1 : 0.979209, exp_rate<=2 : 0.995357, exp_rate<=3 : 0.998408, loss: 0.001317, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18138 s, avg_samples: 56.0, ips: 308.74651 samples/s, eta: 6:35:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:16:49] ppocr INFO: epoch: [455/500], global_step: 423400, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.944135, exp_rate<=1 : 0.978990, exp_rate<=2 : 0.995342, exp_rate<=3 : 0.998352, loss: 0.001239, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19616 s, avg_samples: 56.0, ips: 285.47468 samples/s, eta: 6:35:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:17:07] ppocr INFO: epoch: [455/500], global_step: 423500, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.944480, exp_rate<=1 : 0.979216, exp_rate<=2 : 0.995356, exp_rate<=3 : 0.998402, loss: 0.001244, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17234 s, avg_samples: 56.0, ips: 324.94067 samples/s, eta: 6:34:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:17:24] ppocr INFO: epoch: [455/500], global_step: 423600, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.944441, exp_rate<=1 : 0.979269, exp_rate<=2 : 0.995401, exp_rate<=3 : 0.998406, loss: 0.001614, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17637 s, avg_samples: 56.0, ips: 317.51414 samples/s, eta: 6:34:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:17:43] ppocr INFO: epoch: [455/500], global_step: 423700, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.944164, exp_rate<=1 : 0.979140, exp_rate<=2 : 0.995435, exp_rate<=3 : 0.998407, loss: 0.002101, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18411 s, avg_samples: 56.0, ips: 304.17407 samples/s, eta: 6:34:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:18:02] ppocr INFO: epoch: [455/500], global_step: 423800, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.943063, exp_rate<=1 : 0.978626, exp_rate<=2 : 0.995308, exp_rate<=3 : 0.998334, loss: 0.001203, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19475 s, avg_samples: 56.0, ips: 287.55358 samples/s, eta: 6:33:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:18:21] ppocr INFO: epoch: [455/500], global_step: 423900, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.943243, exp_rate<=1 : 0.978704, exp_rate<=2 : 0.995351, exp_rate<=3 : 0.998323, loss: 0.001691, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18572 s, avg_samples: 56.0, ips: 301.53447 samples/s, eta: 6:33:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:18:39] ppocr INFO: epoch: [455/500], global_step: 424000, lr: 0.000100, edit distance: 0.001282, exp_rate: 0.943185, exp_rate<=1 : 0.978617, exp_rate<=2 : 0.995346, exp_rate<=3 : 0.998288, loss: 0.001336, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18474 s, avg_samples: 56.0, ips: 303.12619 samples/s, eta: 6:33:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:18:58] ppocr INFO: epoch: [455/500], global_step: 424100, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.942875, exp_rate<=1 : 0.978378, exp_rate<=2 : 0.995208, exp_rate<=3 : 0.998235, loss: 0.001298, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18792 s, avg_samples: 56.0, ips: 298.00002 samples/s, eta: 6:32:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:19:18] ppocr INFO: epoch: [455/500], global_step: 424200, lr: 0.000100, edit distance: 0.001294, exp_rate: 0.942683, exp_rate<=1 : 0.978254, exp_rate<=2 : 0.995186, exp_rate<=3 : 0.998263, loss: 0.001639, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19664 s, avg_samples: 56.0, ips: 284.78872 samples/s, eta: 6:32:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:19:38] ppocr INFO: epoch: [455/500], global_step: 424300, lr: 0.000100, edit distance: 0.001305, exp_rate: 0.942367, exp_rate<=1 : 0.978003, exp_rate<=2 : 0.995089, exp_rate<=3 : 0.998248, loss: 0.001351, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20160 s, avg_samples: 56.0, ips: 277.77458 samples/s, eta: 6:32:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:19:57] ppocr INFO: epoch: [455/500], global_step: 424400, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.942326, exp_rate<=1 : 0.978020, exp_rate<=2 : 0.995072, exp_rate<=3 : 0.998229, loss: 0.001552, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18874 s, avg_samples: 56.0, ips: 296.70610 samples/s, eta: 6:31:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:20:15] ppocr INFO: epoch: [455/500], global_step: 424500, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.942209, exp_rate<=1 : 0.978075, exp_rate<=2 : 0.995096, exp_rate<=3 : 0.998264, loss: 0.001540, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18104 s, avg_samples: 56.0, ips: 309.31878 samples/s, eta: 6:31:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:20:33] ppocr INFO: epoch: [455/500], global_step: 424600, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.942116, exp_rate<=1 : 0.978127, exp_rate<=2 : 0.995126, exp_rate<=3 : 0.998261, loss: 0.001531, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17647 s, avg_samples: 56.0, ips: 317.33233 samples/s, eta: 6:31:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:20:51] ppocr INFO: epoch: [455/500], global_step: 424700, lr: 0.000100, edit distance: 0.001313, exp_rate: 0.942056, exp_rate<=1 : 0.978117, exp_rate<=2 : 0.995148, exp_rate<=3 : 0.998280, loss: 0.001544, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18701 s, avg_samples: 56.0, ips: 299.44314 samples/s, eta: 6:31:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:20:54] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 01:20:54] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_455 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 01:21:15] ppocr INFO: epoch: [456/500], global_step: 424800, lr: 0.000100, edit distance: 0.000963, exp_rate: 0.953191, exp_rate<=1 : 0.984275, exp_rate<=2 : 0.995640, exp_rate<=3 : 0.998105, loss: 0.001616, avg_reader_cost: 0.03298 s, avg_batch_cost: 0.23104 s, avg_samples: 56.0, ips: 242.38496 samples/s, eta: 6:30:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:21:32] ppocr INFO: epoch: [456/500], global_step: 424900, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.947503, exp_rate<=1 : 0.981372, exp_rate<=2 : 0.995584, exp_rate<=3 : 0.998006, loss: 0.001215, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17557 s, avg_samples: 56.0, ips: 318.95618 samples/s, eta: 6:30:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:21:50] ppocr INFO: epoch: [456/500], global_step: 425000, lr: 0.000100, edit distance: 0.001144, exp_rate: 0.950953, exp_rate<=1 : 0.982627, exp_rate<=2 : 0.995867, exp_rate<=3 : 0.998365, loss: 0.001682, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18054 s, avg_samples: 56.0, ips: 310.18565 samples/s, eta: 6:30:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:22:08] ppocr INFO: epoch: [456/500], global_step: 425100, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.950123, exp_rate<=1 : 0.981914, exp_rate<=2 : 0.995712, exp_rate<=3 : 0.998385, loss: 0.000823, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18253 s, avg_samples: 56.0, ips: 306.79467 samples/s, eta: 6:29:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:22:26] ppocr INFO: epoch: [456/500], global_step: 425200, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.949112, exp_rate<=1 : 0.981619, exp_rate<=2 : 0.995549, exp_rate<=3 : 0.998474, loss: 0.001266, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.17946 s, avg_samples: 56.0, ips: 312.05311 samples/s, eta: 6:29:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:22:46] ppocr INFO: epoch: [456/500], global_step: 425300, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.947489, exp_rate<=1 : 0.981064, exp_rate<=2 : 0.995282, exp_rate<=3 : 0.998186, loss: 0.001314, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19392 s, avg_samples: 56.0, ips: 288.78288 samples/s, eta: 6:29:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:23:04] ppocr INFO: epoch: [456/500], global_step: 425400, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.947610, exp_rate<=1 : 0.980611, exp_rate<=2 : 0.995383, exp_rate<=3 : 0.998321, loss: 0.001651, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17621 s, avg_samples: 56.0, ips: 317.79746 samples/s, eta: 6:28:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:23:23] ppocr INFO: epoch: [456/500], global_step: 425500, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.946959, exp_rate<=1 : 0.980383, exp_rate<=2 : 0.995324, exp_rate<=3 : 0.998400, loss: 0.001343, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19375 s, avg_samples: 56.0, ips: 289.03079 samples/s, eta: 6:28:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:23:42] ppocr INFO: epoch: [456/500], global_step: 425600, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.945904, exp_rate<=1 : 0.979573, exp_rate<=2 : 0.995129, exp_rate<=3 : 0.998336, loss: 0.000796, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19336 s, avg_samples: 56.0, ips: 289.61259 samples/s, eta: 6:28:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:24:02] ppocr INFO: epoch: [456/500], global_step: 425700, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.945399, exp_rate<=1 : 0.979450, exp_rate<=2 : 0.995156, exp_rate<=3 : 0.998356, loss: 0.001660, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19322 s, avg_samples: 56.0, ips: 289.81779 samples/s, eta: 6:27:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:24:20] ppocr INFO: epoch: [456/500], global_step: 425800, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.945687, exp_rate<=1 : 0.979550, exp_rate<=2 : 0.995218, exp_rate<=3 : 0.998404, loss: 0.001334, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17967 s, avg_samples: 56.0, ips: 311.69013 samples/s, eta: 6:27:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:24:38] ppocr INFO: epoch: [456/500], global_step: 425900, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.943739, exp_rate<=1 : 0.979063, exp_rate<=2 : 0.995027, exp_rate<=3 : 0.998319, loss: 0.001314, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18759 s, avg_samples: 56.0, ips: 298.51884 samples/s, eta: 6:27:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:24:58] ppocr INFO: epoch: [456/500], global_step: 426000, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.942269, exp_rate<=1 : 0.978371, exp_rate<=2 : 0.995025, exp_rate<=3 : 0.998279, loss: 0.001659, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19676 s, avg_samples: 56.0, ips: 284.60650 samples/s, eta: 6:26:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:25:16] ppocr INFO: epoch: [456/500], global_step: 426100, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.941873, exp_rate<=1 : 0.978318, exp_rate<=2 : 0.995057, exp_rate<=3 : 0.998298, loss: 0.002062, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17573 s, avg_samples: 56.0, ips: 318.67548 samples/s, eta: 6:26:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:25:33] ppocr INFO: epoch: [456/500], global_step: 426200, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.942021, exp_rate<=1 : 0.978490, exp_rate<=2 : 0.995090, exp_rate<=3 : 0.998299, loss: 0.001564, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17488 s, avg_samples: 56.0, ips: 320.22465 samples/s, eta: 6:26:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:25:53] ppocr INFO: epoch: [456/500], global_step: 426300, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.942029, exp_rate<=1 : 0.978431, exp_rate<=2 : 0.995020, exp_rate<=3 : 0.998269, loss: 0.001541, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19388 s, avg_samples: 56.0, ips: 288.83167 samples/s, eta: 6:25:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:26:11] ppocr INFO: epoch: [456/500], global_step: 426400, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.942251, exp_rate<=1 : 0.978511, exp_rate<=2 : 0.995088, exp_rate<=3 : 0.998310, loss: 0.001096, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18781 s, avg_samples: 56.0, ips: 298.16697 samples/s, eta: 6:25:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:26:28] ppocr INFO: epoch: [456/500], global_step: 426500, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.942868, exp_rate<=1 : 0.978803, exp_rate<=2 : 0.995178, exp_rate<=3 : 0.998354, loss: 0.001006, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16790 s, avg_samples: 56.0, ips: 333.52724 samples/s, eta: 6:25:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:26:47] ppocr INFO: epoch: [456/500], global_step: 426600, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.942280, exp_rate<=1 : 0.978545, exp_rate<=2 : 0.995090, exp_rate<=3 : 0.998315, loss: 0.002567, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19128 s, avg_samples: 56.0, ips: 292.76373 samples/s, eta: 6:24:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:27:08] ppocr INFO: epoch: [456/500], global_step: 426700, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.941309, exp_rate<=1 : 0.978007, exp_rate<=2 : 0.994963, exp_rate<=3 : 0.998268, loss: 0.001421, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21065 s, avg_samples: 56.0, ips: 265.84894 samples/s, eta: 6:24:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:27:26] ppocr INFO: epoch: [456/500], global_step: 426800, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.941508, exp_rate<=1 : 0.978055, exp_rate<=2 : 0.995002, exp_rate<=3 : 0.998275, loss: 0.001339, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17854 s, avg_samples: 56.0, ips: 313.66325 samples/s, eta: 6:24:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:27:47] ppocr INFO: epoch: [456/500], global_step: 426900, lr: 0.000100, edit distance: 0.001277, exp_rate: 0.941433, exp_rate<=1 : 0.978118, exp_rate<=2 : 0.995031, exp_rate<=3 : 0.998270, loss: 0.002048, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20463 s, avg_samples: 56.0, ips: 273.65949 samples/s, eta: 6:24:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:28:07] ppocr INFO: epoch: [456/500], global_step: 427000, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.940954, exp_rate<=1 : 0.978094, exp_rate<=2 : 0.995017, exp_rate<=3 : 0.998261, loss: 0.001541, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19944 s, avg_samples: 56.0, ips: 280.79020 samples/s, eta: 6:23:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:28:26] ppocr INFO: epoch: [456/500], global_step: 427100, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.940658, exp_rate<=1 : 0.978018, exp_rate<=2 : 0.995036, exp_rate<=3 : 0.998284, loss: 0.001612, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19042 s, avg_samples: 56.0, ips: 294.08607 samples/s, eta: 6:23:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:28:44] ppocr INFO: epoch: [456/500], global_step: 427200, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.940679, exp_rate<=1 : 0.977954, exp_rate<=2 : 0.995005, exp_rate<=3 : 0.998256, loss: 0.001567, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18448 s, avg_samples: 56.0, ips: 303.56138 samples/s, eta: 6:23:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:29:04] ppocr INFO: epoch: [456/500], global_step: 427300, lr: 0.000100, edit distance: 0.001310, exp_rate: 0.940302, exp_rate<=1 : 0.977766, exp_rate<=2 : 0.994918, exp_rate<=3 : 0.998208, loss: 0.000812, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19227 s, avg_samples: 56.0, ips: 291.25227 samples/s, eta: 6:22:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:29:23] ppocr INFO: epoch: [456/500], global_step: 427400, lr: 0.000100, edit distance: 0.001303, exp_rate: 0.940350, exp_rate<=1 : 0.977852, exp_rate<=2 : 0.994943, exp_rate<=3 : 0.998215, loss: 0.001418, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19099 s, avg_samples: 56.0, ips: 293.20681 samples/s, eta: 6:22:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:29:30] ppocr INFO: epoch: [456/500], global_step: 427440, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.940399, exp_rate<=1 : 0.977905, exp_rate<=2 : 0.994975, exp_rate<=3 : 0.998225, loss: 0.001459, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07686 s, avg_samples: 22.4, ips: 291.42837 samples/s, eta: 6:22:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:29:32] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 01:29:43] ppocr INFO: epoch: [457/500], global_step: 427500, lr: 0.000100, edit distance: 0.001106, exp_rate: 0.955171, exp_rate<=1 : 0.984550, exp_rate<=2 : 0.997312, exp_rate<=3 : 0.999293, loss: 0.001000, avg_reader_cost: 0.02581 s, avg_batch_cost: 0.12926 s, avg_samples: 33.6, ips: 259.94833 samples/s, eta: 6:22:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:30:02] ppocr INFO: epoch: [457/500], global_step: 427600, lr: 0.000100, edit distance: 0.000944, exp_rate: 0.954030, exp_rate<=1 : 0.982801, exp_rate<=2 : 0.996203, exp_rate<=3 : 0.998695, loss: 0.000902, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18932 s, avg_samples: 56.0, ips: 295.80018 samples/s, eta: 6:21:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:30:22] ppocr INFO: epoch: [457/500], global_step: 427700, lr: 0.000100, edit distance: 0.000925, exp_rate: 0.952620, exp_rate<=1 : 0.982717, exp_rate<=2 : 0.996306, exp_rate<=3 : 0.998752, loss: 0.001480, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19369 s, avg_samples: 56.0, ips: 289.12369 samples/s, eta: 6:21:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:30:41] ppocr INFO: epoch: [457/500], global_step: 427800, lr: 0.000100, edit distance: 0.000971, exp_rate: 0.949838, exp_rate<=1 : 0.981939, exp_rate<=2 : 0.995926, exp_rate<=3 : 0.998624, loss: 0.000950, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19139 s, avg_samples: 56.0, ips: 292.60208 samples/s, eta: 6:21:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:31:01] ppocr INFO: epoch: [457/500], global_step: 427900, lr: 0.000100, edit distance: 0.000960, exp_rate: 0.950454, exp_rate<=1 : 0.982123, exp_rate<=2 : 0.995756, exp_rate<=3 : 0.998613, loss: 0.001018, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19811 s, avg_samples: 56.0, ips: 282.66796 samples/s, eta: 6:20:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:31:19] ppocr INFO: epoch: [457/500], global_step: 428000, lr: 0.000100, edit distance: 0.000963, exp_rate: 0.949674, exp_rate<=1 : 0.982175, exp_rate<=2 : 0.995853, exp_rate<=3 : 0.998585, loss: 0.001506, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18681 s, avg_samples: 56.0, ips: 299.77098 samples/s, eta: 6:20:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:31:38] ppocr INFO: epoch: [457/500], global_step: 428100, lr: 0.000100, edit distance: 0.000992, exp_rate: 0.949496, exp_rate<=1 : 0.981937, exp_rate<=2 : 0.995833, exp_rate<=3 : 0.998515, loss: 0.001520, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18467 s, avg_samples: 56.0, ips: 303.23662 samples/s, eta: 6:20:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:31:58] ppocr INFO: epoch: [457/500], global_step: 428200, lr: 0.000100, edit distance: 0.000999, exp_rate: 0.949486, exp_rate<=1 : 0.981687, exp_rate<=2 : 0.995717, exp_rate<=3 : 0.998525, loss: 0.000825, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19738 s, avg_samples: 56.0, ips: 283.71186 samples/s, eta: 6:19:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:32:16] ppocr INFO: epoch: [457/500], global_step: 428300, lr: 0.000100, edit distance: 0.000998, exp_rate: 0.950347, exp_rate<=1 : 0.981891, exp_rate<=2 : 0.995822, exp_rate<=3 : 0.998572, loss: 0.001137, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17988 s, avg_samples: 56.0, ips: 311.31120 samples/s, eta: 6:19:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:32:34] ppocr INFO: epoch: [457/500], global_step: 428400, lr: 0.000100, edit distance: 0.001022, exp_rate: 0.950276, exp_rate<=1 : 0.981840, exp_rate<=2 : 0.995842, exp_rate<=3 : 0.998589, loss: 0.001176, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18232 s, avg_samples: 56.0, ips: 307.16064 samples/s, eta: 6:19:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:32:53] ppocr INFO: epoch: [457/500], global_step: 428500, lr: 0.000100, edit distance: 0.001035, exp_rate: 0.949630, exp_rate<=1 : 0.981599, exp_rate<=2 : 0.995814, exp_rate<=3 : 0.998638, loss: 0.001640, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19115 s, avg_samples: 56.0, ips: 292.96689 samples/s, eta: 6:18:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:33:12] ppocr INFO: epoch: [457/500], global_step: 428600, lr: 0.000100, edit distance: 0.001051, exp_rate: 0.948885, exp_rate<=1 : 0.981232, exp_rate<=2 : 0.995790, exp_rate<=3 : 0.998593, loss: 0.001215, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18814 s, avg_samples: 56.0, ips: 297.64604 samples/s, eta: 6:18:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:33:31] ppocr INFO: epoch: [457/500], global_step: 428700, lr: 0.000100, edit distance: 0.001061, exp_rate: 0.948756, exp_rate<=1 : 0.981011, exp_rate<=2 : 0.995759, exp_rate<=3 : 0.998554, loss: 0.001366, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18785 s, avg_samples: 56.0, ips: 298.10984 samples/s, eta: 6:18:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:33:50] ppocr INFO: epoch: [457/500], global_step: 428800, lr: 0.000100, edit distance: 0.001098, exp_rate: 0.947582, exp_rate<=1 : 0.980308, exp_rate<=2 : 0.995733, exp_rate<=3 : 0.998598, loss: 0.001630, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18917 s, avg_samples: 56.0, ips: 296.03542 samples/s, eta: 6:18:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:34:07] ppocr INFO: epoch: [457/500], global_step: 428900, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.947471, exp_rate<=1 : 0.980239, exp_rate<=2 : 0.995694, exp_rate<=3 : 0.998587, loss: 0.001201, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17354 s, avg_samples: 56.0, ips: 322.69206 samples/s, eta: 6:17:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:34:27] ppocr INFO: epoch: [457/500], global_step: 429000, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.945920, exp_rate<=1 : 0.979578, exp_rate<=2 : 0.995448, exp_rate<=3 : 0.998420, loss: 0.001218, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20425 s, avg_samples: 56.0, ips: 274.18051 samples/s, eta: 6:17:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:34:46] ppocr INFO: epoch: [457/500], global_step: 429100, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.944525, exp_rate<=1 : 0.978675, exp_rate<=2 : 0.995143, exp_rate<=3 : 0.998256, loss: 0.001537, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18812 s, avg_samples: 56.0, ips: 297.67818 samples/s, eta: 6:17:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:35:06] ppocr INFO: epoch: [457/500], global_step: 429200, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.943216, exp_rate<=1 : 0.978241, exp_rate<=2 : 0.994902, exp_rate<=3 : 0.998156, loss: 0.001571, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20165 s, avg_samples: 56.0, ips: 277.70347 samples/s, eta: 6:16:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:35:25] ppocr INFO: epoch: [457/500], global_step: 429300, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.942932, exp_rate<=1 : 0.978165, exp_rate<=2 : 0.994892, exp_rate<=3 : 0.998144, loss: 0.001546, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18286 s, avg_samples: 56.0, ips: 306.24860 samples/s, eta: 6:16:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:35:42] ppocr INFO: epoch: [457/500], global_step: 429400, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942868, exp_rate<=1 : 0.978177, exp_rate<=2 : 0.994957, exp_rate<=3 : 0.998196, loss: 0.001801, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17134 s, avg_samples: 56.0, ips: 326.83099 samples/s, eta: 6:16:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:36:00] ppocr INFO: epoch: [457/500], global_step: 429500, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.942727, exp_rate<=1 : 0.978089, exp_rate<=2 : 0.994949, exp_rate<=3 : 0.998171, loss: 0.001375, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18201 s, avg_samples: 56.0, ips: 307.67372 samples/s, eta: 6:15:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:36:19] ppocr INFO: epoch: [457/500], global_step: 429600, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.942520, exp_rate<=1 : 0.978030, exp_rate<=2 : 0.994951, exp_rate<=3 : 0.998181, loss: 0.001112, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18540 s, avg_samples: 56.0, ips: 302.04223 samples/s, eta: 6:15:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:36:38] ppocr INFO: epoch: [457/500], global_step: 429700, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.942188, exp_rate<=1 : 0.977805, exp_rate<=2 : 0.994859, exp_rate<=3 : 0.998149, loss: 0.001558, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19378 s, avg_samples: 56.0, ips: 288.99469 samples/s, eta: 6:15:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:36:58] ppocr INFO: epoch: [457/500], global_step: 429800, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.941766, exp_rate<=1 : 0.977676, exp_rate<=2 : 0.994843, exp_rate<=3 : 0.998179, loss: 0.001595, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20401 s, avg_samples: 56.0, ips: 274.49983 samples/s, eta: 6:14:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:37:16] ppocr INFO: epoch: [457/500], global_step: 429900, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.941810, exp_rate<=1 : 0.977696, exp_rate<=2 : 0.994855, exp_rate<=3 : 0.998222, loss: 0.001731, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17859 s, avg_samples: 56.0, ips: 313.57220 samples/s, eta: 6:14:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:37:37] ppocr INFO: epoch: [457/500], global_step: 430000, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.941918, exp_rate<=1 : 0.977671, exp_rate<=2 : 0.994894, exp_rate<=3 : 0.998272, loss: 0.001907, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20214 s, avg_samples: 56.0, ips: 277.03277 samples/s, eta: 6:14:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:37:55] ppocr INFO: epoch: [457/500], global_step: 430100, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.941729, exp_rate<=1 : 0.977595, exp_rate<=2 : 0.994835, exp_rate<=3 : 0.998248, loss: 0.001535, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18309 s, avg_samples: 56.0, ips: 305.86325 samples/s, eta: 6:13:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:38:11] ppocr INFO: epoch: [457/500], global_step: 430180, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.941797, exp_rate<=1 : 0.977603, exp_rate<=2 : 0.994795, exp_rate<=3 : 0.998267, loss: 0.001503, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16164 s, avg_samples: 44.8, ips: 277.15840 samples/s, eta: 6:13:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:38:14] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 01:38:18] ppocr INFO: epoch: [458/500], global_step: 430200, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.934401, exp_rate<=1 : 0.975818, exp_rate<=2 : 0.997138, exp_rate<=3 : 1.000000, loss: 0.001527, avg_reader_cost: 0.03094 s, avg_batch_cost: 0.06816 s, avg_samples: 11.2, ips: 164.31872 samples/s, eta: 6:13:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:38:34] ppocr INFO: epoch: [458/500], global_step: 430300, lr: 0.000100, edit distance: 0.001057, exp_rate: 0.954013, exp_rate<=1 : 0.983637, exp_rate<=2 : 0.996831, exp_rate<=3 : 0.998707, loss: 0.001161, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16186 s, avg_samples: 56.0, ips: 345.97823 samples/s, eta: 6:13:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:38:52] ppocr INFO: epoch: [458/500], global_step: 430400, lr: 0.000100, edit distance: 0.000984, exp_rate: 0.956948, exp_rate<=1 : 0.984688, exp_rate<=2 : 0.997263, exp_rate<=3 : 0.999019, loss: 0.000746, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17489 s, avg_samples: 56.0, ips: 320.20185 samples/s, eta: 6:12:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:39:09] ppocr INFO: epoch: [458/500], global_step: 430500, lr: 0.000100, edit distance: 0.000991, exp_rate: 0.957223, exp_rate<=1 : 0.984257, exp_rate<=2 : 0.997312, exp_rate<=3 : 0.999080, loss: 0.000798, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17534 s, avg_samples: 56.0, ips: 319.37881 samples/s, eta: 6:12:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:39:27] ppocr INFO: epoch: [458/500], global_step: 430600, lr: 0.000100, edit distance: 0.001080, exp_rate: 0.952535, exp_rate<=1 : 0.982402, exp_rate<=2 : 0.996416, exp_rate<=3 : 0.998541, loss: 0.001584, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18056 s, avg_samples: 56.0, ips: 310.14504 samples/s, eta: 6:12:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:39:46] ppocr INFO: epoch: [458/500], global_step: 430700, lr: 0.000100, edit distance: 0.001074, exp_rate: 0.950985, exp_rate<=1 : 0.981531, exp_rate<=2 : 0.996254, exp_rate<=3 : 0.998569, loss: 0.000821, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18349 s, avg_samples: 56.0, ips: 305.18807 samples/s, eta: 6:11:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:40:05] ppocr INFO: epoch: [458/500], global_step: 430800, lr: 0.000100, edit distance: 0.001103, exp_rate: 0.948900, exp_rate<=1 : 0.980886, exp_rate<=2 : 0.996118, exp_rate<=3 : 0.998547, loss: 0.001503, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19691 s, avg_samples: 56.0, ips: 284.39112 samples/s, eta: 6:11:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:40:25] ppocr INFO: epoch: [458/500], global_step: 430900, lr: 0.000100, edit distance: 0.001134, exp_rate: 0.947056, exp_rate<=1 : 0.980407, exp_rate<=2 : 0.995909, exp_rate<=3 : 0.998488, loss: 0.001383, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19263 s, avg_samples: 56.0, ips: 290.70616 samples/s, eta: 6:11:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:40:46] ppocr INFO: epoch: [458/500], global_step: 431000, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.945644, exp_rate<=1 : 0.979881, exp_rate<=2 : 0.995621, exp_rate<=3 : 0.998440, loss: 0.001294, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21648 s, avg_samples: 56.0, ips: 258.68668 samples/s, eta: 6:11:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:41:05] ppocr INFO: epoch: [458/500], global_step: 431100, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.945824, exp_rate<=1 : 0.979644, exp_rate<=2 : 0.995496, exp_rate<=3 : 0.998493, loss: 0.001024, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18734 s, avg_samples: 56.0, ips: 298.91941 samples/s, eta: 6:10:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:41:23] ppocr INFO: epoch: [458/500], global_step: 431200, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.946189, exp_rate<=1 : 0.979514, exp_rate<=2 : 0.995414, exp_rate<=3 : 0.998450, loss: 0.001184, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18245 s, avg_samples: 56.0, ips: 306.92796 samples/s, eta: 6:10:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:41:42] ppocr INFO: epoch: [458/500], global_step: 431300, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.946509, exp_rate<=1 : 0.979453, exp_rate<=2 : 0.995542, exp_rate<=3 : 0.998511, loss: 0.001761, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18495 s, avg_samples: 56.0, ips: 302.77891 samples/s, eta: 6:10:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:42:01] ppocr INFO: epoch: [458/500], global_step: 431400, lr: 0.000100, edit distance: 0.001175, exp_rate: 0.945708, exp_rate<=1 : 0.979133, exp_rate<=2 : 0.995530, exp_rate<=3 : 0.998516, loss: 0.001214, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19236 s, avg_samples: 56.0, ips: 291.11612 samples/s, eta: 6:09:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:42:21] ppocr INFO: epoch: [458/500], global_step: 431500, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.944678, exp_rate<=1 : 0.978884, exp_rate<=2 : 0.995367, exp_rate<=3 : 0.998460, loss: 0.001333, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19609 s, avg_samples: 56.0, ips: 285.57963 samples/s, eta: 6:09:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:42:39] ppocr INFO: epoch: [458/500], global_step: 431600, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.944608, exp_rate<=1 : 0.978993, exp_rate<=2 : 0.995455, exp_rate<=3 : 0.998506, loss: 0.001753, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18601 s, avg_samples: 56.0, ips: 301.06203 samples/s, eta: 6:09:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:42:59] ppocr INFO: epoch: [458/500], global_step: 431700, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.943929, exp_rate<=1 : 0.978804, exp_rate<=2 : 0.995461, exp_rate<=3 : 0.998504, loss: 0.001132, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19411 s, avg_samples: 56.0, ips: 288.49336 samples/s, eta: 6:08:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:43:17] ppocr INFO: epoch: [458/500], global_step: 431800, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.943932, exp_rate<=1 : 0.978762, exp_rate<=2 : 0.995464, exp_rate<=3 : 0.998492, loss: 0.001359, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18273 s, avg_samples: 56.0, ips: 306.46915 samples/s, eta: 6:08:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:43:36] ppocr INFO: epoch: [458/500], global_step: 431900, lr: 0.000100, edit distance: 0.001214, exp_rate: 0.943489, exp_rate<=1 : 0.978731, exp_rate<=2 : 0.995447, exp_rate<=3 : 0.998476, loss: 0.001760, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19221 s, avg_samples: 56.0, ips: 291.34670 samples/s, eta: 6:08:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:43:56] ppocr INFO: epoch: [458/500], global_step: 432000, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.943007, exp_rate<=1 : 0.978471, exp_rate<=2 : 0.995404, exp_rate<=3 : 0.998426, loss: 0.002128, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19838 s, avg_samples: 56.0, ips: 282.28653 samples/s, eta: 6:07:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:44:14] ppocr INFO: epoch: [458/500], global_step: 432100, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.943162, exp_rate<=1 : 0.978504, exp_rate<=2 : 0.995411, exp_rate<=3 : 0.998421, loss: 0.001864, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17849 s, avg_samples: 56.0, ips: 313.73944 samples/s, eta: 6:07:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:44:34] ppocr INFO: epoch: [458/500], global_step: 432200, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.942793, exp_rate<=1 : 0.978380, exp_rate<=2 : 0.995444, exp_rate<=3 : 0.998428, loss: 0.001375, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19586 s, avg_samples: 56.0, ips: 285.91759 samples/s, eta: 6:07:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:44:53] ppocr INFO: epoch: [458/500], global_step: 432300, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.942651, exp_rate<=1 : 0.978398, exp_rate<=2 : 0.995446, exp_rate<=3 : 0.998431, loss: 0.001427, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19160 s, avg_samples: 56.0, ips: 292.27301 samples/s, eta: 6:06:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:45:12] ppocr INFO: epoch: [458/500], global_step: 432400, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.942812, exp_rate<=1 : 0.978487, exp_rate<=2 : 0.995446, exp_rate<=3 : 0.998427, loss: 0.000938, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19693 s, avg_samples: 56.0, ips: 284.36413 samples/s, eta: 6:06:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:45:31] ppocr INFO: epoch: [458/500], global_step: 432500, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.943299, exp_rate<=1 : 0.978615, exp_rate<=2 : 0.995500, exp_rate<=3 : 0.998441, loss: 0.001317, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18081 s, avg_samples: 56.0, ips: 309.71948 samples/s, eta: 6:06:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:45:49] ppocr INFO: epoch: [458/500], global_step: 432600, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.943258, exp_rate<=1 : 0.978646, exp_rate<=2 : 0.995467, exp_rate<=3 : 0.998435, loss: 0.001616, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18242 s, avg_samples: 56.0, ips: 306.97957 samples/s, eta: 6:05:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:46:08] ppocr INFO: epoch: [458/500], global_step: 432700, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.943075, exp_rate<=1 : 0.978560, exp_rate<=2 : 0.995445, exp_rate<=3 : 0.998417, loss: 0.001733, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19223 s, avg_samples: 56.0, ips: 291.31902 samples/s, eta: 6:05:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:46:27] ppocr INFO: epoch: [458/500], global_step: 432800, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942915, exp_rate<=1 : 0.978458, exp_rate<=2 : 0.995425, exp_rate<=3 : 0.998404, loss: 0.001351, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18638 s, avg_samples: 56.0, ips: 300.45520 samples/s, eta: 6:05:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:46:45] ppocr INFO: epoch: [458/500], global_step: 432900, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.942895, exp_rate<=1 : 0.978460, exp_rate<=2 : 0.995401, exp_rate<=3 : 0.998378, loss: 0.002082, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18249 s, avg_samples: 56.0, ips: 306.86279 samples/s, eta: 6:04:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:46:48] ppocr INFO: epoch: [458/500], global_step: 432920, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942844, exp_rate<=1 : 0.978480, exp_rate<=2 : 0.995397, exp_rate<=3 : 0.998378, loss: 0.001212, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03386 s, avg_samples: 11.2, ips: 330.75545 samples/s, eta: 6:04:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:46:50] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 01:47:07] ppocr INFO: epoch: [459/500], global_step: 433000, lr: 0.000100, edit distance: 0.001021, exp_rate: 0.955034, exp_rate<=1 : 0.983606, exp_rate<=2 : 0.996354, exp_rate<=3 : 0.999078, loss: 0.001015, avg_reader_cost: 0.02529 s, avg_batch_cost: 0.18437 s, avg_samples: 44.8, ips: 242.99088 samples/s, eta: 6:04:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:47:24] ppocr INFO: epoch: [459/500], global_step: 433100, lr: 0.000100, edit distance: 0.001103, exp_rate: 0.950956, exp_rate<=1 : 0.981070, exp_rate<=2 : 0.996129, exp_rate<=3 : 0.999258, loss: 0.001485, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17494 s, avg_samples: 56.0, ips: 320.11164 samples/s, eta: 6:04:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:47:43] ppocr INFO: epoch: [459/500], global_step: 433200, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.949470, exp_rate<=1 : 0.981067, exp_rate<=2 : 0.996355, exp_rate<=3 : 0.999327, loss: 0.001093, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18424 s, avg_samples: 56.0, ips: 303.95843 samples/s, eta: 6:04:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:48:02] ppocr INFO: epoch: [459/500], global_step: 433300, lr: 0.000100, edit distance: 0.001117, exp_rate: 0.948863, exp_rate<=1 : 0.980653, exp_rate<=2 : 0.995662, exp_rate<=3 : 0.998754, loss: 0.001155, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19254 s, avg_samples: 56.0, ips: 290.84747 samples/s, eta: 6:03:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:48:22] ppocr INFO: epoch: [459/500], global_step: 433400, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.947928, exp_rate<=1 : 0.980342, exp_rate<=2 : 0.995853, exp_rate<=3 : 0.998911, loss: 0.001790, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19476 s, avg_samples: 56.0, ips: 287.53239 samples/s, eta: 6:03:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:48:41] ppocr INFO: epoch: [459/500], global_step: 433500, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.946180, exp_rate<=1 : 0.979752, exp_rate<=2 : 0.995761, exp_rate<=3 : 0.998815, loss: 0.001227, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18950 s, avg_samples: 56.0, ips: 295.52079 samples/s, eta: 6:03:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:48:59] ppocr INFO: epoch: [459/500], global_step: 433600, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.946522, exp_rate<=1 : 0.979826, exp_rate<=2 : 0.995765, exp_rate<=3 : 0.998866, loss: 0.001794, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18554 s, avg_samples: 56.0, ips: 301.81971 samples/s, eta: 6:02:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:49:18] ppocr INFO: epoch: [459/500], global_step: 433700, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.944973, exp_rate<=1 : 0.979088, exp_rate<=2 : 0.995420, exp_rate<=3 : 0.998739, loss: 0.001233, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19185 s, avg_samples: 56.0, ips: 291.90183 samples/s, eta: 6:02:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:49:37] ppocr INFO: epoch: [459/500], global_step: 433800, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.944758, exp_rate<=1 : 0.978971, exp_rate<=2 : 0.995199, exp_rate<=3 : 0.998626, loss: 0.001999, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18725 s, avg_samples: 56.0, ips: 299.06417 samples/s, eta: 6:02:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:49:57] ppocr INFO: epoch: [459/500], global_step: 433900, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.945037, exp_rate<=1 : 0.979114, exp_rate<=2 : 0.995324, exp_rate<=3 : 0.998666, loss: 0.001111, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19401 s, avg_samples: 56.0, ips: 288.65032 samples/s, eta: 6:01:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:50:15] ppocr INFO: epoch: [459/500], global_step: 434000, lr: 0.000100, edit distance: 0.001175, exp_rate: 0.945513, exp_rate<=1 : 0.979430, exp_rate<=2 : 0.995425, exp_rate<=3 : 0.998657, loss: 0.001250, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18923 s, avg_samples: 56.0, ips: 295.93126 samples/s, eta: 6:01:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:50:33] ppocr INFO: epoch: [459/500], global_step: 434100, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.944979, exp_rate<=1 : 0.978878, exp_rate<=2 : 0.995452, exp_rate<=3 : 0.998609, loss: 0.001392, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17537 s, avg_samples: 56.0, ips: 319.32528 samples/s, eta: 6:01:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:50:53] ppocr INFO: epoch: [459/500], global_step: 434200, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.944211, exp_rate<=1 : 0.978723, exp_rate<=2 : 0.995430, exp_rate<=3 : 0.998623, loss: 0.001621, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19587 s, avg_samples: 56.0, ips: 285.90143 samples/s, eta: 6:00:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:51:11] ppocr INFO: epoch: [459/500], global_step: 434300, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.943816, exp_rate<=1 : 0.978540, exp_rate<=2 : 0.995367, exp_rate<=3 : 0.998593, loss: 0.001656, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18728 s, avg_samples: 56.0, ips: 299.01790 samples/s, eta: 6:00:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:51:30] ppocr INFO: epoch: [459/500], global_step: 434400, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.942929, exp_rate<=1 : 0.978140, exp_rate<=2 : 0.995183, exp_rate<=3 : 0.998516, loss: 0.001664, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18596 s, avg_samples: 56.0, ips: 301.13384 samples/s, eta: 6:00:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:51:50] ppocr INFO: epoch: [459/500], global_step: 434500, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942870, exp_rate<=1 : 0.977980, exp_rate<=2 : 0.995112, exp_rate<=3 : 0.998427, loss: 0.001070, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19547 s, avg_samples: 56.0, ips: 286.48336 samples/s, eta: 5:59:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:52:08] ppocr INFO: epoch: [459/500], global_step: 434600, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.943067, exp_rate<=1 : 0.978067, exp_rate<=2 : 0.995202, exp_rate<=3 : 0.998444, loss: 0.002204, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18248 s, avg_samples: 56.0, ips: 306.88089 samples/s, eta: 5:59:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:52:27] ppocr INFO: epoch: [459/500], global_step: 434700, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.942534, exp_rate<=1 : 0.977655, exp_rate<=2 : 0.995051, exp_rate<=3 : 0.998384, loss: 0.002471, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19653 s, avg_samples: 56.0, ips: 284.93932 samples/s, eta: 5:59:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:52:46] ppocr INFO: epoch: [459/500], global_step: 434800, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.942332, exp_rate<=1 : 0.977588, exp_rate<=2 : 0.995034, exp_rate<=3 : 0.998310, loss: 0.001416, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18650 s, avg_samples: 56.0, ips: 300.26390 samples/s, eta: 5:58:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:53:06] ppocr INFO: epoch: [459/500], global_step: 434900, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.942424, exp_rate<=1 : 0.977683, exp_rate<=2 : 0.995051, exp_rate<=3 : 0.998257, loss: 0.001270, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20313 s, avg_samples: 56.0, ips: 275.69116 samples/s, eta: 5:58:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:53:25] ppocr INFO: epoch: [459/500], global_step: 435000, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.942471, exp_rate<=1 : 0.977602, exp_rate<=2 : 0.995086, exp_rate<=3 : 0.998257, loss: 0.001419, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18676 s, avg_samples: 56.0, ips: 299.85711 samples/s, eta: 5:58:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:53:43] ppocr INFO: epoch: [459/500], global_step: 435100, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942692, exp_rate<=1 : 0.977689, exp_rate<=2 : 0.995109, exp_rate<=3 : 0.998262, loss: 0.001251, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17972 s, avg_samples: 56.0, ips: 311.58786 samples/s, eta: 5:57:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:54:02] ppocr INFO: epoch: [459/500], global_step: 435200, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.942858, exp_rate<=1 : 0.977746, exp_rate<=2 : 0.995122, exp_rate<=3 : 0.998244, loss: 0.001422, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19092 s, avg_samples: 56.0, ips: 293.32371 samples/s, eta: 5:57:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:54:21] ppocr INFO: epoch: [459/500], global_step: 435300, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942528, exp_rate<=1 : 0.977681, exp_rate<=2 : 0.995136, exp_rate<=3 : 0.998256, loss: 0.001102, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18489 s, avg_samples: 56.0, ips: 302.87822 samples/s, eta: 5:57:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:54:39] ppocr INFO: epoch: [459/500], global_step: 435400, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942396, exp_rate<=1 : 0.977644, exp_rate<=2 : 0.995126, exp_rate<=3 : 0.998233, loss: 0.001273, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18444 s, avg_samples: 56.0, ips: 303.62693 samples/s, eta: 5:57:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:54:58] ppocr INFO: epoch: [459/500], global_step: 435500, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.942454, exp_rate<=1 : 0.977703, exp_rate<=2 : 0.995137, exp_rate<=3 : 0.998225, loss: 0.001231, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18480 s, avg_samples: 56.0, ips: 303.02819 samples/s, eta: 5:56:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:55:17] ppocr INFO: epoch: [459/500], global_step: 435600, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.942271, exp_rate<=1 : 0.977592, exp_rate<=2 : 0.995045, exp_rate<=3 : 0.998161, loss: 0.001544, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18830 s, avg_samples: 56.0, ips: 297.39016 samples/s, eta: 5:56:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:55:27] ppocr INFO: epoch: [459/500], global_step: 435660, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.942338, exp_rate<=1 : 0.977619, exp_rate<=2 : 0.995030, exp_rate<=3 : 0.998177, loss: 0.001562, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10452 s, avg_samples: 33.6, ips: 321.47574 samples/s, eta: 5:56:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:55:29] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 01:55:37] ppocr INFO: epoch: [460/500], global_step: 435700, lr: 0.000100, edit distance: 0.000924, exp_rate: 0.959131, exp_rate<=1 : 0.983766, exp_rate<=2 : 0.996936, exp_rate<=3 : 0.999245, loss: 0.000873, avg_reader_cost: 0.02586 s, avg_batch_cost: 0.09752 s, avg_samples: 22.4, ips: 229.69936 samples/s, eta: 5:56:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:55:56] ppocr INFO: epoch: [460/500], global_step: 435800, lr: 0.000100, edit distance: 0.001099, exp_rate: 0.950319, exp_rate<=1 : 0.980684, exp_rate<=2 : 0.995363, exp_rate<=3 : 0.998084, loss: 0.001433, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19514 s, avg_samples: 56.0, ips: 286.96878 samples/s, eta: 5:55:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:56:15] ppocr INFO: epoch: [460/500], global_step: 435900, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.946856, exp_rate<=1 : 0.980042, exp_rate<=2 : 0.995236, exp_rate<=3 : 0.998373, loss: 0.001208, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18505 s, avg_samples: 56.0, ips: 302.62658 samples/s, eta: 5:55:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:56:33] ppocr INFO: epoch: [460/500], global_step: 436000, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.945811, exp_rate<=1 : 0.979291, exp_rate<=2 : 0.995083, exp_rate<=3 : 0.998325, loss: 0.001080, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18513 s, avg_samples: 56.0, ips: 302.48401 samples/s, eta: 5:55:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:56:52] ppocr INFO: epoch: [460/500], global_step: 436100, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.945301, exp_rate<=1 : 0.979601, exp_rate<=2 : 0.995064, exp_rate<=3 : 0.998465, loss: 0.000896, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18407 s, avg_samples: 56.0, ips: 304.22432 samples/s, eta: 5:54:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:57:12] ppocr INFO: epoch: [460/500], global_step: 436200, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.944601, exp_rate<=1 : 0.979795, exp_rate<=2 : 0.995331, exp_rate<=3 : 0.998595, loss: 0.001232, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20562 s, avg_samples: 56.0, ips: 272.34716 samples/s, eta: 5:54:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:57:31] ppocr INFO: epoch: [460/500], global_step: 436300, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.944204, exp_rate<=1 : 0.979467, exp_rate<=2 : 0.995441, exp_rate<=3 : 0.998683, loss: 0.002025, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18685 s, avg_samples: 56.0, ips: 299.70910 samples/s, eta: 5:54:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:57:48] ppocr INFO: epoch: [460/500], global_step: 436400, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.944764, exp_rate<=1 : 0.979902, exp_rate<=2 : 0.995653, exp_rate<=3 : 0.998685, loss: 0.001005, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17245 s, avg_samples: 56.0, ips: 324.73867 samples/s, eta: 5:53:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:58:08] ppocr INFO: epoch: [460/500], global_step: 436500, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.943943, exp_rate<=1 : 0.979722, exp_rate<=2 : 0.995664, exp_rate<=3 : 0.998663, loss: 0.001924, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19415 s, avg_samples: 56.0, ips: 288.43404 samples/s, eta: 5:53:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:58:27] ppocr INFO: epoch: [460/500], global_step: 436600, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.943606, exp_rate<=1 : 0.979377, exp_rate<=2 : 0.995718, exp_rate<=3 : 0.998640, loss: 0.001848, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18750 s, avg_samples: 56.0, ips: 298.67124 samples/s, eta: 5:53:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:58:46] ppocr INFO: epoch: [460/500], global_step: 436700, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.943386, exp_rate<=1 : 0.978947, exp_rate<=2 : 0.995443, exp_rate<=3 : 0.998486, loss: 0.001672, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19341 s, avg_samples: 56.0, ips: 289.53570 samples/s, eta: 5:52:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:59:04] ppocr INFO: epoch: [460/500], global_step: 436800, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.943943, exp_rate<=1 : 0.979242, exp_rate<=2 : 0.995565, exp_rate<=3 : 0.998519, loss: 0.001467, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17565 s, avg_samples: 56.0, ips: 318.81228 samples/s, eta: 5:52:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:59:22] ppocr INFO: epoch: [460/500], global_step: 436900, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.943816, exp_rate<=1 : 0.979045, exp_rate<=2 : 0.995383, exp_rate<=3 : 0.998394, loss: 0.001310, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18570 s, avg_samples: 56.0, ips: 301.56648 samples/s, eta: 5:52:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 01:59:40] ppocr INFO: epoch: [460/500], global_step: 437000, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.943988, exp_rate<=1 : 0.979123, exp_rate<=2 : 0.995448, exp_rate<=3 : 0.998464, loss: 0.001820, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18131 s, avg_samples: 56.0, ips: 308.85745 samples/s, eta: 5:51:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:00:01] ppocr INFO: epoch: [460/500], global_step: 437100, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.943149, exp_rate<=1 : 0.978821, exp_rate<=2 : 0.995363, exp_rate<=3 : 0.998440, loss: 0.001102, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20697 s, avg_samples: 56.0, ips: 270.56763 samples/s, eta: 5:51:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:00:21] ppocr INFO: epoch: [460/500], global_step: 437200, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.943258, exp_rate<=1 : 0.978799, exp_rate<=2 : 0.995417, exp_rate<=3 : 0.998491, loss: 0.001448, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19993 s, avg_samples: 56.0, ips: 280.09893 samples/s, eta: 5:51:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:00:39] ppocr INFO: epoch: [460/500], global_step: 437300, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.943296, exp_rate<=1 : 0.978822, exp_rate<=2 : 0.995378, exp_rate<=3 : 0.998504, loss: 0.001837, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17725 s, avg_samples: 56.0, ips: 315.93441 samples/s, eta: 5:51:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:00:58] ppocr INFO: epoch: [460/500], global_step: 437400, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.943390, exp_rate<=1 : 0.978842, exp_rate<=2 : 0.995378, exp_rate<=3 : 0.998501, loss: 0.001698, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19232 s, avg_samples: 56.0, ips: 291.18155 samples/s, eta: 5:50:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:01:17] ppocr INFO: epoch: [460/500], global_step: 437500, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.943107, exp_rate<=1 : 0.978650, exp_rate<=2 : 0.995278, exp_rate<=3 : 0.998431, loss: 0.000867, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19294 s, avg_samples: 56.0, ips: 290.25003 samples/s, eta: 5:50:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:01:35] ppocr INFO: epoch: [460/500], global_step: 437600, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.943471, exp_rate<=1 : 0.978924, exp_rate<=2 : 0.995339, exp_rate<=3 : 0.998414, loss: 0.001730, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17508 s, avg_samples: 56.0, ips: 319.85681 samples/s, eta: 5:50:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:01:52] ppocr INFO: epoch: [460/500], global_step: 437700, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.943507, exp_rate<=1 : 0.978775, exp_rate<=2 : 0.995373, exp_rate<=3 : 0.998435, loss: 0.001094, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17222 s, avg_samples: 56.0, ips: 325.16011 samples/s, eta: 5:49:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:02:10] ppocr INFO: epoch: [460/500], global_step: 437800, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.943358, exp_rate<=1 : 0.978790, exp_rate<=2 : 0.995360, exp_rate<=3 : 0.998472, loss: 0.001317, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18046 s, avg_samples: 56.0, ips: 310.32660 samples/s, eta: 5:49:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:02:31] ppocr INFO: epoch: [460/500], global_step: 437900, lr: 0.000100, edit distance: 0.001214, exp_rate: 0.942273, exp_rate<=1 : 0.978298, exp_rate<=2 : 0.995149, exp_rate<=3 : 0.998352, loss: 0.001059, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20808 s, avg_samples: 56.0, ips: 269.13348 samples/s, eta: 5:49:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:02:50] ppocr INFO: epoch: [460/500], global_step: 438000, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.942222, exp_rate<=1 : 0.978217, exp_rate<=2 : 0.995164, exp_rate<=3 : 0.998349, loss: 0.001570, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19329 s, avg_samples: 56.0, ips: 289.71519 samples/s, eta: 5:48:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:03:09] ppocr INFO: epoch: [460/500], global_step: 438100, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.941654, exp_rate<=1 : 0.978050, exp_rate<=2 : 0.995077, exp_rate<=3 : 0.998291, loss: 0.001803, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18767 s, avg_samples: 56.0, ips: 298.40347 samples/s, eta: 5:48:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:03:28] ppocr INFO: epoch: [460/500], global_step: 438200, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.941361, exp_rate<=1 : 0.978015, exp_rate<=2 : 0.995086, exp_rate<=3 : 0.998279, loss: 0.001583, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18811 s, avg_samples: 56.0, ips: 297.70054 samples/s, eta: 5:48:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:03:47] ppocr INFO: epoch: [460/500], global_step: 438300, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.941382, exp_rate<=1 : 0.978061, exp_rate<=2 : 0.995133, exp_rate<=3 : 0.998288, loss: 0.001464, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18586 s, avg_samples: 56.0, ips: 301.30116 samples/s, eta: 5:47:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:04:05] ppocr INFO: epoch: [460/500], global_step: 438400, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.941183, exp_rate<=1 : 0.978029, exp_rate<=2 : 0.995155, exp_rate<=3 : 0.998305, loss: 0.001455, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18120 s, avg_samples: 56.0, ips: 309.05776 samples/s, eta: 5:47:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:04:07] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 02:04:08] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_460 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 02:04:27] ppocr INFO: epoch: [461/500], global_step: 438500, lr: 0.000100, edit distance: 0.001562, exp_rate: 0.945949, exp_rate<=1 : 0.978738, exp_rate<=2 : 0.995024, exp_rate<=3 : 0.998368, loss: 0.002376, avg_reader_cost: 0.03252 s, avg_batch_cost: 0.21968 s, avg_samples: 56.0, ips: 254.91947 samples/s, eta: 5:47:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:04:45] ppocr INFO: epoch: [461/500], global_step: 438600, lr: 0.000100, edit distance: 0.001489, exp_rate: 0.941851, exp_rate<=1 : 0.977095, exp_rate<=2 : 0.995653, exp_rate<=3 : 0.998500, loss: 0.001811, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18722 s, avg_samples: 56.0, ips: 299.11367 samples/s, eta: 5:46:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:05:05] ppocr INFO: epoch: [461/500], global_step: 438700, lr: 0.000100, edit distance: 0.001336, exp_rate: 0.944438, exp_rate<=1 : 0.978608, exp_rate<=2 : 0.995767, exp_rate<=3 : 0.998648, loss: 0.000793, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19473 s, avg_samples: 56.0, ips: 287.58210 samples/s, eta: 5:46:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:05:22] ppocr INFO: epoch: [461/500], global_step: 438800, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.947353, exp_rate<=1 : 0.979945, exp_rate<=2 : 0.996136, exp_rate<=3 : 0.998828, loss: 0.001461, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17340 s, avg_samples: 56.0, ips: 322.95255 samples/s, eta: 5:46:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:05:41] ppocr INFO: epoch: [461/500], global_step: 438900, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.946138, exp_rate<=1 : 0.979429, exp_rate<=2 : 0.995971, exp_rate<=3 : 0.998544, loss: 0.000841, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18673 s, avg_samples: 56.0, ips: 299.89383 samples/s, eta: 5:45:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:05:59] ppocr INFO: epoch: [461/500], global_step: 439000, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.947762, exp_rate<=1 : 0.980102, exp_rate<=2 : 0.995970, exp_rate<=3 : 0.998596, loss: 0.001093, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17600 s, avg_samples: 56.0, ips: 318.17966 samples/s, eta: 5:45:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:06:17] ppocr INFO: epoch: [461/500], global_step: 439100, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.947528, exp_rate<=1 : 0.979996, exp_rate<=2 : 0.996019, exp_rate<=3 : 0.998578, loss: 0.001367, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18800 s, avg_samples: 56.0, ips: 297.86973 samples/s, eta: 5:45:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:06:37] ppocr INFO: epoch: [461/500], global_step: 439200, lr: 0.000100, edit distance: 0.001273, exp_rate: 0.945413, exp_rate<=1 : 0.978870, exp_rate<=2 : 0.995567, exp_rate<=3 : 0.998441, loss: 0.001770, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19585 s, avg_samples: 56.0, ips: 285.93082 samples/s, eta: 5:44:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:06:57] ppocr INFO: epoch: [461/500], global_step: 439300, lr: 0.000100, edit distance: 0.001268, exp_rate: 0.944416, exp_rate<=1 : 0.978600, exp_rate<=2 : 0.995591, exp_rate<=3 : 0.998466, loss: 0.001283, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20010 s, avg_samples: 56.0, ips: 279.86452 samples/s, eta: 5:44:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:07:16] ppocr INFO: epoch: [461/500], global_step: 439400, lr: 0.000100, edit distance: 0.001286, exp_rate: 0.944583, exp_rate<=1 : 0.978347, exp_rate<=2 : 0.995549, exp_rate<=3 : 0.998473, loss: 0.001881, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19095 s, avg_samples: 56.0, ips: 293.27668 samples/s, eta: 5:44:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:07:35] ppocr INFO: epoch: [461/500], global_step: 439500, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.944143, exp_rate<=1 : 0.978258, exp_rate<=2 : 0.995397, exp_rate<=3 : 0.998413, loss: 0.001230, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19024 s, avg_samples: 56.0, ips: 294.37071 samples/s, eta: 5:44:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:07:55] ppocr INFO: epoch: [461/500], global_step: 439600, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.944051, exp_rate<=1 : 0.978256, exp_rate<=2 : 0.995494, exp_rate<=3 : 0.998380, loss: 0.001536, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19536 s, avg_samples: 56.0, ips: 286.64497 samples/s, eta: 5:43:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:08:15] ppocr INFO: epoch: [461/500], global_step: 439700, lr: 0.000100, edit distance: 0.001321, exp_rate: 0.943655, exp_rate<=1 : 0.978017, exp_rate<=2 : 0.995414, exp_rate<=3 : 0.998345, loss: 0.001522, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19829 s, avg_samples: 56.0, ips: 282.41760 samples/s, eta: 5:43:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:08:32] ppocr INFO: epoch: [461/500], global_step: 439800, lr: 0.000100, edit distance: 0.001302, exp_rate: 0.943987, exp_rate<=1 : 0.978104, exp_rate<=2 : 0.995504, exp_rate<=3 : 0.998408, loss: 0.001551, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17562 s, avg_samples: 56.0, ips: 318.86407 samples/s, eta: 5:43:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:08:50] ppocr INFO: epoch: [461/500], global_step: 439900, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.944338, exp_rate<=1 : 0.978321, exp_rate<=2 : 0.995543, exp_rate<=3 : 0.998431, loss: 0.000928, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17654 s, avg_samples: 56.0, ips: 317.20285 samples/s, eta: 5:42:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:09:09] ppocr INFO: epoch: [461/500], global_step: 440000, lr: 0.000100, edit distance: 0.001287, exp_rate: 0.944249, exp_rate<=1 : 0.978498, exp_rate<=2 : 0.995534, exp_rate<=3 : 0.998428, loss: 0.000990, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19325 s, avg_samples: 56.0, ips: 289.77411 samples/s, eta: 5:42:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:09:28] ppocr INFO: epoch: [461/500], global_step: 440100, lr: 0.000100, edit distance: 0.001276, exp_rate: 0.944527, exp_rate<=1 : 0.978597, exp_rate<=2 : 0.995561, exp_rate<=3 : 0.998451, loss: 0.001206, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18733 s, avg_samples: 56.0, ips: 298.92976 samples/s, eta: 5:42:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:09:47] ppocr INFO: epoch: [461/500], global_step: 440200, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.944463, exp_rate<=1 : 0.978593, exp_rate<=2 : 0.995532, exp_rate<=3 : 0.998450, loss: 0.001616, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19304 s, avg_samples: 56.0, ips: 290.09177 samples/s, eta: 5:41:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:10:05] ppocr INFO: epoch: [461/500], global_step: 440300, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.944335, exp_rate<=1 : 0.978517, exp_rate<=2 : 0.995481, exp_rate<=3 : 0.998463, loss: 0.001597, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18035 s, avg_samples: 56.0, ips: 310.50387 samples/s, eta: 5:41:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:10:23] ppocr INFO: epoch: [461/500], global_step: 440400, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.944271, exp_rate<=1 : 0.978522, exp_rate<=2 : 0.995426, exp_rate<=3 : 0.998412, loss: 0.000996, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18053 s, avg_samples: 56.0, ips: 310.20077 samples/s, eta: 5:41:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:10:42] ppocr INFO: epoch: [461/500], global_step: 440500, lr: 0.000100, edit distance: 0.001285, exp_rate: 0.944046, exp_rate<=1 : 0.978407, exp_rate<=2 : 0.995455, exp_rate<=3 : 0.998430, loss: 0.001537, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19019 s, avg_samples: 56.0, ips: 294.43552 samples/s, eta: 5:40:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:11:02] ppocr INFO: epoch: [461/500], global_step: 440600, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.943694, exp_rate<=1 : 0.978258, exp_rate<=2 : 0.995406, exp_rate<=3 : 0.998399, loss: 0.001603, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19497 s, avg_samples: 56.0, ips: 287.22836 samples/s, eta: 5:40:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:11:21] ppocr INFO: epoch: [461/500], global_step: 440700, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.943275, exp_rate<=1 : 0.978052, exp_rate<=2 : 0.995322, exp_rate<=3 : 0.998392, loss: 0.001031, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18761 s, avg_samples: 56.0, ips: 298.48870 samples/s, eta: 5:40:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:11:39] ppocr INFO: epoch: [461/500], global_step: 440800, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.943327, exp_rate<=1 : 0.978105, exp_rate<=2 : 0.995353, exp_rate<=3 : 0.998401, loss: 0.001564, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18133 s, avg_samples: 56.0, ips: 308.82178 samples/s, eta: 5:39:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:11:57] ppocr INFO: epoch: [461/500], global_step: 440900, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.943200, exp_rate<=1 : 0.978031, exp_rate<=2 : 0.995348, exp_rate<=3 : 0.998416, loss: 0.001418, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18359 s, avg_samples: 56.0, ips: 305.02209 samples/s, eta: 5:39:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:12:17] ppocr INFO: epoch: [461/500], global_step: 441000, lr: 0.000100, edit distance: 0.001319, exp_rate: 0.942783, exp_rate<=1 : 0.977785, exp_rate<=2 : 0.995206, exp_rate<=3 : 0.998348, loss: 0.001474, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19244 s, avg_samples: 56.0, ips: 291.00386 samples/s, eta: 5:39:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:12:37] ppocr INFO: epoch: [461/500], global_step: 441100, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.942583, exp_rate<=1 : 0.977703, exp_rate<=2 : 0.995160, exp_rate<=3 : 0.998339, loss: 0.001281, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19917 s, avg_samples: 56.0, ips: 281.17274 samples/s, eta: 5:38:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:12:44] ppocr INFO: epoch: [461/500], global_step: 441140, lr: 0.000100, edit distance: 0.001312, exp_rate: 0.942547, exp_rate<=1 : 0.977670, exp_rate<=2 : 0.995106, exp_rate<=3 : 0.998328, loss: 0.001235, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07291 s, avg_samples: 22.4, ips: 307.24621 samples/s, eta: 5:38:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:12:46] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 02:12:58] ppocr INFO: epoch: [462/500], global_step: 441200, lr: 0.000100, edit distance: 0.001401, exp_rate: 0.951978, exp_rate<=1 : 0.981081, exp_rate<=2 : 0.995403, exp_rate<=3 : 0.997449, loss: 0.001134, avg_reader_cost: 0.02575 s, avg_batch_cost: 0.14526 s, avg_samples: 33.6, ips: 231.30702 samples/s, eta: 5:38:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:13:18] ppocr INFO: epoch: [462/500], global_step: 441300, lr: 0.000100, edit distance: 0.001077, exp_rate: 0.952320, exp_rate<=1 : 0.982024, exp_rate<=2 : 0.995857, exp_rate<=3 : 0.997970, loss: 0.000913, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19190 s, avg_samples: 56.0, ips: 291.81462 samples/s, eta: 5:38:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:13:35] ppocr INFO: epoch: [462/500], global_step: 441400, lr: 0.000100, edit distance: 0.001106, exp_rate: 0.952538, exp_rate<=1 : 0.982537, exp_rate<=2 : 0.996492, exp_rate<=3 : 0.998432, loss: 0.000681, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17778 s, avg_samples: 56.0, ips: 314.98860 samples/s, eta: 5:37:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:13:54] ppocr INFO: epoch: [462/500], global_step: 441500, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.950616, exp_rate<=1 : 0.982474, exp_rate<=2 : 0.996536, exp_rate<=3 : 0.998726, loss: 0.001572, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18287 s, avg_samples: 56.0, ips: 306.23136 samples/s, eta: 5:37:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:14:13] ppocr INFO: epoch: [462/500], global_step: 441600, lr: 0.000100, edit distance: 0.001206, exp_rate: 0.944823, exp_rate<=1 : 0.980104, exp_rate<=2 : 0.995833, exp_rate<=3 : 0.998375, loss: 0.001911, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19461 s, avg_samples: 56.0, ips: 287.75892 samples/s, eta: 5:37:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:14:31] ppocr INFO: epoch: [462/500], global_step: 441700, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.943974, exp_rate<=1 : 0.979429, exp_rate<=2 : 0.995511, exp_rate<=3 : 0.998427, loss: 0.001745, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18161 s, avg_samples: 56.0, ips: 308.34817 samples/s, eta: 5:37:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:14:50] ppocr INFO: epoch: [462/500], global_step: 441800, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.944259, exp_rate<=1 : 0.979665, exp_rate<=2 : 0.995420, exp_rate<=3 : 0.998359, loss: 0.001620, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18551 s, avg_samples: 56.0, ips: 301.87667 samples/s, eta: 5:36:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:15:09] ppocr INFO: epoch: [462/500], global_step: 441900, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.944543, exp_rate<=1 : 0.979605, exp_rate<=2 : 0.995527, exp_rate<=3 : 0.998501, loss: 0.001114, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19031 s, avg_samples: 56.0, ips: 294.25370 samples/s, eta: 5:36:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:15:26] ppocr INFO: epoch: [462/500], global_step: 442000, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.945965, exp_rate<=1 : 0.980257, exp_rate<=2 : 0.995760, exp_rate<=3 : 0.998593, loss: 0.000852, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17117 s, avg_samples: 56.0, ips: 327.16630 samples/s, eta: 5:36:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:15:45] ppocr INFO: epoch: [462/500], global_step: 442100, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.946466, exp_rate<=1 : 0.980597, exp_rate<=2 : 0.995883, exp_rate<=3 : 0.998635, loss: 0.001277, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18562 s, avg_samples: 56.0, ips: 301.69239 samples/s, eta: 5:35:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:16:04] ppocr INFO: epoch: [462/500], global_step: 442200, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.945987, exp_rate<=1 : 0.980375, exp_rate<=2 : 0.995625, exp_rate<=3 : 0.998566, loss: 0.002141, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19285 s, avg_samples: 56.0, ips: 290.37888 samples/s, eta: 5:35:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:16:21] ppocr INFO: epoch: [462/500], global_step: 442300, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.946049, exp_rate<=1 : 0.980506, exp_rate<=2 : 0.995687, exp_rate<=3 : 0.998598, loss: 0.001476, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17312 s, avg_samples: 56.0, ips: 323.46679 samples/s, eta: 5:35:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:16:39] ppocr INFO: epoch: [462/500], global_step: 442400, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.946172, exp_rate<=1 : 0.980523, exp_rate<=2 : 0.995701, exp_rate<=3 : 0.998667, loss: 0.000949, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17170 s, avg_samples: 56.0, ips: 326.14589 samples/s, eta: 5:34:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:16:58] ppocr INFO: epoch: [462/500], global_step: 442500, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.945421, exp_rate<=1 : 0.980040, exp_rate<=2 : 0.995633, exp_rate<=3 : 0.998646, loss: 0.001339, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19906 s, avg_samples: 56.0, ips: 281.32606 samples/s, eta: 5:34:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:17:17] ppocr INFO: epoch: [462/500], global_step: 442600, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.945229, exp_rate<=1 : 0.979905, exp_rate<=2 : 0.995695, exp_rate<=3 : 0.998646, loss: 0.001474, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18279 s, avg_samples: 56.0, ips: 306.36301 samples/s, eta: 5:34:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:17:36] ppocr INFO: epoch: [462/500], global_step: 442700, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.944722, exp_rate<=1 : 0.979490, exp_rate<=2 : 0.995470, exp_rate<=3 : 0.998500, loss: 0.001322, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19670 s, avg_samples: 56.0, ips: 284.69676 samples/s, eta: 5:33:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:17:56] ppocr INFO: epoch: [462/500], global_step: 442800, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.944835, exp_rate<=1 : 0.979357, exp_rate<=2 : 0.995431, exp_rate<=3 : 0.998456, loss: 0.001240, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19593 s, avg_samples: 56.0, ips: 285.81582 samples/s, eta: 5:33:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:18:15] ppocr INFO: epoch: [462/500], global_step: 442900, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.945128, exp_rate<=1 : 0.979716, exp_rate<=2 : 0.995540, exp_rate<=3 : 0.998511, loss: 0.000901, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18947 s, avg_samples: 56.0, ips: 295.55958 samples/s, eta: 5:33:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:18:34] ppocr INFO: epoch: [462/500], global_step: 443000, lr: 0.000100, edit distance: 0.001175, exp_rate: 0.945098, exp_rate<=1 : 0.979650, exp_rate<=2 : 0.995493, exp_rate<=3 : 0.998481, loss: 0.001335, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19030 s, avg_samples: 56.0, ips: 294.27302 samples/s, eta: 5:32:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:18:55] ppocr INFO: epoch: [462/500], global_step: 443100, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.944718, exp_rate<=1 : 0.979341, exp_rate<=2 : 0.995508, exp_rate<=3 : 0.998468, loss: 0.001442, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20447 s, avg_samples: 56.0, ips: 273.88048 samples/s, eta: 5:32:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:19:14] ppocr INFO: epoch: [462/500], global_step: 443200, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.944175, exp_rate<=1 : 0.979151, exp_rate<=2 : 0.995489, exp_rate<=3 : 0.998454, loss: 0.001526, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19429 s, avg_samples: 56.0, ips: 288.23354 samples/s, eta: 5:32:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:19:34] ppocr INFO: epoch: [462/500], global_step: 443300, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.943957, exp_rate<=1 : 0.979085, exp_rate<=2 : 0.995458, exp_rate<=3 : 0.998456, loss: 0.001360, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20079 s, avg_samples: 56.0, ips: 278.89497 samples/s, eta: 5:31:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:19:53] ppocr INFO: epoch: [462/500], global_step: 443400, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.943953, exp_rate<=1 : 0.979159, exp_rate<=2 : 0.995494, exp_rate<=3 : 0.998471, loss: 0.000776, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18488 s, avg_samples: 56.0, ips: 302.89605 samples/s, eta: 5:31:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:20:11] ppocr INFO: epoch: [462/500], global_step: 443500, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.943632, exp_rate<=1 : 0.979093, exp_rate<=2 : 0.995484, exp_rate<=3 : 0.998447, loss: 0.001565, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18712 s, avg_samples: 56.0, ips: 299.27271 samples/s, eta: 5:31:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:20:30] ppocr INFO: epoch: [462/500], global_step: 443600, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.943357, exp_rate<=1 : 0.979006, exp_rate<=2 : 0.995408, exp_rate<=3 : 0.998433, loss: 0.001227, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18481 s, avg_samples: 56.0, ips: 303.00832 samples/s, eta: 5:30:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:20:50] ppocr INFO: epoch: [462/500], global_step: 443700, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.943194, exp_rate<=1 : 0.979006, exp_rate<=2 : 0.995468, exp_rate<=3 : 0.998442, loss: 0.001319, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19691 s, avg_samples: 56.0, ips: 284.39237 samples/s, eta: 5:30:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:21:10] ppocr INFO: epoch: [462/500], global_step: 443800, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.943107, exp_rate<=1 : 0.979082, exp_rate<=2 : 0.995449, exp_rate<=3 : 0.998434, loss: 0.001401, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20064 s, avg_samples: 56.0, ips: 279.11009 samples/s, eta: 5:30:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:21:24] ppocr INFO: epoch: [462/500], global_step: 443880, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.943090, exp_rate<=1 : 0.978975, exp_rate<=2 : 0.995423, exp_rate<=3 : 0.998450, loss: 0.001221, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14720 s, avg_samples: 44.8, ips: 304.34850 samples/s, eta: 5:30:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:21:27] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 02:21:32] ppocr INFO: epoch: [463/500], global_step: 443900, lr: 0.000100, edit distance: 0.001098, exp_rate: 0.935119, exp_rate<=1 : 0.969643, exp_rate<=2 : 0.993080, exp_rate<=3 : 0.994894, loss: 0.001118, avg_reader_cost: 0.03199 s, avg_batch_cost: 0.07404 s, avg_samples: 11.2, ips: 151.26301 samples/s, eta: 5:30:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:21:50] ppocr INFO: epoch: [463/500], global_step: 444000, lr: 0.000100, edit distance: 0.000891, exp_rate: 0.954583, exp_rate<=1 : 0.981640, exp_rate<=2 : 0.996720, exp_rate<=3 : 0.998633, loss: 0.001386, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18490 s, avg_samples: 56.0, ips: 302.87186 samples/s, eta: 5:29:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:22:09] ppocr INFO: epoch: [463/500], global_step: 444100, lr: 0.000100, edit distance: 0.001017, exp_rate: 0.954742, exp_rate<=1 : 0.982058, exp_rate<=2 : 0.996055, exp_rate<=3 : 0.998721, loss: 0.000798, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18336 s, avg_samples: 56.0, ips: 305.40485 samples/s, eta: 5:29:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:22:28] ppocr INFO: epoch: [463/500], global_step: 444200, lr: 0.000100, edit distance: 0.000997, exp_rate: 0.955882, exp_rate<=1 : 0.982823, exp_rate<=2 : 0.996194, exp_rate<=3 : 0.998677, loss: 0.001013, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19203 s, avg_samples: 56.0, ips: 291.62319 samples/s, eta: 5:29:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:22:46] ppocr INFO: epoch: [463/500], global_step: 444300, lr: 0.000100, edit distance: 0.000962, exp_rate: 0.956558, exp_rate<=1 : 0.983267, exp_rate<=2 : 0.996511, exp_rate<=3 : 0.998899, loss: 0.001227, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18358 s, avg_samples: 56.0, ips: 305.04222 samples/s, eta: 5:28:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:23:05] ppocr INFO: epoch: [463/500], global_step: 444400, lr: 0.000100, edit distance: 0.000957, exp_rate: 0.956064, exp_rate<=1 : 0.983211, exp_rate<=2 : 0.996602, exp_rate<=3 : 0.998949, loss: 0.000993, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18659 s, avg_samples: 56.0, ips: 300.12015 samples/s, eta: 5:28:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:23:22] ppocr INFO: epoch: [463/500], global_step: 444500, lr: 0.000100, edit distance: 0.000941, exp_rate: 0.956368, exp_rate<=1 : 0.983752, exp_rate<=2 : 0.996832, exp_rate<=3 : 0.998980, loss: 0.001952, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17304 s, avg_samples: 56.0, ips: 323.62379 samples/s, eta: 5:28:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:23:40] ppocr INFO: epoch: [463/500], global_step: 444600, lr: 0.000100, edit distance: 0.000978, exp_rate: 0.955258, exp_rate<=1 : 0.983365, exp_rate<=2 : 0.996706, exp_rate<=3 : 0.998985, loss: 0.001671, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17869 s, avg_samples: 56.0, ips: 313.40024 samples/s, eta: 5:27:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:24:00] ppocr INFO: epoch: [463/500], global_step: 444700, lr: 0.000100, edit distance: 0.001011, exp_rate: 0.953022, exp_rate<=1 : 0.982564, exp_rate<=2 : 0.996418, exp_rate<=3 : 0.998920, loss: 0.001066, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19741 s, avg_samples: 56.0, ips: 283.67726 samples/s, eta: 5:27:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:24:20] ppocr INFO: epoch: [463/500], global_step: 444800, lr: 0.000100, edit distance: 0.001019, exp_rate: 0.952322, exp_rate<=1 : 0.982458, exp_rate<=2 : 0.996487, exp_rate<=3 : 0.998887, loss: 0.001222, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19867 s, avg_samples: 56.0, ips: 281.87407 samples/s, eta: 5:27:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:24:38] ppocr INFO: epoch: [463/500], global_step: 444900, lr: 0.000100, edit distance: 0.001040, exp_rate: 0.951330, exp_rate<=1 : 0.981957, exp_rate<=2 : 0.996406, exp_rate<=3 : 0.998922, loss: 0.001604, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18309 s, avg_samples: 56.0, ips: 305.86353 samples/s, eta: 5:26:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:24:56] ppocr INFO: epoch: [463/500], global_step: 445000, lr: 0.000100, edit distance: 0.001055, exp_rate: 0.951204, exp_rate<=1 : 0.981771, exp_rate<=2 : 0.996346, exp_rate<=3 : 0.998981, loss: 0.001165, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18067 s, avg_samples: 56.0, ips: 309.96412 samples/s, eta: 5:26:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:25:16] ppocr INFO: epoch: [463/500], global_step: 445100, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.949502, exp_rate<=1 : 0.980843, exp_rate<=2 : 0.996032, exp_rate<=3 : 0.998825, loss: 0.001552, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19577 s, avg_samples: 56.0, ips: 286.04925 samples/s, eta: 5:26:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:25:34] ppocr INFO: epoch: [463/500], global_step: 445200, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.948340, exp_rate<=1 : 0.980558, exp_rate<=2 : 0.995941, exp_rate<=3 : 0.998787, loss: 0.001652, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17747 s, avg_samples: 56.0, ips: 315.55217 samples/s, eta: 5:25:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:25:54] ppocr INFO: epoch: [463/500], global_step: 445300, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.946511, exp_rate<=1 : 0.979756, exp_rate<=2 : 0.995702, exp_rate<=3 : 0.998683, loss: 0.001447, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20450 s, avg_samples: 56.0, ips: 273.84071 samples/s, eta: 5:25:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:26:13] ppocr INFO: epoch: [463/500], global_step: 445400, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.946310, exp_rate<=1 : 0.979879, exp_rate<=2 : 0.995814, exp_rate<=3 : 0.998738, loss: 0.001299, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18805 s, avg_samples: 56.0, ips: 297.79972 samples/s, eta: 5:25:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:26:32] ppocr INFO: epoch: [463/500], global_step: 445500, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.945624, exp_rate<=1 : 0.979408, exp_rate<=2 : 0.995559, exp_rate<=3 : 0.998614, loss: 0.002298, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18981 s, avg_samples: 56.0, ips: 295.03715 samples/s, eta: 5:24:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:26:52] ppocr INFO: epoch: [463/500], global_step: 445600, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.944989, exp_rate<=1 : 0.979107, exp_rate<=2 : 0.995459, exp_rate<=3 : 0.998622, loss: 0.002198, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19920 s, avg_samples: 56.0, ips: 281.12716 samples/s, eta: 5:24:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:27:11] ppocr INFO: epoch: [463/500], global_step: 445700, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.944391, exp_rate<=1 : 0.978826, exp_rate<=2 : 0.995330, exp_rate<=3 : 0.998547, loss: 0.001498, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19466 s, avg_samples: 56.0, ips: 287.67929 samples/s, eta: 5:24:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:27:31] ppocr INFO: epoch: [463/500], global_step: 445800, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.943856, exp_rate<=1 : 0.978639, exp_rate<=2 : 0.995275, exp_rate<=3 : 0.998480, loss: 0.001547, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20019 s, avg_samples: 56.0, ips: 279.73208 samples/s, eta: 5:24:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:27:49] ppocr INFO: epoch: [463/500], global_step: 445900, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.943672, exp_rate<=1 : 0.978664, exp_rate<=2 : 0.995343, exp_rate<=3 : 0.998488, loss: 0.001408, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18044 s, avg_samples: 56.0, ips: 310.35543 samples/s, eta: 5:23:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:28:08] ppocr INFO: epoch: [463/500], global_step: 446000, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.943759, exp_rate<=1 : 0.978763, exp_rate<=2 : 0.995424, exp_rate<=3 : 0.998504, loss: 0.001185, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19013 s, avg_samples: 56.0, ips: 294.53092 samples/s, eta: 5:23:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:28:27] ppocr INFO: epoch: [463/500], global_step: 446100, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.943619, exp_rate<=1 : 0.978698, exp_rate<=2 : 0.995355, exp_rate<=3 : 0.998481, loss: 0.001807, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18708 s, avg_samples: 56.0, ips: 299.33531 samples/s, eta: 5:23:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:28:46] ppocr INFO: epoch: [463/500], global_step: 446200, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.943754, exp_rate<=1 : 0.978756, exp_rate<=2 : 0.995365, exp_rate<=3 : 0.998522, loss: 0.001019, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18656 s, avg_samples: 56.0, ips: 300.16511 samples/s, eta: 5:22:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:29:05] ppocr INFO: epoch: [463/500], global_step: 446300, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.943865, exp_rate<=1 : 0.978763, exp_rate<=2 : 0.995346, exp_rate<=3 : 0.998506, loss: 0.001867, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19043 s, avg_samples: 56.0, ips: 294.07369 samples/s, eta: 5:22:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:29:24] ppocr INFO: epoch: [463/500], global_step: 446400, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.943333, exp_rate<=1 : 0.978667, exp_rate<=2 : 0.995340, exp_rate<=3 : 0.998503, loss: 0.001784, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19543 s, avg_samples: 56.0, ips: 286.54155 samples/s, eta: 5:22:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:29:44] ppocr INFO: epoch: [463/500], global_step: 446500, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.942915, exp_rate<=1 : 0.978487, exp_rate<=2 : 0.995272, exp_rate<=3 : 0.998463, loss: 0.001499, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19780 s, avg_samples: 56.0, ips: 283.12053 samples/s, eta: 5:21:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:30:02] ppocr INFO: epoch: [463/500], global_step: 446600, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.943030, exp_rate<=1 : 0.978577, exp_rate<=2 : 0.995348, exp_rate<=3 : 0.998491, loss: 0.001263, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17767 s, avg_samples: 56.0, ips: 315.18738 samples/s, eta: 5:21:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:30:06] ppocr INFO: epoch: [463/500], global_step: 446620, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.943004, exp_rate<=1 : 0.978572, exp_rate<=2 : 0.995354, exp_rate<=3 : 0.998493, loss: 0.001356, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03845 s, avg_samples: 11.2, ips: 291.30837 samples/s, eta: 5:21:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:30:08] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 02:30:23] ppocr INFO: epoch: [464/500], global_step: 446700, lr: 0.000100, edit distance: 0.001071, exp_rate: 0.952201, exp_rate<=1 : 0.979591, exp_rate<=2 : 0.996509, exp_rate<=3 : 0.998926, loss: 0.000744, avg_reader_cost: 0.02539 s, avg_batch_cost: 0.17255 s, avg_samples: 44.8, ips: 259.62900 samples/s, eta: 5:21:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:30:43] ppocr INFO: epoch: [464/500], global_step: 446800, lr: 0.000100, edit distance: 0.000985, exp_rate: 0.951326, exp_rate<=1 : 0.980437, exp_rate<=2 : 0.995628, exp_rate<=3 : 0.998574, loss: 0.001008, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19410 s, avg_samples: 56.0, ips: 288.51824 samples/s, eta: 5:20:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:31:02] ppocr INFO: epoch: [464/500], global_step: 446900, lr: 0.000100, edit distance: 0.001058, exp_rate: 0.948188, exp_rate<=1 : 0.980529, exp_rate<=2 : 0.995705, exp_rate<=3 : 0.998504, loss: 0.001445, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19183 s, avg_samples: 56.0, ips: 291.92933 samples/s, eta: 5:20:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:31:21] ppocr INFO: epoch: [464/500], global_step: 447000, lr: 0.000100, edit distance: 0.001033, exp_rate: 0.949107, exp_rate<=1 : 0.980940, exp_rate<=2 : 0.995959, exp_rate<=3 : 0.998554, loss: 0.001212, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19011 s, avg_samples: 56.0, ips: 294.57263 samples/s, eta: 5:20:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:31:41] ppocr INFO: epoch: [464/500], global_step: 447100, lr: 0.000100, edit distance: 0.001038, exp_rate: 0.948413, exp_rate<=1 : 0.980353, exp_rate<=2 : 0.995710, exp_rate<=3 : 0.998435, loss: 0.001165, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19791 s, avg_samples: 56.0, ips: 282.96169 samples/s, eta: 5:19:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:31:59] ppocr INFO: epoch: [464/500], global_step: 447200, lr: 0.000100, edit distance: 0.001062, exp_rate: 0.948027, exp_rate<=1 : 0.980136, exp_rate<=2 : 0.995877, exp_rate<=3 : 0.998591, loss: 0.001246, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18209 s, avg_samples: 56.0, ips: 307.54866 samples/s, eta: 5:19:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:32:18] ppocr INFO: epoch: [464/500], global_step: 447300, lr: 0.000100, edit distance: 0.001065, exp_rate: 0.947608, exp_rate<=1 : 0.980091, exp_rate<=2 : 0.995649, exp_rate<=3 : 0.998550, loss: 0.000925, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19021 s, avg_samples: 56.0, ips: 294.41898 samples/s, eta: 5:19:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:32:36] ppocr INFO: epoch: [464/500], global_step: 447400, lr: 0.000100, edit distance: 0.001085, exp_rate: 0.947620, exp_rate<=1 : 0.980381, exp_rate<=2 : 0.995596, exp_rate<=3 : 0.998563, loss: 0.001574, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18344 s, avg_samples: 56.0, ips: 305.28013 samples/s, eta: 5:18:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:32:55] ppocr INFO: epoch: [464/500], global_step: 447500, lr: 0.000100, edit distance: 0.001107, exp_rate: 0.947603, exp_rate<=1 : 0.980533, exp_rate<=2 : 0.995595, exp_rate<=3 : 0.998598, loss: 0.001171, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19092 s, avg_samples: 56.0, ips: 293.31107 samples/s, eta: 5:18:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:33:13] ppocr INFO: epoch: [464/500], global_step: 447600, lr: 0.000100, edit distance: 0.001094, exp_rate: 0.948104, exp_rate<=1 : 0.980722, exp_rate<=2 : 0.995506, exp_rate<=3 : 0.998660, loss: 0.000908, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18062 s, avg_samples: 56.0, ips: 310.04571 samples/s, eta: 5:18:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:33:33] ppocr INFO: epoch: [464/500], global_step: 447700, lr: 0.000100, edit distance: 0.001103, exp_rate: 0.946822, exp_rate<=1 : 0.980068, exp_rate<=2 : 0.995242, exp_rate<=3 : 0.998578, loss: 0.002102, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19363 s, avg_samples: 56.0, ips: 289.21410 samples/s, eta: 5:18:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:33:53] ppocr INFO: epoch: [464/500], global_step: 447800, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.944539, exp_rate<=1 : 0.979132, exp_rate<=2 : 0.995043, exp_rate<=3 : 0.998543, loss: 0.001461, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19935 s, avg_samples: 56.0, ips: 280.91296 samples/s, eta: 5:17:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:34:10] ppocr INFO: epoch: [464/500], global_step: 447900, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.944019, exp_rate<=1 : 0.979025, exp_rate<=2 : 0.995147, exp_rate<=3 : 0.998562, loss: 0.002350, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17581 s, avg_samples: 56.0, ips: 318.52161 samples/s, eta: 5:17:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:34:29] ppocr INFO: epoch: [464/500], global_step: 448000, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.942976, exp_rate<=1 : 0.978540, exp_rate<=2 : 0.995111, exp_rate<=3 : 0.998634, loss: 0.001776, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19089 s, avg_samples: 56.0, ips: 293.36826 samples/s, eta: 5:17:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:34:48] ppocr INFO: epoch: [464/500], global_step: 448100, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.942472, exp_rate<=1 : 0.978204, exp_rate<=2 : 0.995041, exp_rate<=3 : 0.998611, loss: 0.001797, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18192 s, avg_samples: 56.0, ips: 307.82525 samples/s, eta: 5:16:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:35:07] ppocr INFO: epoch: [464/500], global_step: 448200, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.941645, exp_rate<=1 : 0.977882, exp_rate<=2 : 0.994807, exp_rate<=3 : 0.998505, loss: 0.001380, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19017 s, avg_samples: 56.0, ips: 294.48072 samples/s, eta: 5:16:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:35:25] ppocr INFO: epoch: [464/500], global_step: 448300, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.941869, exp_rate<=1 : 0.977862, exp_rate<=2 : 0.994852, exp_rate<=3 : 0.998523, loss: 0.001720, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17893 s, avg_samples: 56.0, ips: 312.97331 samples/s, eta: 5:16:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:35:42] ppocr INFO: epoch: [464/500], global_step: 448400, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.942071, exp_rate<=1 : 0.977939, exp_rate<=2 : 0.994909, exp_rate<=3 : 0.998522, loss: 0.001566, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16896 s, avg_samples: 56.0, ips: 331.43187 samples/s, eta: 5:15:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:36:01] ppocr INFO: epoch: [464/500], global_step: 448500, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.941572, exp_rate<=1 : 0.977724, exp_rate<=2 : 0.994818, exp_rate<=3 : 0.998453, loss: 0.001201, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19368 s, avg_samples: 56.0, ips: 289.13340 samples/s, eta: 5:15:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:36:22] ppocr INFO: epoch: [464/500], global_step: 448600, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.941234, exp_rate<=1 : 0.977595, exp_rate<=2 : 0.994761, exp_rate<=3 : 0.998465, loss: 0.001332, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21315 s, avg_samples: 56.0, ips: 262.72014 samples/s, eta: 5:15:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:36:41] ppocr INFO: epoch: [464/500], global_step: 448700, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.941204, exp_rate<=1 : 0.977574, exp_rate<=2 : 0.994680, exp_rate<=3 : 0.998397, loss: 0.001114, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18683 s, avg_samples: 56.0, ips: 299.74141 samples/s, eta: 5:14:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:37:02] ppocr INFO: epoch: [464/500], global_step: 448800, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.940761, exp_rate<=1 : 0.977425, exp_rate<=2 : 0.994659, exp_rate<=3 : 0.998312, loss: 0.001435, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20995 s, avg_samples: 56.0, ips: 266.73434 samples/s, eta: 5:14:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:37:21] ppocr INFO: epoch: [464/500], global_step: 448900, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.940776, exp_rate<=1 : 0.977549, exp_rate<=2 : 0.994743, exp_rate<=3 : 0.998337, loss: 0.001256, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18845 s, avg_samples: 56.0, ips: 297.15929 samples/s, eta: 5:14:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:37:40] ppocr INFO: epoch: [464/500], global_step: 449000, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.941011, exp_rate<=1 : 0.977632, exp_rate<=2 : 0.994762, exp_rate<=3 : 0.998330, loss: 0.001134, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19105 s, avg_samples: 56.0, ips: 293.12051 samples/s, eta: 5:13:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:37:57] ppocr INFO: epoch: [464/500], global_step: 449100, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.941401, exp_rate<=1 : 0.977870, exp_rate<=2 : 0.994861, exp_rate<=3 : 0.998359, loss: 0.001090, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17300 s, avg_samples: 56.0, ips: 323.69975 samples/s, eta: 5:13:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:38:17] ppocr INFO: epoch: [464/500], global_step: 449200, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.941434, exp_rate<=1 : 0.977796, exp_rate<=2 : 0.994810, exp_rate<=3 : 0.998354, loss: 0.001274, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19717 s, avg_samples: 56.0, ips: 284.01659 samples/s, eta: 5:13:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:38:35] ppocr INFO: epoch: [464/500], global_step: 449300, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.941818, exp_rate<=1 : 0.978023, exp_rate<=2 : 0.994878, exp_rate<=3 : 0.998364, loss: 0.001381, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18190 s, avg_samples: 56.0, ips: 307.85302 samples/s, eta: 5:12:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:38:47] ppocr INFO: epoch: [464/500], global_step: 449360, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.941596, exp_rate<=1 : 0.977927, exp_rate<=2 : 0.994860, exp_rate<=3 : 0.998354, loss: 0.001715, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11481 s, avg_samples: 33.6, ips: 292.65561 samples/s, eta: 5:12:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:38:49] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 02:38:57] ppocr INFO: epoch: [465/500], global_step: 449400, lr: 0.000100, edit distance: 0.001515, exp_rate: 0.943788, exp_rate<=1 : 0.980293, exp_rate<=2 : 0.995744, exp_rate<=3 : 0.998829, loss: 0.001108, avg_reader_cost: 0.02737 s, avg_batch_cost: 0.09948 s, avg_samples: 22.4, ips: 225.16031 samples/s, eta: 5:12:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:39:15] ppocr INFO: epoch: [465/500], global_step: 449500, lr: 0.000100, edit distance: 0.001346, exp_rate: 0.948542, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.997084, exp_rate<=3 : 0.999265, loss: 0.000996, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18075 s, avg_samples: 56.0, ips: 309.82664 samples/s, eta: 5:12:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:39:35] ppocr INFO: epoch: [465/500], global_step: 449600, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.945608, exp_rate<=1 : 0.980086, exp_rate<=2 : 0.996491, exp_rate<=3 : 0.998652, loss: 0.000713, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19746 s, avg_samples: 56.0, ips: 283.59747 samples/s, eta: 5:11:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:39:55] ppocr INFO: epoch: [465/500], global_step: 449700, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.947775, exp_rate<=1 : 0.980194, exp_rate<=2 : 0.996512, exp_rate<=3 : 0.998671, loss: 0.001320, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20484 s, avg_samples: 56.0, ips: 273.38912 samples/s, eta: 5:11:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:40:13] ppocr INFO: epoch: [465/500], global_step: 449800, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.948315, exp_rate<=1 : 0.980159, exp_rate<=2 : 0.996408, exp_rate<=3 : 0.998660, loss: 0.001049, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17757 s, avg_samples: 56.0, ips: 315.36544 samples/s, eta: 5:11:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:40:32] ppocr INFO: epoch: [465/500], global_step: 449900, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.948175, exp_rate<=1 : 0.980409, exp_rate<=2 : 0.996328, exp_rate<=3 : 0.998581, loss: 0.001095, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19350 s, avg_samples: 56.0, ips: 289.40370 samples/s, eta: 5:11:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:40:51] ppocr INFO: epoch: [465/500], global_step: 450000, lr: 0.000100, edit distance: 0.001180, exp_rate: 0.946825, exp_rate<=1 : 0.979810, exp_rate<=2 : 0.995973, exp_rate<=3 : 0.998416, loss: 0.001164, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18783 s, avg_samples: 56.0, ips: 298.14318 samples/s, eta: 5:10:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:41:09] ppocr INFO: epoch: [465/500], global_step: 450100, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.947844, exp_rate<=1 : 0.980302, exp_rate<=2 : 0.996040, exp_rate<=3 : 0.998479, loss: 0.001465, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17694 s, avg_samples: 56.0, ips: 316.49976 samples/s, eta: 5:10:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:41:27] ppocr INFO: epoch: [465/500], global_step: 450200, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.948455, exp_rate<=1 : 0.980476, exp_rate<=2 : 0.995873, exp_rate<=3 : 0.998394, loss: 0.001119, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18715 s, avg_samples: 56.0, ips: 299.21933 samples/s, eta: 5:10:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:41:47] ppocr INFO: epoch: [465/500], global_step: 450300, lr: 0.000100, edit distance: 0.001129, exp_rate: 0.948392, exp_rate<=1 : 0.980628, exp_rate<=2 : 0.995793, exp_rate<=3 : 0.998365, loss: 0.001688, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19230 s, avg_samples: 56.0, ips: 291.21502 samples/s, eta: 5:09:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:42:06] ppocr INFO: epoch: [465/500], global_step: 450400, lr: 0.000100, edit distance: 0.001133, exp_rate: 0.948105, exp_rate<=1 : 0.980474, exp_rate<=2 : 0.995703, exp_rate<=3 : 0.998350, loss: 0.001445, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19191 s, avg_samples: 56.0, ips: 291.80056 samples/s, eta: 5:09:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:42:26] ppocr INFO: epoch: [465/500], global_step: 450500, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.946808, exp_rate<=1 : 0.980020, exp_rate<=2 : 0.995644, exp_rate<=3 : 0.998313, loss: 0.001180, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19915 s, avg_samples: 56.0, ips: 281.19066 samples/s, eta: 5:09:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:42:44] ppocr INFO: epoch: [465/500], global_step: 450600, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.946544, exp_rate<=1 : 0.979700, exp_rate<=2 : 0.995641, exp_rate<=3 : 0.998308, loss: 0.001108, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17967 s, avg_samples: 56.0, ips: 311.67570 samples/s, eta: 5:08:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:43:02] ppocr INFO: epoch: [465/500], global_step: 450700, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.946214, exp_rate<=1 : 0.979758, exp_rate<=2 : 0.995665, exp_rate<=3 : 0.998355, loss: 0.001694, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18127 s, avg_samples: 56.0, ips: 308.93819 samples/s, eta: 5:08:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:43:21] ppocr INFO: epoch: [465/500], global_step: 450800, lr: 0.000100, edit distance: 0.001169, exp_rate: 0.945929, exp_rate<=1 : 0.979684, exp_rate<=2 : 0.995662, exp_rate<=3 : 0.998414, loss: 0.001347, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18626 s, avg_samples: 56.0, ips: 300.64881 samples/s, eta: 5:08:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:43:40] ppocr INFO: epoch: [465/500], global_step: 450900, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.945998, exp_rate<=1 : 0.979759, exp_rate<=2 : 0.995644, exp_rate<=3 : 0.998378, loss: 0.002065, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19765 s, avg_samples: 56.0, ips: 283.32374 samples/s, eta: 5:07:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:44:01] ppocr INFO: epoch: [465/500], global_step: 451000, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.945235, exp_rate<=1 : 0.979459, exp_rate<=2 : 0.995510, exp_rate<=3 : 0.998324, loss: 0.001300, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20489 s, avg_samples: 56.0, ips: 273.31108 samples/s, eta: 5:07:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:44:20] ppocr INFO: epoch: [465/500], global_step: 451100, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.944844, exp_rate<=1 : 0.979370, exp_rate<=2 : 0.995511, exp_rate<=3 : 0.998369, loss: 0.001804, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19442 s, avg_samples: 56.0, ips: 288.03912 samples/s, eta: 5:07:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:44:40] ppocr INFO: epoch: [465/500], global_step: 451200, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.943995, exp_rate<=1 : 0.978900, exp_rate<=2 : 0.995362, exp_rate<=3 : 0.998345, loss: 0.001872, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19336 s, avg_samples: 56.0, ips: 289.60833 samples/s, eta: 5:06:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:44:58] ppocr INFO: epoch: [465/500], global_step: 451300, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.943764, exp_rate<=1 : 0.978694, exp_rate<=2 : 0.995357, exp_rate<=3 : 0.998355, loss: 0.001363, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17928 s, avg_samples: 56.0, ips: 312.35656 samples/s, eta: 5:06:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:45:15] ppocr INFO: epoch: [465/500], global_step: 451400, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.943365, exp_rate<=1 : 0.978474, exp_rate<=2 : 0.995316, exp_rate<=3 : 0.998342, loss: 0.001403, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17466 s, avg_samples: 56.0, ips: 320.61492 samples/s, eta: 5:06:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:45:34] ppocr INFO: epoch: [465/500], global_step: 451500, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.943192, exp_rate<=1 : 0.978359, exp_rate<=2 : 0.995164, exp_rate<=3 : 0.998301, loss: 0.001685, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18619 s, avg_samples: 56.0, ips: 300.76894 samples/s, eta: 5:05:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:45:53] ppocr INFO: epoch: [465/500], global_step: 451600, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.942759, exp_rate<=1 : 0.978152, exp_rate<=2 : 0.995060, exp_rate<=3 : 0.998255, loss: 0.001271, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18783 s, avg_samples: 56.0, ips: 298.13570 samples/s, eta: 5:05:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:46:11] ppocr INFO: epoch: [465/500], global_step: 451700, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.942550, exp_rate<=1 : 0.978067, exp_rate<=2 : 0.995079, exp_rate<=3 : 0.998266, loss: 0.001496, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18556 s, avg_samples: 56.0, ips: 301.79372 samples/s, eta: 5:05:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:46:29] ppocr INFO: epoch: [465/500], global_step: 451800, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.942231, exp_rate<=1 : 0.977890, exp_rate<=2 : 0.995108, exp_rate<=3 : 0.998280, loss: 0.001644, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18109 s, avg_samples: 56.0, ips: 309.23434 samples/s, eta: 5:04:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:46:48] ppocr INFO: epoch: [465/500], global_step: 451900, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.942048, exp_rate<=1 : 0.977767, exp_rate<=2 : 0.995043, exp_rate<=3 : 0.998213, loss: 0.001371, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18890 s, avg_samples: 56.0, ips: 296.45051 samples/s, eta: 5:04:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:47:07] ppocr INFO: epoch: [465/500], global_step: 452000, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.941699, exp_rate<=1 : 0.977676, exp_rate<=2 : 0.994993, exp_rate<=3 : 0.998227, loss: 0.001576, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18264 s, avg_samples: 56.0, ips: 306.60833 samples/s, eta: 5:04:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:47:24] ppocr INFO: epoch: [465/500], global_step: 452100, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.941863, exp_rate<=1 : 0.977779, exp_rate<=2 : 0.994984, exp_rate<=3 : 0.998241, loss: 0.001718, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17472 s, avg_samples: 56.0, ips: 320.50394 samples/s, eta: 5:04:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:47:26] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 02:47:27] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_465 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 02:47:44] ppocr INFO: epoch: [466/500], global_step: 452200, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.952898, exp_rate<=1 : 0.981537, exp_rate<=2 : 0.996088, exp_rate<=3 : 0.998295, loss: 0.001212, avg_reader_cost: 0.03329 s, avg_batch_cost: 0.20124 s, avg_samples: 56.0, ips: 278.27955 samples/s, eta: 5:03:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:48:03] ppocr INFO: epoch: [466/500], global_step: 452300, lr: 0.000100, edit distance: 0.001038, exp_rate: 0.951727, exp_rate<=1 : 0.980819, exp_rate<=2 : 0.996513, exp_rate<=3 : 0.998586, loss: 0.001309, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19137 s, avg_samples: 56.0, ips: 292.63137 samples/s, eta: 5:03:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:48:22] ppocr INFO: epoch: [466/500], global_step: 452400, lr: 0.000100, edit distance: 0.001075, exp_rate: 0.948578, exp_rate<=1 : 0.979480, exp_rate<=2 : 0.996124, exp_rate<=3 : 0.998738, loss: 0.001628, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18356 s, avg_samples: 56.0, ips: 305.07168 samples/s, eta: 5:03:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:48:40] ppocr INFO: epoch: [466/500], global_step: 452500, lr: 0.000100, edit distance: 0.001109, exp_rate: 0.946794, exp_rate<=1 : 0.979312, exp_rate<=2 : 0.995785, exp_rate<=3 : 0.998778, loss: 0.001203, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18733 s, avg_samples: 56.0, ips: 298.94091 samples/s, eta: 5:02:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:49:00] ppocr INFO: epoch: [466/500], global_step: 452600, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.945613, exp_rate<=1 : 0.978964, exp_rate<=2 : 0.995567, exp_rate<=3 : 0.998550, loss: 0.001679, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19292 s, avg_samples: 56.0, ips: 290.27701 samples/s, eta: 5:02:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:49:19] ppocr INFO: epoch: [466/500], global_step: 452700, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.944532, exp_rate<=1 : 0.978783, exp_rate<=2 : 0.995322, exp_rate<=3 : 0.998301, loss: 0.001463, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18915 s, avg_samples: 56.0, ips: 296.06206 samples/s, eta: 5:02:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:49:37] ppocr INFO: epoch: [466/500], global_step: 452800, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.943686, exp_rate<=1 : 0.978318, exp_rate<=2 : 0.994855, exp_rate<=3 : 0.998065, loss: 0.001131, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18474 s, avg_samples: 56.0, ips: 303.12292 samples/s, eta: 5:01:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:49:56] ppocr INFO: epoch: [466/500], global_step: 452900, lr: 0.000100, edit distance: 0.001180, exp_rate: 0.944482, exp_rate<=1 : 0.979011, exp_rate<=2 : 0.995050, exp_rate<=3 : 0.998170, loss: 0.001060, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18801 s, avg_samples: 56.0, ips: 297.85342 samples/s, eta: 5:01:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:50:16] ppocr INFO: epoch: [466/500], global_step: 453000, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.943895, exp_rate<=1 : 0.978821, exp_rate<=2 : 0.995002, exp_rate<=3 : 0.998214, loss: 0.000900, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19488 s, avg_samples: 56.0, ips: 287.35595 samples/s, eta: 5:01:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:50:33] ppocr INFO: epoch: [466/500], global_step: 453100, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.944619, exp_rate<=1 : 0.979183, exp_rate<=2 : 0.995148, exp_rate<=3 : 0.998333, loss: 0.001656, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17843 s, avg_samples: 56.0, ips: 313.84841 samples/s, eta: 5:00:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:50:52] ppocr INFO: epoch: [466/500], global_step: 453200, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.943678, exp_rate<=1 : 0.978830, exp_rate<=2 : 0.995155, exp_rate<=3 : 0.998318, loss: 0.001289, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18808 s, avg_samples: 56.0, ips: 297.74101 samples/s, eta: 5:00:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:51:13] ppocr INFO: epoch: [466/500], global_step: 453300, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.943717, exp_rate<=1 : 0.978778, exp_rate<=2 : 0.995137, exp_rate<=3 : 0.998317, loss: 0.001271, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20333 s, avg_samples: 56.0, ips: 275.41044 samples/s, eta: 5:00:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:51:31] ppocr INFO: epoch: [466/500], global_step: 453400, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.944494, exp_rate<=1 : 0.979088, exp_rate<=2 : 0.995253, exp_rate<=3 : 0.998341, loss: 0.001160, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18370 s, avg_samples: 56.0, ips: 304.84458 samples/s, eta: 4:59:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:51:50] ppocr INFO: epoch: [466/500], global_step: 453500, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.944445, exp_rate<=1 : 0.979170, exp_rate<=2 : 0.995287, exp_rate<=3 : 0.998318, loss: 0.001359, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19215 s, avg_samples: 56.0, ips: 291.43766 samples/s, eta: 4:59:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:52:09] ppocr INFO: epoch: [466/500], global_step: 453600, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.944709, exp_rate<=1 : 0.979399, exp_rate<=2 : 0.995383, exp_rate<=3 : 0.998365, loss: 0.001337, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18425 s, avg_samples: 56.0, ips: 303.94188 samples/s, eta: 4:59:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:52:27] ppocr INFO: epoch: [466/500], global_step: 453700, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.944727, exp_rate<=1 : 0.979403, exp_rate<=2 : 0.995368, exp_rate<=3 : 0.998358, loss: 0.001588, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17935 s, avg_samples: 56.0, ips: 312.24718 samples/s, eta: 4:58:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:52:45] ppocr INFO: epoch: [466/500], global_step: 453800, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.943877, exp_rate<=1 : 0.979023, exp_rate<=2 : 0.995335, exp_rate<=3 : 0.998338, loss: 0.001498, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18740 s, avg_samples: 56.0, ips: 298.82911 samples/s, eta: 4:58:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:53:05] ppocr INFO: epoch: [466/500], global_step: 453900, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.943302, exp_rate<=1 : 0.978708, exp_rate<=2 : 0.995348, exp_rate<=3 : 0.998363, loss: 0.002107, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19176 s, avg_samples: 56.0, ips: 292.02699 samples/s, eta: 4:58:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:53:23] ppocr INFO: epoch: [466/500], global_step: 454000, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.943194, exp_rate<=1 : 0.978545, exp_rate<=2 : 0.995334, exp_rate<=3 : 0.998404, loss: 0.001252, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18758 s, avg_samples: 56.0, ips: 298.53762 samples/s, eta: 4:58:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:53:42] ppocr INFO: epoch: [466/500], global_step: 454100, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.943275, exp_rate<=1 : 0.978694, exp_rate<=2 : 0.995366, exp_rate<=3 : 0.998424, loss: 0.001195, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18257 s, avg_samples: 56.0, ips: 306.73720 samples/s, eta: 4:57:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:54:00] ppocr INFO: epoch: [466/500], global_step: 454200, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.943316, exp_rate<=1 : 0.978652, exp_rate<=2 : 0.995353, exp_rate<=3 : 0.998403, loss: 0.001436, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18825 s, avg_samples: 56.0, ips: 297.47614 samples/s, eta: 4:57:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:54:21] ppocr INFO: epoch: [466/500], global_step: 454300, lr: 0.000100, edit distance: 0.001238, exp_rate: 0.942349, exp_rate<=1 : 0.978364, exp_rate<=2 : 0.995223, exp_rate<=3 : 0.998342, loss: 0.001518, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20904 s, avg_samples: 56.0, ips: 267.89296 samples/s, eta: 4:57:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:54:41] ppocr INFO: epoch: [466/500], global_step: 454400, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942212, exp_rate<=1 : 0.978206, exp_rate<=2 : 0.995127, exp_rate<=3 : 0.998314, loss: 0.001196, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19619 s, avg_samples: 56.0, ips: 285.43168 samples/s, eta: 4:56:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:55:00] ppocr INFO: epoch: [466/500], global_step: 454500, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.942451, exp_rate<=1 : 0.978255, exp_rate<=2 : 0.995107, exp_rate<=3 : 0.998317, loss: 0.001060, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18715 s, avg_samples: 56.0, ips: 299.22492 samples/s, eta: 4:56:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:55:19] ppocr INFO: epoch: [466/500], global_step: 454600, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942451, exp_rate<=1 : 0.978352, exp_rate<=2 : 0.995183, exp_rate<=3 : 0.998345, loss: 0.001784, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18811 s, avg_samples: 56.0, ips: 297.70448 samples/s, eta: 4:56:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:55:37] ppocr INFO: epoch: [466/500], global_step: 454700, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.942197, exp_rate<=1 : 0.978160, exp_rate<=2 : 0.995096, exp_rate<=3 : 0.998295, loss: 0.001508, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18485 s, avg_samples: 56.0, ips: 302.95262 samples/s, eta: 4:55:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:55:55] ppocr INFO: epoch: [466/500], global_step: 454800, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.942471, exp_rate<=1 : 0.978178, exp_rate<=2 : 0.995115, exp_rate<=3 : 0.998294, loss: 0.001371, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17406 s, avg_samples: 56.0, ips: 321.72043 samples/s, eta: 4:55:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:56:02] ppocr INFO: epoch: [466/500], global_step: 454840, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.942492, exp_rate<=1 : 0.978242, exp_rate<=2 : 0.995145, exp_rate<=3 : 0.998319, loss: 0.001461, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07825 s, avg_samples: 22.4, ips: 286.27450 samples/s, eta: 4:55:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:56:05] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 02:56:16] ppocr INFO: epoch: [467/500], global_step: 454900, lr: 0.000100, edit distance: 0.000981, exp_rate: 0.948015, exp_rate<=1 : 0.978074, exp_rate<=2 : 0.995949, exp_rate<=3 : 0.999599, loss: 0.000809, avg_reader_cost: 0.02617 s, avg_batch_cost: 0.14007 s, avg_samples: 33.6, ips: 239.87764 samples/s, eta: 4:55:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:56:36] ppocr INFO: epoch: [467/500], global_step: 455000, lr: 0.000100, edit distance: 0.000913, exp_rate: 0.950819, exp_rate<=1 : 0.981416, exp_rate<=2 : 0.996081, exp_rate<=3 : 0.999104, loss: 0.001011, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19935 s, avg_samples: 56.0, ips: 280.91232 samples/s, eta: 4:54:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:56:55] ppocr INFO: epoch: [467/500], global_step: 455100, lr: 0.000100, edit distance: 0.000941, exp_rate: 0.952345, exp_rate<=1 : 0.980526, exp_rate<=2 : 0.996120, exp_rate<=3 : 0.998859, loss: 0.001099, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18355 s, avg_samples: 56.0, ips: 305.08874 samples/s, eta: 4:54:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:57:14] ppocr INFO: epoch: [467/500], global_step: 455200, lr: 0.000100, edit distance: 0.000994, exp_rate: 0.947860, exp_rate<=1 : 0.979555, exp_rate<=2 : 0.995516, exp_rate<=3 : 0.998536, loss: 0.001314, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.74743 samples/s, eta: 4:54:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:57:33] ppocr INFO: epoch: [467/500], global_step: 455300, lr: 0.000100, edit distance: 0.001044, exp_rate: 0.945757, exp_rate<=1 : 0.978406, exp_rate<=2 : 0.995413, exp_rate<=3 : 0.998418, loss: 0.001423, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18794 s, avg_samples: 56.0, ips: 297.96185 samples/s, eta: 4:53:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:57:52] ppocr INFO: epoch: [467/500], global_step: 455400, lr: 0.000100, edit distance: 0.001047, exp_rate: 0.945089, exp_rate<=1 : 0.977772, exp_rate<=2 : 0.995114, exp_rate<=3 : 0.998360, loss: 0.001504, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18981 s, avg_samples: 56.0, ips: 295.03187 samples/s, eta: 4:53:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:58:10] ppocr INFO: epoch: [467/500], global_step: 455500, lr: 0.000100, edit distance: 0.001051, exp_rate: 0.946085, exp_rate<=1 : 0.978350, exp_rate<=2 : 0.995257, exp_rate<=3 : 0.998369, loss: 0.000989, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18523 s, avg_samples: 56.0, ips: 302.33028 samples/s, eta: 4:53:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:58:30] ppocr INFO: epoch: [467/500], global_step: 455600, lr: 0.000100, edit distance: 0.001038, exp_rate: 0.945916, exp_rate<=1 : 0.978536, exp_rate<=2 : 0.995434, exp_rate<=3 : 0.998426, loss: 0.001513, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19552 s, avg_samples: 56.0, ips: 286.41456 samples/s, eta: 4:52:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:58:49] ppocr INFO: epoch: [467/500], global_step: 455700, lr: 0.000100, edit distance: 0.001073, exp_rate: 0.944757, exp_rate<=1 : 0.977932, exp_rate<=2 : 0.995359, exp_rate<=3 : 0.998347, loss: 0.001177, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18946 s, avg_samples: 56.0, ips: 295.57839 samples/s, eta: 4:52:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:59:06] ppocr INFO: epoch: [467/500], global_step: 455800, lr: 0.000100, edit distance: 0.001063, exp_rate: 0.945480, exp_rate<=1 : 0.978451, exp_rate<=2 : 0.995503, exp_rate<=3 : 0.998448, loss: 0.001277, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17327 s, avg_samples: 56.0, ips: 323.20029 samples/s, eta: 4:52:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:59:24] ppocr INFO: epoch: [467/500], global_step: 455900, lr: 0.000100, edit distance: 0.001067, exp_rate: 0.946276, exp_rate<=1 : 0.978874, exp_rate<=2 : 0.995519, exp_rate<=3 : 0.998453, loss: 0.001797, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18062 s, avg_samples: 56.0, ips: 310.04005 samples/s, eta: 4:51:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 02:59:43] ppocr INFO: epoch: [467/500], global_step: 456000, lr: 0.000100, edit distance: 0.001067, exp_rate: 0.946219, exp_rate<=1 : 0.978965, exp_rate<=2 : 0.995489, exp_rate<=3 : 0.998447, loss: 0.001167, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18446 s, avg_samples: 56.0, ips: 303.58113 samples/s, eta: 4:51:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:00:03] ppocr INFO: epoch: [467/500], global_step: 456100, lr: 0.000100, edit distance: 0.001099, exp_rate: 0.945418, exp_rate<=1 : 0.978988, exp_rate<=2 : 0.995561, exp_rate<=3 : 0.998483, loss: 0.002628, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19779 s, avg_samples: 56.0, ips: 283.12855 samples/s, eta: 4:51:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:00:21] ppocr INFO: epoch: [467/500], global_step: 456200, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.944869, exp_rate<=1 : 0.978685, exp_rate<=2 : 0.995347, exp_rate<=3 : 0.998393, loss: 0.001862, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18788 s, avg_samples: 56.0, ips: 298.06803 samples/s, eta: 4:51:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:00:40] ppocr INFO: epoch: [467/500], global_step: 456300, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.944319, exp_rate<=1 : 0.978491, exp_rate<=2 : 0.995310, exp_rate<=3 : 0.998355, loss: 0.000883, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18781 s, avg_samples: 56.0, ips: 298.17711 samples/s, eta: 4:50:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:01:00] ppocr INFO: epoch: [467/500], global_step: 456400, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.944476, exp_rate<=1 : 0.978612, exp_rate<=2 : 0.995305, exp_rate<=3 : 0.998373, loss: 0.001370, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19915 s, avg_samples: 56.0, ips: 281.20207 samples/s, eta: 4:50:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:01:19] ppocr INFO: epoch: [467/500], global_step: 456500, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.944470, exp_rate<=1 : 0.978427, exp_rate<=2 : 0.995216, exp_rate<=3 : 0.998352, loss: 0.001410, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18828 s, avg_samples: 56.0, ips: 297.43098 samples/s, eta: 4:50:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:01:38] ppocr INFO: epoch: [467/500], global_step: 456600, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.944250, exp_rate<=1 : 0.978359, exp_rate<=2 : 0.995147, exp_rate<=3 : 0.998369, loss: 0.001140, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18565 s, avg_samples: 56.0, ips: 301.64013 samples/s, eta: 4:49:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:01:57] ppocr INFO: epoch: [467/500], global_step: 456700, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.944289, exp_rate<=1 : 0.978461, exp_rate<=2 : 0.995163, exp_rate<=3 : 0.998369, loss: 0.001274, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19626 s, avg_samples: 56.0, ips: 285.33612 samples/s, eta: 4:49:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:02:17] ppocr INFO: epoch: [467/500], global_step: 456800, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.943763, exp_rate<=1 : 0.978236, exp_rate<=2 : 0.995090, exp_rate<=3 : 0.998350, loss: 0.001486, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19876 s, avg_samples: 56.0, ips: 281.75021 samples/s, eta: 4:49:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:02:37] ppocr INFO: epoch: [467/500], global_step: 456900, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.943915, exp_rate<=1 : 0.978331, exp_rate<=2 : 0.995125, exp_rate<=3 : 0.998340, loss: 0.001591, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19582 s, avg_samples: 56.0, ips: 285.97693 samples/s, eta: 4:48:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:02:55] ppocr INFO: epoch: [467/500], global_step: 457000, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.944181, exp_rate<=1 : 0.978492, exp_rate<=2 : 0.995184, exp_rate<=3 : 0.998388, loss: 0.001094, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18032 s, avg_samples: 56.0, ips: 310.56205 samples/s, eta: 4:48:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:03:13] ppocr INFO: epoch: [467/500], global_step: 457100, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.944462, exp_rate<=1 : 0.978554, exp_rate<=2 : 0.995244, exp_rate<=3 : 0.998389, loss: 0.001029, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17868 s, avg_samples: 56.0, ips: 313.41316 samples/s, eta: 4:48:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:03:32] ppocr INFO: epoch: [467/500], global_step: 457200, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.944270, exp_rate<=1 : 0.978479, exp_rate<=2 : 0.995180, exp_rate<=3 : 0.998357, loss: 0.001026, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19094 s, avg_samples: 56.0, ips: 293.28980 samples/s, eta: 4:47:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:03:50] ppocr INFO: epoch: [467/500], global_step: 457300, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.944429, exp_rate<=1 : 0.978486, exp_rate<=2 : 0.995169, exp_rate<=3 : 0.998347, loss: 0.001562, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18400 s, avg_samples: 56.0, ips: 304.35409 samples/s, eta: 4:47:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:04:08] ppocr INFO: epoch: [467/500], global_step: 457400, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.944211, exp_rate<=1 : 0.978433, exp_rate<=2 : 0.995168, exp_rate<=3 : 0.998333, loss: 0.001354, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18179 s, avg_samples: 56.0, ips: 308.05377 samples/s, eta: 4:47:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:04:26] ppocr INFO: epoch: [467/500], global_step: 457500, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.944337, exp_rate<=1 : 0.978496, exp_rate<=2 : 0.995221, exp_rate<=3 : 0.998344, loss: 0.001200, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17680 s, avg_samples: 56.0, ips: 316.73466 samples/s, eta: 4:46:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:04:41] ppocr INFO: epoch: [467/500], global_step: 457580, lr: 0.000100, edit distance: 0.001180, exp_rate: 0.944361, exp_rate<=1 : 0.978503, exp_rate<=2 : 0.995183, exp_rate<=3 : 0.998327, loss: 0.001185, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14469 s, avg_samples: 44.8, ips: 309.62414 samples/s, eta: 4:46:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:04:43] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 03:04:47] ppocr INFO: epoch: [468/500], global_step: 457600, lr: 0.000100, edit distance: 0.000910, exp_rate: 0.940476, exp_rate<=1 : 0.980087, exp_rate<=2 : 0.997834, exp_rate<=3 : 1.000000, loss: 0.001668, avg_reader_cost: 0.02544 s, avg_batch_cost: 0.06630 s, avg_samples: 11.2, ips: 168.92342 samples/s, eta: 4:46:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:05:06] ppocr INFO: epoch: [468/500], global_step: 457700, lr: 0.000100, edit distance: 0.001121, exp_rate: 0.946909, exp_rate<=1 : 0.981574, exp_rate<=2 : 0.995362, exp_rate<=3 : 0.999039, loss: 0.001703, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18824 s, avg_samples: 56.0, ips: 297.48960 samples/s, eta: 4:46:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:05:26] ppocr INFO: epoch: [468/500], global_step: 457800, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.943896, exp_rate<=1 : 0.977859, exp_rate<=2 : 0.994131, exp_rate<=3 : 0.998269, loss: 0.000912, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20281 s, avg_samples: 56.0, ips: 276.11765 samples/s, eta: 4:45:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:05:46] ppocr INFO: epoch: [468/500], global_step: 457900, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.945871, exp_rate<=1 : 0.979093, exp_rate<=2 : 0.994723, exp_rate<=3 : 0.998287, loss: 0.001586, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19214 s, avg_samples: 56.0, ips: 291.45094 samples/s, eta: 4:45:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:06:05] ppocr INFO: epoch: [468/500], global_step: 458000, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.945990, exp_rate<=1 : 0.979195, exp_rate<=2 : 0.994833, exp_rate<=3 : 0.998347, loss: 0.000877, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19507 s, avg_samples: 56.0, ips: 287.07939 samples/s, eta: 4:45:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:06:23] ppocr INFO: epoch: [468/500], global_step: 458100, lr: 0.000100, edit distance: 0.001069, exp_rate: 0.947461, exp_rate<=1 : 0.979704, exp_rate<=2 : 0.995087, exp_rate<=3 : 0.998472, loss: 0.001313, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18190 s, avg_samples: 56.0, ips: 307.86273 samples/s, eta: 4:45:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:06:42] ppocr INFO: epoch: [468/500], global_step: 458200, lr: 0.000100, edit distance: 0.001074, exp_rate: 0.947173, exp_rate<=1 : 0.979805, exp_rate<=2 : 0.995019, exp_rate<=3 : 0.998516, loss: 0.001535, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18803 s, avg_samples: 56.0, ips: 297.82201 samples/s, eta: 4:44:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:06:59] ppocr INFO: epoch: [468/500], global_step: 458300, lr: 0.000100, edit distance: 0.001094, exp_rate: 0.947809, exp_rate<=1 : 0.980071, exp_rate<=2 : 0.995156, exp_rate<=3 : 0.998542, loss: 0.001433, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17284 s, avg_samples: 56.0, ips: 323.99814 samples/s, eta: 4:44:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:07:17] ppocr INFO: epoch: [468/500], global_step: 458400, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.947700, exp_rate<=1 : 0.980218, exp_rate<=2 : 0.995337, exp_rate<=3 : 0.998664, loss: 0.002127, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17870 s, avg_samples: 56.0, ips: 313.37832 samples/s, eta: 4:44:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:07:37] ppocr INFO: epoch: [468/500], global_step: 458500, lr: 0.000100, edit distance: 0.001092, exp_rate: 0.946890, exp_rate<=1 : 0.979896, exp_rate<=2 : 0.995389, exp_rate<=3 : 0.998651, loss: 0.001767, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20084 s, avg_samples: 56.0, ips: 278.82422 samples/s, eta: 4:43:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:07:55] ppocr INFO: epoch: [468/500], global_step: 458600, lr: 0.000100, edit distance: 0.001097, exp_rate: 0.946836, exp_rate<=1 : 0.980017, exp_rate<=2 : 0.995474, exp_rate<=3 : 0.998608, loss: 0.001471, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17895 s, avg_samples: 56.0, ips: 312.93630 samples/s, eta: 4:43:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:08:14] ppocr INFO: epoch: [468/500], global_step: 458700, lr: 0.000100, edit distance: 0.001105, exp_rate: 0.946534, exp_rate<=1 : 0.980040, exp_rate<=2 : 0.995473, exp_rate<=3 : 0.998596, loss: 0.001189, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18310 s, avg_samples: 56.0, ips: 305.83864 samples/s, eta: 4:43:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:08:32] ppocr INFO: epoch: [468/500], global_step: 458800, lr: 0.000100, edit distance: 0.001088, exp_rate: 0.946480, exp_rate<=1 : 0.979982, exp_rate<=2 : 0.995394, exp_rate<=3 : 0.998584, loss: 0.000773, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18648 s, avg_samples: 56.0, ips: 300.29322 samples/s, eta: 4:42:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:08:51] ppocr INFO: epoch: [468/500], global_step: 458900, lr: 0.000100, edit distance: 0.001106, exp_rate: 0.946605, exp_rate<=1 : 0.979959, exp_rate<=2 : 0.995259, exp_rate<=3 : 0.998547, loss: 0.001492, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18948 s, avg_samples: 56.0, ips: 295.54687 samples/s, eta: 4:42:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:09:10] ppocr INFO: epoch: [468/500], global_step: 459000, lr: 0.000100, edit distance: 0.001105, exp_rate: 0.946365, exp_rate<=1 : 0.979874, exp_rate<=2 : 0.995244, exp_rate<=3 : 0.998549, loss: 0.001199, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18622 s, avg_samples: 56.0, ips: 300.71871 samples/s, eta: 4:42:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:09:29] ppocr INFO: epoch: [468/500], global_step: 459100, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.945857, exp_rate<=1 : 0.979631, exp_rate<=2 : 0.995162, exp_rate<=3 : 0.998457, loss: 0.001683, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18886 s, avg_samples: 56.0, ips: 296.52104 samples/s, eta: 4:41:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:09:47] ppocr INFO: epoch: [468/500], global_step: 459200, lr: 0.000100, edit distance: 0.001124, exp_rate: 0.945316, exp_rate<=1 : 0.979402, exp_rate<=2 : 0.995194, exp_rate<=3 : 0.998461, loss: 0.001901, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18516 s, avg_samples: 56.0, ips: 302.44908 samples/s, eta: 4:41:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:10:06] ppocr INFO: epoch: [468/500], global_step: 459300, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.945162, exp_rate<=1 : 0.979408, exp_rate<=2 : 0.995239, exp_rate<=3 : 0.998498, loss: 0.001930, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18367 s, avg_samples: 56.0, ips: 304.88673 samples/s, eta: 4:41:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:10:23] ppocr INFO: epoch: [468/500], global_step: 459400, lr: 0.000100, edit distance: 0.001151, exp_rate: 0.944508, exp_rate<=1 : 0.979455, exp_rate<=2 : 0.995272, exp_rate<=3 : 0.998508, loss: 0.001918, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17458 s, avg_samples: 56.0, ips: 320.77645 samples/s, eta: 4:40:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:10:40] ppocr INFO: epoch: [468/500], global_step: 459500, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.944302, exp_rate<=1 : 0.979341, exp_rate<=2 : 0.995312, exp_rate<=3 : 0.998503, loss: 0.001584, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17110 s, avg_samples: 56.0, ips: 327.29624 samples/s, eta: 4:40:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:11:00] ppocr INFO: epoch: [468/500], global_step: 459600, lr: 0.000100, edit distance: 0.001177, exp_rate: 0.943340, exp_rate<=1 : 0.978904, exp_rate<=2 : 0.995229, exp_rate<=3 : 0.998418, loss: 0.002073, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19504 s, avg_samples: 56.0, ips: 287.11881 samples/s, eta: 4:40:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:11:19] ppocr INFO: epoch: [468/500], global_step: 459700, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.942909, exp_rate<=1 : 0.978632, exp_rate<=2 : 0.995210, exp_rate<=3 : 0.998388, loss: 0.001899, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19079 s, avg_samples: 56.0, ips: 293.51252 samples/s, eta: 4:39:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:11:38] ppocr INFO: epoch: [468/500], global_step: 459800, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.942808, exp_rate<=1 : 0.978684, exp_rate<=2 : 0.995226, exp_rate<=3 : 0.998414, loss: 0.001730, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18570 s, avg_samples: 56.0, ips: 301.56924 samples/s, eta: 4:39:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:11:58] ppocr INFO: epoch: [468/500], global_step: 459900, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.941988, exp_rate<=1 : 0.978338, exp_rate<=2 : 0.995142, exp_rate<=3 : 0.998401, loss: 0.001403, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19887 s, avg_samples: 56.0, ips: 281.58589 samples/s, eta: 4:39:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:12:16] ppocr INFO: epoch: [468/500], global_step: 460000, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.941711, exp_rate<=1 : 0.978083, exp_rate<=2 : 0.995043, exp_rate<=3 : 0.998329, loss: 0.001664, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18750 s, avg_samples: 56.0, ips: 298.66396 samples/s, eta: 4:38:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:12:37] ppocr INFO: epoch: [468/500], global_step: 460100, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.941158, exp_rate<=1 : 0.977906, exp_rate<=2 : 0.995006, exp_rate<=3 : 0.998326, loss: 0.001628, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20146 s, avg_samples: 56.0, ips: 277.97098 samples/s, eta: 4:38:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:12:56] ppocr INFO: epoch: [468/500], global_step: 460200, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.940582, exp_rate<=1 : 0.977684, exp_rate<=2 : 0.994979, exp_rate<=3 : 0.998328, loss: 0.001373, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19264 s, avg_samples: 56.0, ips: 290.70015 samples/s, eta: 4:38:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:13:15] ppocr INFO: epoch: [468/500], global_step: 460300, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.940319, exp_rate<=1 : 0.977514, exp_rate<=2 : 0.995005, exp_rate<=3 : 0.998330, loss: 0.001547, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19579 s, avg_samples: 56.0, ips: 286.01396 samples/s, eta: 4:38:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:13:19] ppocr INFO: epoch: [468/500], global_step: 460320, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.940459, exp_rate<=1 : 0.977555, exp_rate<=2 : 0.995007, exp_rate<=3 : 0.998339, loss: 0.001143, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03265 s, avg_samples: 11.2, ips: 343.04197 samples/s, eta: 4:37:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:13:21] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 03:13:36] ppocr INFO: epoch: [469/500], global_step: 460400, lr: 0.000100, edit distance: 0.000930, exp_rate: 0.952759, exp_rate<=1 : 0.979645, exp_rate<=2 : 0.994724, exp_rate<=3 : 0.997502, loss: 0.000937, avg_reader_cost: 0.02523 s, avg_batch_cost: 0.17536 s, avg_samples: 44.8, ips: 255.47016 samples/s, eta: 4:37:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:13:56] ppocr INFO: epoch: [469/500], global_step: 460500, lr: 0.000100, edit distance: 0.000987, exp_rate: 0.954443, exp_rate<=1 : 0.980860, exp_rate<=2 : 0.995134, exp_rate<=3 : 0.998324, loss: 0.000947, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19435 s, avg_samples: 56.0, ips: 288.13461 samples/s, eta: 4:37:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:14:13] ppocr INFO: epoch: [469/500], global_step: 460600, lr: 0.000100, edit distance: 0.000921, exp_rate: 0.954683, exp_rate<=1 : 0.981193, exp_rate<=2 : 0.995674, exp_rate<=3 : 0.998680, loss: 0.001319, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17740 s, avg_samples: 56.0, ips: 315.67405 samples/s, eta: 4:37:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:14:32] ppocr INFO: epoch: [469/500], global_step: 460700, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.949862, exp_rate<=1 : 0.979629, exp_rate<=2 : 0.994597, exp_rate<=3 : 0.998056, loss: 0.001890, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18162 s, avg_samples: 56.0, ips: 308.33116 samples/s, eta: 4:36:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:14:50] ppocr INFO: epoch: [469/500], global_step: 460800, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.947896, exp_rate<=1 : 0.979277, exp_rate<=2 : 0.994599, exp_rate<=3 : 0.998123, loss: 0.001741, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18116 s, avg_samples: 56.0, ips: 309.12574 samples/s, eta: 4:36:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:15:09] ppocr INFO: epoch: [469/500], global_step: 460900, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.946960, exp_rate<=1 : 0.978855, exp_rate<=2 : 0.994468, exp_rate<=3 : 0.998153, loss: 0.001204, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18975 s, avg_samples: 56.0, ips: 295.11998 samples/s, eta: 4:36:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:15:28] ppocr INFO: epoch: [469/500], global_step: 461000, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.946362, exp_rate<=1 : 0.978894, exp_rate<=2 : 0.994460, exp_rate<=3 : 0.998189, loss: 0.000961, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19292 s, avg_samples: 56.0, ips: 290.28267 samples/s, eta: 4:35:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:15:48] ppocr INFO: epoch: [469/500], global_step: 461100, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.945891, exp_rate<=1 : 0.979026, exp_rate<=2 : 0.994563, exp_rate<=3 : 0.998146, loss: 0.001602, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20061 s, avg_samples: 56.0, ips: 279.15195 samples/s, eta: 4:35:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:16:07] ppocr INFO: epoch: [469/500], global_step: 461200, lr: 0.000100, edit distance: 0.001183, exp_rate: 0.946500, exp_rate<=1 : 0.979210, exp_rate<=2 : 0.994716, exp_rate<=3 : 0.998256, loss: 0.001380, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18744 s, avg_samples: 56.0, ips: 298.76921 samples/s, eta: 4:35:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:16:26] ppocr INFO: epoch: [469/500], global_step: 461300, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.946620, exp_rate<=1 : 0.979483, exp_rate<=2 : 0.994910, exp_rate<=3 : 0.998289, loss: 0.001502, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19032 s, avg_samples: 56.0, ips: 294.24819 samples/s, eta: 4:34:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:16:44] ppocr INFO: epoch: [469/500], global_step: 461400, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.947261, exp_rate<=1 : 0.979749, exp_rate<=2 : 0.995029, exp_rate<=3 : 0.998315, loss: 0.001137, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17919 s, avg_samples: 56.0, ips: 312.51471 samples/s, eta: 4:34:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:17:02] ppocr INFO: epoch: [469/500], global_step: 461500, lr: 0.000100, edit distance: 0.001123, exp_rate: 0.947739, exp_rate<=1 : 0.980075, exp_rate<=2 : 0.995202, exp_rate<=3 : 0.998442, loss: 0.001305, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17954 s, avg_samples: 56.0, ips: 311.90992 samples/s, eta: 4:34:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:17:21] ppocr INFO: epoch: [469/500], global_step: 461600, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.947507, exp_rate<=1 : 0.979971, exp_rate<=2 : 0.995264, exp_rate<=3 : 0.998514, loss: 0.001166, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18704 s, avg_samples: 56.0, ips: 299.39893 samples/s, eta: 4:33:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:17:41] ppocr INFO: epoch: [469/500], global_step: 461700, lr: 0.000100, edit distance: 0.001129, exp_rate: 0.946892, exp_rate<=1 : 0.979833, exp_rate<=2 : 0.995221, exp_rate<=3 : 0.998459, loss: 0.000946, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19926 s, avg_samples: 56.0, ips: 281.04169 samples/s, eta: 4:33:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:17:59] ppocr INFO: epoch: [469/500], global_step: 461800, lr: 0.000100, edit distance: 0.001132, exp_rate: 0.946695, exp_rate<=1 : 0.979699, exp_rate<=2 : 0.995217, exp_rate<=3 : 0.998460, loss: 0.001510, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18641 s, avg_samples: 56.0, ips: 300.40798 samples/s, eta: 4:33:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:18:19] ppocr INFO: epoch: [469/500], global_step: 461900, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.946218, exp_rate<=1 : 0.979445, exp_rate<=2 : 0.995171, exp_rate<=3 : 0.998376, loss: 0.001532, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19588 s, avg_samples: 56.0, ips: 285.88528 samples/s, eta: 4:32:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:18:37] ppocr INFO: epoch: [469/500], global_step: 462000, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.945915, exp_rate<=1 : 0.979384, exp_rate<=2 : 0.995209, exp_rate<=3 : 0.998392, loss: 0.001862, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18350 s, avg_samples: 56.0, ips: 305.18187 samples/s, eta: 4:32:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:18:57] ppocr INFO: epoch: [469/500], global_step: 462100, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.944912, exp_rate<=1 : 0.978754, exp_rate<=2 : 0.994953, exp_rate<=3 : 0.998214, loss: 0.001360, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19720 s, avg_samples: 56.0, ips: 283.97326 samples/s, eta: 4:32:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:19:15] ppocr INFO: epoch: [469/500], global_step: 462200, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.944793, exp_rate<=1 : 0.978793, exp_rate<=2 : 0.994950, exp_rate<=3 : 0.998177, loss: 0.001040, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17919 s, avg_samples: 56.0, ips: 312.52234 samples/s, eta: 4:31:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:19:34] ppocr INFO: epoch: [469/500], global_step: 462300, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.944796, exp_rate<=1 : 0.978671, exp_rate<=2 : 0.994893, exp_rate<=3 : 0.998158, loss: 0.001300, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19138 s, avg_samples: 56.0, ips: 292.61412 samples/s, eta: 4:31:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:19:53] ppocr INFO: epoch: [469/500], global_step: 462400, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.944489, exp_rate<=1 : 0.978261, exp_rate<=2 : 0.994808, exp_rate<=3 : 0.998130, loss: 0.000915, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18508 s, avg_samples: 56.0, ips: 302.57924 samples/s, eta: 4:31:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:20:11] ppocr INFO: epoch: [469/500], global_step: 462500, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.944443, exp_rate<=1 : 0.978266, exp_rate<=2 : 0.994899, exp_rate<=3 : 0.998174, loss: 0.001622, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18376 s, avg_samples: 56.0, ips: 304.73850 samples/s, eta: 4:31:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:20:29] ppocr INFO: epoch: [469/500], global_step: 462600, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.944234, exp_rate<=1 : 0.978165, exp_rate<=2 : 0.994846, exp_rate<=3 : 0.998191, loss: 0.001163, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17642 s, avg_samples: 56.0, ips: 317.42168 samples/s, eta: 4:30:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:20:47] ppocr INFO: epoch: [469/500], global_step: 462700, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.944219, exp_rate<=1 : 0.978073, exp_rate<=2 : 0.994912, exp_rate<=3 : 0.998218, loss: 0.001300, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18516 s, avg_samples: 56.0, ips: 302.44630 samples/s, eta: 4:30:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:21:06] ppocr INFO: epoch: [469/500], global_step: 462800, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.944135, exp_rate<=1 : 0.978022, exp_rate<=2 : 0.994890, exp_rate<=3 : 0.998224, loss: 0.001729, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18525 s, avg_samples: 56.0, ips: 302.29948 samples/s, eta: 4:30:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:21:27] ppocr INFO: epoch: [469/500], global_step: 462900, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.943311, exp_rate<=1 : 0.977694, exp_rate<=2 : 0.994838, exp_rate<=3 : 0.998199, loss: 0.001425, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21346 s, avg_samples: 56.0, ips: 262.34054 samples/s, eta: 4:29:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:21:47] ppocr INFO: epoch: [469/500], global_step: 463000, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.943256, exp_rate<=1 : 0.977689, exp_rate<=2 : 0.994808, exp_rate<=3 : 0.998194, loss: 0.001218, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19635 s, avg_samples: 56.0, ips: 285.20458 samples/s, eta: 4:29:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:21:58] ppocr INFO: epoch: [469/500], global_step: 463060, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.943050, exp_rate<=1 : 0.977546, exp_rate<=2 : 0.994799, exp_rate<=3 : 0.998191, loss: 0.001599, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10903 s, avg_samples: 33.6, ips: 308.17202 samples/s, eta: 4:29:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:22:00] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 03:22:08] ppocr INFO: epoch: [470/500], global_step: 463100, lr: 0.000100, edit distance: 0.000815, exp_rate: 0.963388, exp_rate<=1 : 0.986063, exp_rate<=2 : 0.996917, exp_rate<=3 : 0.999169, loss: 0.001191, avg_reader_cost: 0.02525 s, avg_batch_cost: 0.09936 s, avg_samples: 22.4, ips: 225.44708 samples/s, eta: 4:29:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:22:26] ppocr INFO: epoch: [470/500], global_step: 463200, lr: 0.000100, edit distance: 0.001072, exp_rate: 0.950860, exp_rate<=1 : 0.981406, exp_rate<=2 : 0.995587, exp_rate<=3 : 0.998023, loss: 0.001308, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18526 s, avg_samples: 56.0, ips: 302.27700 samples/s, eta: 4:28:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:22:45] ppocr INFO: epoch: [470/500], global_step: 463300, lr: 0.000100, edit distance: 0.001126, exp_rate: 0.948283, exp_rate<=1 : 0.981091, exp_rate<=2 : 0.995497, exp_rate<=3 : 0.998342, loss: 0.001240, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18426 s, avg_samples: 56.0, ips: 303.91383 samples/s, eta: 4:28:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:23:04] ppocr INFO: epoch: [470/500], global_step: 463400, lr: 0.000100, edit distance: 0.001134, exp_rate: 0.945334, exp_rate<=1 : 0.979158, exp_rate<=2 : 0.994512, exp_rate<=3 : 0.997942, loss: 0.001697, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19256 s, avg_samples: 56.0, ips: 290.81802 samples/s, eta: 4:28:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:23:22] ppocr INFO: epoch: [470/500], global_step: 463500, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.946929, exp_rate<=1 : 0.979924, exp_rate<=2 : 0.995021, exp_rate<=3 : 0.998046, loss: 0.001045, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17851 s, avg_samples: 56.0, ips: 313.69958 samples/s, eta: 4:27:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:23:40] ppocr INFO: epoch: [470/500], global_step: 463600, lr: 0.000100, edit distance: 0.001104, exp_rate: 0.948286, exp_rate<=1 : 0.980592, exp_rate<=2 : 0.995370, exp_rate<=3 : 0.998155, loss: 0.001191, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18137 s, avg_samples: 56.0, ips: 308.76334 samples/s, eta: 4:27:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:23:59] ppocr INFO: epoch: [470/500], global_step: 463700, lr: 0.000100, edit distance: 0.001103, exp_rate: 0.947544, exp_rate<=1 : 0.979939, exp_rate<=2 : 0.995345, exp_rate<=3 : 0.998166, loss: 0.001383, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19380 s, avg_samples: 56.0, ips: 288.95356 samples/s, eta: 4:27:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:24:17] ppocr INFO: epoch: [470/500], global_step: 463800, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.947818, exp_rate<=1 : 0.979803, exp_rate<=2 : 0.995429, exp_rate<=3 : 0.998380, loss: 0.001342, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17936 s, avg_samples: 56.0, ips: 312.22801 samples/s, eta: 4:26:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:24:36] ppocr INFO: epoch: [470/500], global_step: 463900, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.946202, exp_rate<=1 : 0.979304, exp_rate<=2 : 0.995512, exp_rate<=3 : 0.998477, loss: 0.001492, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18809 s, avg_samples: 56.0, ips: 297.72949 samples/s, eta: 4:26:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:24:54] ppocr INFO: epoch: [470/500], global_step: 464000, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.946314, exp_rate<=1 : 0.979480, exp_rate<=2 : 0.995720, exp_rate<=3 : 0.998592, loss: 0.001491, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17871 s, avg_samples: 56.0, ips: 313.34844 samples/s, eta: 4:26:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:25:13] ppocr INFO: epoch: [470/500], global_step: 464100, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.945237, exp_rate<=1 : 0.978960, exp_rate<=2 : 0.995627, exp_rate<=3 : 0.998578, loss: 0.001221, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19230 s, avg_samples: 56.0, ips: 291.21559 samples/s, eta: 4:25:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:25:31] ppocr INFO: epoch: [470/500], global_step: 464200, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.945032, exp_rate<=1 : 0.978933, exp_rate<=2 : 0.995646, exp_rate<=3 : 0.998531, loss: 0.001464, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17936 s, avg_samples: 56.0, ips: 312.21886 samples/s, eta: 4:25:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:25:50] ppocr INFO: epoch: [470/500], global_step: 464300, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.944353, exp_rate<=1 : 0.978551, exp_rate<=2 : 0.995632, exp_rate<=3 : 0.998578, loss: 0.001327, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18898 s, avg_samples: 56.0, ips: 296.33072 samples/s, eta: 4:25:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:26:10] ppocr INFO: epoch: [470/500], global_step: 464400, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.944040, exp_rate<=1 : 0.978633, exp_rate<=2 : 0.995617, exp_rate<=3 : 0.998538, loss: 0.001706, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19769 s, avg_samples: 56.0, ips: 283.27148 samples/s, eta: 4:24:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:26:29] ppocr INFO: epoch: [470/500], global_step: 464500, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.943814, exp_rate<=1 : 0.978637, exp_rate<=2 : 0.995631, exp_rate<=3 : 0.998555, loss: 0.001131, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18841 s, avg_samples: 56.0, ips: 297.22537 samples/s, eta: 4:24:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:26:48] ppocr INFO: epoch: [470/500], global_step: 464600, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.943450, exp_rate<=1 : 0.978517, exp_rate<=2 : 0.995596, exp_rate<=3 : 0.998567, loss: 0.001291, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18939 s, avg_samples: 56.0, ips: 295.67911 samples/s, eta: 4:24:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:27:07] ppocr INFO: epoch: [470/500], global_step: 464700, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.942984, exp_rate<=1 : 0.978195, exp_rate<=2 : 0.995514, exp_rate<=3 : 0.998511, loss: 0.001188, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19244 s, avg_samples: 56.0, ips: 291.00611 samples/s, eta: 4:24:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:27:25] ppocr INFO: epoch: [470/500], global_step: 464800, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.943130, exp_rate<=1 : 0.978286, exp_rate<=2 : 0.995496, exp_rate<=3 : 0.998495, loss: 0.001312, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18294 s, avg_samples: 56.0, ips: 306.10555 samples/s, eta: 4:23:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:27:44] ppocr INFO: epoch: [470/500], global_step: 464900, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.942914, exp_rate<=1 : 0.978182, exp_rate<=2 : 0.995465, exp_rate<=3 : 0.998478, loss: 0.001399, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18919 s, avg_samples: 56.0, ips: 295.99933 samples/s, eta: 4:23:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:28:04] ppocr INFO: epoch: [470/500], global_step: 465000, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.942795, exp_rate<=1 : 0.978124, exp_rate<=2 : 0.995499, exp_rate<=3 : 0.998455, loss: 0.001280, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19595 s, avg_samples: 56.0, ips: 285.78377 samples/s, eta: 4:23:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:28:22] ppocr INFO: epoch: [470/500], global_step: 465100, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.943117, exp_rate<=1 : 0.978243, exp_rate<=2 : 0.995594, exp_rate<=3 : 0.998470, loss: 0.001337, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18585 s, avg_samples: 56.0, ips: 301.31749 samples/s, eta: 4:22:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:28:43] ppocr INFO: epoch: [470/500], global_step: 465200, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.942380, exp_rate<=1 : 0.977928, exp_rate<=2 : 0.995462, exp_rate<=3 : 0.998423, loss: 0.001314, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20677 s, avg_samples: 56.0, ips: 270.82614 samples/s, eta: 4:22:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:29:01] ppocr INFO: epoch: [470/500], global_step: 465300, lr: 0.000100, edit distance: 0.001250, exp_rate: 0.942327, exp_rate<=1 : 0.977904, exp_rate<=2 : 0.995409, exp_rate<=3 : 0.998383, loss: 0.001290, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18151 s, avg_samples: 56.0, ips: 308.52252 samples/s, eta: 4:22:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:29:21] ppocr INFO: epoch: [470/500], global_step: 465400, lr: 0.000100, edit distance: 0.001262, exp_rate: 0.941942, exp_rate<=1 : 0.977741, exp_rate<=2 : 0.995330, exp_rate<=3 : 0.998324, loss: 0.001792, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19490 s, avg_samples: 56.0, ips: 287.33200 samples/s, eta: 4:21:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:29:40] ppocr INFO: epoch: [470/500], global_step: 465500, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.941850, exp_rate<=1 : 0.977735, exp_rate<=2 : 0.995324, exp_rate<=3 : 0.998287, loss: 0.001604, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18814 s, avg_samples: 56.0, ips: 297.65236 samples/s, eta: 4:21:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:29:58] ppocr INFO: epoch: [470/500], global_step: 465600, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.941774, exp_rate<=1 : 0.977810, exp_rate<=2 : 0.995350, exp_rate<=3 : 0.998282, loss: 0.001221, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18785 s, avg_samples: 56.0, ips: 298.10751 samples/s, eta: 4:21:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:30:19] ppocr INFO: epoch: [470/500], global_step: 465700, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.941544, exp_rate<=1 : 0.977646, exp_rate<=2 : 0.995307, exp_rate<=3 : 0.998288, loss: 0.001193, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20517 s, avg_samples: 56.0, ips: 272.94070 samples/s, eta: 4:20:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:30:37] ppocr INFO: epoch: [470/500], global_step: 465800, lr: 0.000100, edit distance: 0.001264, exp_rate: 0.941597, exp_rate<=1 : 0.977665, exp_rate<=2 : 0.995295, exp_rate<=3 : 0.998280, loss: 0.001679, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18415 s, avg_samples: 56.0, ips: 304.10395 samples/s, eta: 4:20:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:30:40] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 03:30:41] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_470 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 03:30:59] ppocr INFO: epoch: [471/500], global_step: 465900, lr: 0.000100, edit distance: 0.000975, exp_rate: 0.957879, exp_rate<=1 : 0.984979, exp_rate<=2 : 0.997130, exp_rate<=3 : 0.998921, loss: 0.001808, avg_reader_cost: 0.03481 s, avg_batch_cost: 0.21249 s, avg_samples: 56.0, ips: 263.54567 samples/s, eta: 4:20:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:31:17] ppocr INFO: epoch: [471/500], global_step: 466000, lr: 0.000100, edit distance: 0.001020, exp_rate: 0.955516, exp_rate<=1 : 0.983245, exp_rate<=2 : 0.996663, exp_rate<=3 : 0.998947, loss: 0.001223, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18036 s, avg_samples: 56.0, ips: 310.48437 samples/s, eta: 4:19:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:31:38] ppocr INFO: epoch: [471/500], global_step: 466100, lr: 0.000100, edit distance: 0.001098, exp_rate: 0.951377, exp_rate<=1 : 0.982237, exp_rate<=2 : 0.996560, exp_rate<=3 : 0.998682, loss: 0.001657, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20919 s, avg_samples: 56.0, ips: 267.70535 samples/s, eta: 4:19:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:31:57] ppocr INFO: epoch: [471/500], global_step: 466200, lr: 0.000100, edit distance: 0.001122, exp_rate: 0.948508, exp_rate<=1 : 0.980866, exp_rate<=2 : 0.995633, exp_rate<=3 : 0.998543, loss: 0.001385, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18877 s, avg_samples: 56.0, ips: 296.65713 samples/s, eta: 4:19:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:32:14] ppocr INFO: epoch: [471/500], global_step: 466300, lr: 0.000100, edit distance: 0.001116, exp_rate: 0.949029, exp_rate<=1 : 0.980708, exp_rate<=2 : 0.995618, exp_rate<=3 : 0.998373, loss: 0.001635, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17798 s, avg_samples: 56.0, ips: 314.64748 samples/s, eta: 4:18:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:32:33] ppocr INFO: epoch: [471/500], global_step: 466400, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.948063, exp_rate<=1 : 0.980165, exp_rate<=2 : 0.995592, exp_rate<=3 : 0.998370, loss: 0.001380, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19039 s, avg_samples: 56.0, ips: 294.13752 samples/s, eta: 4:18:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:32:52] ppocr INFO: epoch: [471/500], global_step: 466500, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.947903, exp_rate<=1 : 0.979929, exp_rate<=2 : 0.995448, exp_rate<=3 : 0.998414, loss: 0.001367, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18310 s, avg_samples: 56.0, ips: 305.84932 samples/s, eta: 4:18:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:33:11] ppocr INFO: epoch: [471/500], global_step: 466600, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.948267, exp_rate<=1 : 0.979978, exp_rate<=2 : 0.995386, exp_rate<=3 : 0.998554, loss: 0.001223, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19316 s, avg_samples: 56.0, ips: 289.90844 samples/s, eta: 4:18:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:33:31] ppocr INFO: epoch: [471/500], global_step: 466700, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.948396, exp_rate<=1 : 0.979955, exp_rate<=2 : 0.995377, exp_rate<=3 : 0.998569, loss: 0.001293, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19854 s, avg_samples: 56.0, ips: 282.05206 samples/s, eta: 4:17:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:33:49] ppocr INFO: epoch: [471/500], global_step: 466800, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.948364, exp_rate<=1 : 0.980104, exp_rate<=2 : 0.995470, exp_rate<=3 : 0.998612, loss: 0.001201, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18032 s, avg_samples: 56.0, ips: 310.55572 samples/s, eta: 4:17:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:34:07] ppocr INFO: epoch: [471/500], global_step: 466900, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.948317, exp_rate<=1 : 0.980203, exp_rate<=2 : 0.995632, exp_rate<=3 : 0.998660, loss: 0.001123, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18346 s, avg_samples: 56.0, ips: 305.24978 samples/s, eta: 4:17:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:34:26] ppocr INFO: epoch: [471/500], global_step: 467000, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.948338, exp_rate<=1 : 0.980269, exp_rate<=2 : 0.995622, exp_rate<=3 : 0.998686, loss: 0.001548, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18050 s, avg_samples: 56.0, ips: 310.24787 samples/s, eta: 4:16:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:34:44] ppocr INFO: epoch: [471/500], global_step: 467100, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.948305, exp_rate<=1 : 0.980190, exp_rate<=2 : 0.995669, exp_rate<=3 : 0.998699, loss: 0.001344, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18423 s, avg_samples: 56.0, ips: 303.96130 samples/s, eta: 4:16:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:35:02] ppocr INFO: epoch: [471/500], global_step: 467200, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.948537, exp_rate<=1 : 0.980384, exp_rate<=2 : 0.995685, exp_rate<=3 : 0.998703, loss: 0.001129, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17803 s, avg_samples: 56.0, ips: 314.54909 samples/s, eta: 4:16:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:35:21] ppocr INFO: epoch: [471/500], global_step: 467300, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.948147, exp_rate<=1 : 0.980338, exp_rate<=2 : 0.995762, exp_rate<=3 : 0.998750, loss: 0.001239, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18822 s, avg_samples: 56.0, ips: 297.53155 samples/s, eta: 4:15:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:35:41] ppocr INFO: epoch: [471/500], global_step: 467400, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.947051, exp_rate<=1 : 0.979917, exp_rate<=2 : 0.995655, exp_rate<=3 : 0.998684, loss: 0.001460, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20136 s, avg_samples: 56.0, ips: 278.10221 samples/s, eta: 4:15:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:35:59] ppocr INFO: epoch: [471/500], global_step: 467500, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.946249, exp_rate<=1 : 0.979486, exp_rate<=2 : 0.995502, exp_rate<=3 : 0.998654, loss: 0.001426, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18002 s, avg_samples: 56.0, ips: 311.07309 samples/s, eta: 4:15:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:36:17] ppocr INFO: epoch: [471/500], global_step: 467600, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.945763, exp_rate<=1 : 0.979296, exp_rate<=2 : 0.995466, exp_rate<=3 : 0.998660, loss: 0.001106, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18566 s, avg_samples: 56.0, ips: 301.63375 samples/s, eta: 4:14:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:36:35] ppocr INFO: epoch: [471/500], global_step: 467700, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.945849, exp_rate<=1 : 0.979315, exp_rate<=2 : 0.995486, exp_rate<=3 : 0.998636, loss: 0.001965, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17965 s, avg_samples: 56.0, ips: 311.71475 samples/s, eta: 4:14:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:36:53] ppocr INFO: epoch: [471/500], global_step: 467800, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.945448, exp_rate<=1 : 0.979347, exp_rate<=2 : 0.995495, exp_rate<=3 : 0.998612, loss: 0.001578, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17250 s, avg_samples: 56.0, ips: 324.63537 samples/s, eta: 4:14:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:37:12] ppocr INFO: epoch: [471/500], global_step: 467900, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.944703, exp_rate<=1 : 0.978961, exp_rate<=2 : 0.995478, exp_rate<=3 : 0.998571, loss: 0.001765, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19634 s, avg_samples: 56.0, ips: 285.22678 samples/s, eta: 4:13:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:37:30] ppocr INFO: epoch: [471/500], global_step: 468000, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.944532, exp_rate<=1 : 0.978846, exp_rate<=2 : 0.995499, exp_rate<=3 : 0.998568, loss: 0.001326, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17951 s, avg_samples: 56.0, ips: 311.96091 samples/s, eta: 4:13:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:37:50] ppocr INFO: epoch: [471/500], global_step: 468100, lr: 0.000100, edit distance: 0.001254, exp_rate: 0.944335, exp_rate<=1 : 0.978654, exp_rate<=2 : 0.995484, exp_rate<=3 : 0.998566, loss: 0.001147, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19770 s, avg_samples: 56.0, ips: 283.26266 samples/s, eta: 4:13:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:38:09] ppocr INFO: epoch: [471/500], global_step: 468200, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.943917, exp_rate<=1 : 0.978475, exp_rate<=2 : 0.995403, exp_rate<=3 : 0.998537, loss: 0.001449, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19165 s, avg_samples: 56.0, ips: 292.20644 samples/s, eta: 4:12:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:38:31] ppocr INFO: epoch: [471/500], global_step: 468300, lr: 0.000100, edit distance: 0.001265, exp_rate: 0.943561, exp_rate<=1 : 0.978322, exp_rate<=2 : 0.995347, exp_rate<=3 : 0.998529, loss: 0.001322, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21223 s, avg_samples: 56.0, ips: 263.86057 samples/s, eta: 4:12:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:38:50] ppocr INFO: epoch: [471/500], global_step: 468400, lr: 0.000100, edit distance: 0.001267, exp_rate: 0.943564, exp_rate<=1 : 0.978358, exp_rate<=2 : 0.995380, exp_rate<=3 : 0.998551, loss: 0.001222, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19043 s, avg_samples: 56.0, ips: 294.06509 samples/s, eta: 4:12:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:39:10] ppocr INFO: epoch: [471/500], global_step: 468500, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.943608, exp_rate<=1 : 0.978394, exp_rate<=2 : 0.995344, exp_rate<=3 : 0.998542, loss: 0.001133, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20107 s, avg_samples: 56.0, ips: 278.50832 samples/s, eta: 4:11:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:39:18] ppocr INFO: epoch: [471/500], global_step: 468540, lr: 0.000100, edit distance: 0.001259, exp_rate: 0.943212, exp_rate<=1 : 0.978186, exp_rate<=2 : 0.995264, exp_rate<=3 : 0.998489, loss: 0.001312, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.08513 s, avg_samples: 22.4, ips: 263.12059 samples/s, eta: 4:11:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:39:20] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 03:39:33] ppocr INFO: epoch: [472/500], global_step: 468600, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.954594, exp_rate<=1 : 0.981082, exp_rate<=2 : 0.995617, exp_rate<=3 : 0.999232, loss: 0.001257, avg_reader_cost: 0.02501 s, avg_batch_cost: 0.14454 s, avg_samples: 33.6, ips: 232.45864 samples/s, eta: 4:11:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:39:52] ppocr INFO: epoch: [472/500], global_step: 468700, lr: 0.000100, edit distance: 0.001009, exp_rate: 0.954722, exp_rate<=1 : 0.982263, exp_rate<=2 : 0.995916, exp_rate<=3 : 0.998884, loss: 0.001230, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18986 s, avg_samples: 56.0, ips: 294.96112 samples/s, eta: 4:11:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:40:10] ppocr INFO: epoch: [472/500], global_step: 468800, lr: 0.000100, edit distance: 0.000981, exp_rate: 0.955757, exp_rate<=1 : 0.982665, exp_rate<=2 : 0.995855, exp_rate<=3 : 0.998742, loss: 0.001225, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17829 s, avg_samples: 56.0, ips: 314.09411 samples/s, eta: 4:11:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:40:29] ppocr INFO: epoch: [472/500], global_step: 468900, lr: 0.000100, edit distance: 0.000971, exp_rate: 0.957014, exp_rate<=1 : 0.983273, exp_rate<=2 : 0.996077, exp_rate<=3 : 0.998763, loss: 0.001136, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19473 s, avg_samples: 56.0, ips: 287.58263 samples/s, eta: 4:10:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:40:50] ppocr INFO: epoch: [472/500], global_step: 469000, lr: 0.000100, edit distance: 0.001011, exp_rate: 0.953720, exp_rate<=1 : 0.982064, exp_rate<=2 : 0.995932, exp_rate<=3 : 0.998771, loss: 0.001311, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20782 s, avg_samples: 56.0, ips: 269.45817 samples/s, eta: 4:10:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:41:09] ppocr INFO: epoch: [472/500], global_step: 469100, lr: 0.000100, edit distance: 0.001031, exp_rate: 0.953321, exp_rate<=1 : 0.982207, exp_rate<=2 : 0.995694, exp_rate<=3 : 0.998657, loss: 0.001352, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19071 s, avg_samples: 56.0, ips: 293.63630 samples/s, eta: 4:10:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:41:27] ppocr INFO: epoch: [472/500], global_step: 469200, lr: 0.000100, edit distance: 0.001017, exp_rate: 0.953399, exp_rate<=1 : 0.982416, exp_rate<=2 : 0.995706, exp_rate<=3 : 0.998688, loss: 0.001442, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18022 s, avg_samples: 56.0, ips: 310.73422 samples/s, eta: 4:09:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:41:46] ppocr INFO: epoch: [472/500], global_step: 469300, lr: 0.000100, edit distance: 0.001036, exp_rate: 0.953030, exp_rate<=1 : 0.982511, exp_rate<=2 : 0.995655, exp_rate<=3 : 0.998730, loss: 0.002228, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18565 s, avg_samples: 56.0, ips: 301.64768 samples/s, eta: 4:09:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:42:03] ppocr INFO: epoch: [472/500], global_step: 469400, lr: 0.000100, edit distance: 0.001078, exp_rate: 0.952212, exp_rate<=1 : 0.982037, exp_rate<=2 : 0.995654, exp_rate<=3 : 0.998745, loss: 0.001430, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17713 s, avg_samples: 56.0, ips: 316.16055 samples/s, eta: 4:09:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:42:24] ppocr INFO: epoch: [472/500], global_step: 469500, lr: 0.000100, edit distance: 0.001077, exp_rate: 0.950890, exp_rate<=1 : 0.981689, exp_rate<=2 : 0.995515, exp_rate<=3 : 0.998614, loss: 0.001174, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20356 s, avg_samples: 56.0, ips: 275.10209 samples/s, eta: 4:08:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:42:43] ppocr INFO: epoch: [472/500], global_step: 469600, lr: 0.000100, edit distance: 0.001068, exp_rate: 0.951113, exp_rate<=1 : 0.981446, exp_rate<=2 : 0.995485, exp_rate<=3 : 0.998615, loss: 0.000938, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19052 s, avg_samples: 56.0, ips: 293.93336 samples/s, eta: 4:08:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:43:02] ppocr INFO: epoch: [472/500], global_step: 469700, lr: 0.000100, edit distance: 0.001100, exp_rate: 0.950275, exp_rate<=1 : 0.981300, exp_rate<=2 : 0.995485, exp_rate<=3 : 0.998557, loss: 0.001705, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18796 s, avg_samples: 56.0, ips: 297.93360 samples/s, eta: 4:08:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:43:20] ppocr INFO: epoch: [472/500], global_step: 469800, lr: 0.000100, edit distance: 0.001111, exp_rate: 0.949581, exp_rate<=1 : 0.980791, exp_rate<=2 : 0.995348, exp_rate<=3 : 0.998547, loss: 0.001491, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18680 s, avg_samples: 56.0, ips: 299.78296 samples/s, eta: 4:07:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:43:39] ppocr INFO: epoch: [472/500], global_step: 469900, lr: 0.000100, edit distance: 0.001131, exp_rate: 0.948842, exp_rate<=1 : 0.980430, exp_rate<=2 : 0.995417, exp_rate<=3 : 0.998504, loss: 0.002132, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18509 s, avg_samples: 56.0, ips: 302.56273 samples/s, eta: 4:07:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:43:56] ppocr INFO: epoch: [472/500], global_step: 470000, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.948357, exp_rate<=1 : 0.980225, exp_rate<=2 : 0.995336, exp_rate<=3 : 0.998505, loss: 0.001389, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17221 s, avg_samples: 56.0, ips: 325.17493 samples/s, eta: 4:07:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:44:14] ppocr INFO: epoch: [472/500], global_step: 470100, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.948003, exp_rate<=1 : 0.980143, exp_rate<=2 : 0.995380, exp_rate<=3 : 0.998498, loss: 0.001519, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17877 s, avg_samples: 56.0, ips: 313.24464 samples/s, eta: 4:06:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:44:32] ppocr INFO: epoch: [472/500], global_step: 470200, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.947452, exp_rate<=1 : 0.979928, exp_rate<=2 : 0.995304, exp_rate<=3 : 0.998494, loss: 0.001500, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18195 s, avg_samples: 56.0, ips: 307.76920 samples/s, eta: 4:06:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:44:51] ppocr INFO: epoch: [472/500], global_step: 470300, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.947136, exp_rate<=1 : 0.979796, exp_rate<=2 : 0.995212, exp_rate<=3 : 0.998445, loss: 0.001413, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18714 s, avg_samples: 56.0, ips: 299.23460 samples/s, eta: 4:06:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:45:11] ppocr INFO: epoch: [472/500], global_step: 470400, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.946568, exp_rate<=1 : 0.979515, exp_rate<=2 : 0.995115, exp_rate<=3 : 0.998397, loss: 0.000735, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19609 s, avg_samples: 56.0, ips: 285.58094 samples/s, eta: 4:05:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:45:30] ppocr INFO: epoch: [472/500], global_step: 470500, lr: 0.000100, edit distance: 0.001185, exp_rate: 0.946661, exp_rate<=1 : 0.979520, exp_rate<=2 : 0.995087, exp_rate<=3 : 0.998398, loss: 0.001814, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19157 s, avg_samples: 56.0, ips: 292.32827 samples/s, eta: 4:05:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:45:49] ppocr INFO: epoch: [472/500], global_step: 470600, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.946044, exp_rate<=1 : 0.979299, exp_rate<=2 : 0.995097, exp_rate<=3 : 0.998396, loss: 0.002034, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19518 s, avg_samples: 56.0, ips: 286.91085 samples/s, eta: 4:05:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:46:08] ppocr INFO: epoch: [472/500], global_step: 470700, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.945648, exp_rate<=1 : 0.979254, exp_rate<=2 : 0.995098, exp_rate<=3 : 0.998397, loss: 0.002128, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19089 s, avg_samples: 56.0, ips: 293.35794 samples/s, eta: 4:05:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:46:26] ppocr INFO: epoch: [472/500], global_step: 470800, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.945448, exp_rate<=1 : 0.979151, exp_rate<=2 : 0.995015, exp_rate<=3 : 0.998321, loss: 0.001247, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18019 s, avg_samples: 56.0, ips: 310.78433 samples/s, eta: 4:04:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:46:45] ppocr INFO: epoch: [472/500], global_step: 470900, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.945260, exp_rate<=1 : 0.979013, exp_rate<=2 : 0.995059, exp_rate<=3 : 0.998359, loss: 0.002076, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18572 s, avg_samples: 56.0, ips: 301.53565 samples/s, eta: 4:04:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:47:04] ppocr INFO: epoch: [472/500], global_step: 471000, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.944945, exp_rate<=1 : 0.978883, exp_rate<=2 : 0.995095, exp_rate<=3 : 0.998381, loss: 0.001934, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18914 s, avg_samples: 56.0, ips: 296.08259 samples/s, eta: 4:04:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:47:23] ppocr INFO: epoch: [472/500], global_step: 471100, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.944809, exp_rate<=1 : 0.978887, exp_rate<=2 : 0.995087, exp_rate<=3 : 0.998375, loss: 0.001894, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18889 s, avg_samples: 56.0, ips: 296.46758 samples/s, eta: 4:03:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:47:42] ppocr INFO: epoch: [472/500], global_step: 471200, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.944399, exp_rate<=1 : 0.978686, exp_rate<=2 : 0.995051, exp_rate<=3 : 0.998334, loss: 0.001150, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19084 s, avg_samples: 56.0, ips: 293.44596 samples/s, eta: 4:03:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:47:58] ppocr INFO: epoch: [472/500], global_step: 471280, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.944152, exp_rate<=1 : 0.978569, exp_rate<=2 : 0.995007, exp_rate<=3 : 0.998307, loss: 0.001408, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.15638 s, avg_samples: 44.8, ips: 286.48452 samples/s, eta: 4:03:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:48:00] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 03:48:04] ppocr INFO: epoch: [473/500], global_step: 471300, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.934957, exp_rate<=1 : 0.974851, exp_rate<=2 : 0.992857, exp_rate<=3 : 0.996207, loss: 0.001206, avg_reader_cost: 0.02617 s, avg_batch_cost: 0.06682 s, avg_samples: 11.2, ips: 167.61945 samples/s, eta: 4:03:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:48:24] ppocr INFO: epoch: [473/500], global_step: 471400, lr: 0.000100, edit distance: 0.001019, exp_rate: 0.946592, exp_rate<=1 : 0.978028, exp_rate<=2 : 0.995370, exp_rate<=3 : 0.998072, loss: 0.001321, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20209 s, avg_samples: 56.0, ips: 277.09801 samples/s, eta: 4:02:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:48:43] ppocr INFO: epoch: [473/500], global_step: 471500, lr: 0.000100, edit distance: 0.001053, exp_rate: 0.948547, exp_rate<=1 : 0.979636, exp_rate<=2 : 0.995947, exp_rate<=3 : 0.998310, loss: 0.001446, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18098 s, avg_samples: 56.0, ips: 309.43453 samples/s, eta: 4:02:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:49:00] ppocr INFO: epoch: [473/500], global_step: 471600, lr: 0.000100, edit distance: 0.001058, exp_rate: 0.951260, exp_rate<=1 : 0.980935, exp_rate<=2 : 0.996048, exp_rate<=3 : 0.998447, loss: 0.000768, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17242 s, avg_samples: 56.0, ips: 324.79580 samples/s, eta: 4:02:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:49:19] ppocr INFO: epoch: [473/500], global_step: 471700, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.949387, exp_rate<=1 : 0.981449, exp_rate<=2 : 0.996200, exp_rate<=3 : 0.998695, loss: 0.000934, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18700 s, avg_samples: 56.0, ips: 299.46688 samples/s, eta: 4:01:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:49:37] ppocr INFO: epoch: [473/500], global_step: 471800, lr: 0.000100, edit distance: 0.001167, exp_rate: 0.949645, exp_rate<=1 : 0.981357, exp_rate<=2 : 0.996330, exp_rate<=3 : 0.998727, loss: 0.001200, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18668 s, avg_samples: 56.0, ips: 299.98201 samples/s, eta: 4:01:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:49:56] ppocr INFO: epoch: [473/500], global_step: 471900, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.948154, exp_rate<=1 : 0.980953, exp_rate<=2 : 0.996338, exp_rate<=3 : 0.998713, loss: 0.001733, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18918 s, avg_samples: 56.0, ips: 296.01226 samples/s, eta: 4:01:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:50:14] ppocr INFO: epoch: [473/500], global_step: 472000, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.946972, exp_rate<=1 : 0.980525, exp_rate<=2 : 0.995991, exp_rate<=3 : 0.998583, loss: 0.001662, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17366 s, avg_samples: 56.0, ips: 322.46346 samples/s, eta: 4:00:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:50:32] ppocr INFO: epoch: [473/500], global_step: 472100, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.946902, exp_rate<=1 : 0.980392, exp_rate<=2 : 0.995882, exp_rate<=3 : 0.998436, loss: 0.001039, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18390 s, avg_samples: 56.0, ips: 304.51495 samples/s, eta: 4:00:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:50:51] ppocr INFO: epoch: [473/500], global_step: 472200, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.947278, exp_rate<=1 : 0.980397, exp_rate<=2 : 0.995826, exp_rate<=3 : 0.998518, loss: 0.001573, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18759 s, avg_samples: 56.0, ips: 298.52971 samples/s, eta: 4:00:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:51:10] ppocr INFO: epoch: [473/500], global_step: 472300, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.946217, exp_rate<=1 : 0.980107, exp_rate<=2 : 0.995811, exp_rate<=3 : 0.998436, loss: 0.001407, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19087 s, avg_samples: 56.0, ips: 293.39039 samples/s, eta: 3:59:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:51:28] ppocr INFO: epoch: [473/500], global_step: 472400, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.946412, exp_rate<=1 : 0.979964, exp_rate<=2 : 0.995793, exp_rate<=3 : 0.998449, loss: 0.001277, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17811 s, avg_samples: 56.0, ips: 314.40477 samples/s, eta: 3:59:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:51:48] ppocr INFO: epoch: [473/500], global_step: 472500, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.946013, exp_rate<=1 : 0.979778, exp_rate<=2 : 0.995723, exp_rate<=3 : 0.998460, loss: 0.001239, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20287 s, avg_samples: 56.0, ips: 276.03601 samples/s, eta: 3:59:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:52:06] ppocr INFO: epoch: [473/500], global_step: 472600, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.946333, exp_rate<=1 : 0.979876, exp_rate<=2 : 0.995775, exp_rate<=3 : 0.998437, loss: 0.001292, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17505 s, avg_samples: 56.0, ips: 319.91332 samples/s, eta: 3:58:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:52:25] ppocr INFO: epoch: [473/500], global_step: 472700, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.944733, exp_rate<=1 : 0.979224, exp_rate<=2 : 0.995536, exp_rate<=3 : 0.998277, loss: 0.001426, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19640 s, avg_samples: 56.0, ips: 285.13506 samples/s, eta: 3:58:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:52:43] ppocr INFO: epoch: [473/500], global_step: 472800, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.944509, exp_rate<=1 : 0.979196, exp_rate<=2 : 0.995605, exp_rate<=3 : 0.998345, loss: 0.001296, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17563 s, avg_samples: 56.0, ips: 318.85325 samples/s, eta: 3:58:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:53:02] ppocr INFO: epoch: [473/500], global_step: 472900, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.944151, exp_rate<=1 : 0.979061, exp_rate<=2 : 0.995530, exp_rate<=3 : 0.998358, loss: 0.001385, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19170 s, avg_samples: 56.0, ips: 292.12113 samples/s, eta: 3:58:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:53:21] ppocr INFO: epoch: [473/500], global_step: 473000, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.943892, exp_rate<=1 : 0.978947, exp_rate<=2 : 0.995417, exp_rate<=3 : 0.998340, loss: 0.001074, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19158 s, avg_samples: 56.0, ips: 292.29955 samples/s, eta: 3:57:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:53:39] ppocr INFO: epoch: [473/500], global_step: 473100, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.944334, exp_rate<=1 : 0.979068, exp_rate<=2 : 0.995498, exp_rate<=3 : 0.998392, loss: 0.001531, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18146 s, avg_samples: 56.0, ips: 308.60360 samples/s, eta: 3:57:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:54:00] ppocr INFO: epoch: [473/500], global_step: 473200, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.944053, exp_rate<=1 : 0.978997, exp_rate<=2 : 0.995464, exp_rate<=3 : 0.998392, loss: 0.001563, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20327 s, avg_samples: 56.0, ips: 275.49590 samples/s, eta: 3:57:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:54:18] ppocr INFO: epoch: [473/500], global_step: 473300, lr: 0.000100, edit distance: 0.001251, exp_rate: 0.943726, exp_rate<=1 : 0.978869, exp_rate<=2 : 0.995442, exp_rate<=3 : 0.998351, loss: 0.001351, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17884 s, avg_samples: 56.0, ips: 313.12965 samples/s, eta: 3:56:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:54:36] ppocr INFO: epoch: [473/500], global_step: 473400, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.943810, exp_rate<=1 : 0.978992, exp_rate<=2 : 0.995462, exp_rate<=3 : 0.998365, loss: 0.001605, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17895 s, avg_samples: 56.0, ips: 312.93375 samples/s, eta: 3:56:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:54:57] ppocr INFO: epoch: [473/500], global_step: 473500, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.943461, exp_rate<=1 : 0.978768, exp_rate<=2 : 0.995359, exp_rate<=3 : 0.998317, loss: 0.000900, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20996 s, avg_samples: 56.0, ips: 266.72266 samples/s, eta: 3:56:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:55:16] ppocr INFO: epoch: [473/500], global_step: 473600, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.943397, exp_rate<=1 : 0.978674, exp_rate<=2 : 0.995248, exp_rate<=3 : 0.998283, loss: 0.001320, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19736 s, avg_samples: 56.0, ips: 283.74673 samples/s, eta: 3:55:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:55:36] ppocr INFO: epoch: [473/500], global_step: 473700, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.943334, exp_rate<=1 : 0.978723, exp_rate<=2 : 0.995293, exp_rate<=3 : 0.998295, loss: 0.001429, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19401 s, avg_samples: 56.0, ips: 288.64598 samples/s, eta: 3:55:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:55:54] ppocr INFO: epoch: [473/500], global_step: 473800, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.943630, exp_rate<=1 : 0.978840, exp_rate<=2 : 0.995323, exp_rate<=3 : 0.998300, loss: 0.001512, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18302 s, avg_samples: 56.0, ips: 305.97384 samples/s, eta: 3:55:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:56:14] ppocr INFO: epoch: [473/500], global_step: 473900, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.943734, exp_rate<=1 : 0.978847, exp_rate<=2 : 0.995312, exp_rate<=3 : 0.998300, loss: 0.001626, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19912 s, avg_samples: 56.0, ips: 281.24169 samples/s, eta: 3:54:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:56:33] ppocr INFO: epoch: [473/500], global_step: 474000, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.943564, exp_rate<=1 : 0.978719, exp_rate<=2 : 0.995270, exp_rate<=3 : 0.998283, loss: 0.001031, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18625 s, avg_samples: 56.0, ips: 300.67282 samples/s, eta: 3:54:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:56:36] ppocr INFO: epoch: [473/500], global_step: 474020, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.943541, exp_rate<=1 : 0.978686, exp_rate<=2 : 0.995237, exp_rate<=3 : 0.998282, loss: 0.001313, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03600 s, avg_samples: 11.2, ips: 311.07294 samples/s, eta: 3:54:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:56:38] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 03:56:54] ppocr INFO: epoch: [474/500], global_step: 474100, lr: 0.000100, edit distance: 0.001108, exp_rate: 0.948794, exp_rate<=1 : 0.982629, exp_rate<=2 : 0.994950, exp_rate<=3 : 0.998241, loss: 0.001307, avg_reader_cost: 0.02711 s, avg_batch_cost: 0.17787 s, avg_samples: 44.8, ips: 251.87210 samples/s, eta: 3:54:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:57:11] ppocr INFO: epoch: [474/500], global_step: 474200, lr: 0.000100, edit distance: 0.001017, exp_rate: 0.952927, exp_rate<=1 : 0.983536, exp_rate<=2 : 0.996114, exp_rate<=3 : 0.999017, loss: 0.001418, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17060 s, avg_samples: 56.0, ips: 328.25088 samples/s, eta: 3:53:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:57:29] ppocr INFO: epoch: [474/500], global_step: 474300, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.950683, exp_rate<=1 : 0.982815, exp_rate<=2 : 0.996340, exp_rate<=3 : 0.999119, loss: 0.001185, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18065 s, avg_samples: 56.0, ips: 309.99906 samples/s, eta: 3:53:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:57:48] ppocr INFO: epoch: [474/500], global_step: 474400, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.949754, exp_rate<=1 : 0.981926, exp_rate<=2 : 0.995900, exp_rate<=3 : 0.998913, loss: 0.001423, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18915 s, avg_samples: 56.0, ips: 296.05675 samples/s, eta: 3:53:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:58:08] ppocr INFO: epoch: [474/500], global_step: 474500, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.948365, exp_rate<=1 : 0.981221, exp_rate<=2 : 0.995784, exp_rate<=3 : 0.998937, loss: 0.001266, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19367 s, avg_samples: 56.0, ips: 289.15200 samples/s, eta: 3:52:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:58:26] ppocr INFO: epoch: [474/500], global_step: 474600, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.947005, exp_rate<=1 : 0.980426, exp_rate<=2 : 0.995598, exp_rate<=3 : 0.998825, loss: 0.001507, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18198 s, avg_samples: 56.0, ips: 307.72819 samples/s, eta: 3:52:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:58:45] ppocr INFO: epoch: [474/500], global_step: 474700, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.945785, exp_rate<=1 : 0.979877, exp_rate<=2 : 0.995552, exp_rate<=3 : 0.998821, loss: 0.001461, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18936 s, avg_samples: 56.0, ips: 295.74052 samples/s, eta: 3:52:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:59:05] ppocr INFO: epoch: [474/500], global_step: 474800, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.943281, exp_rate<=1 : 0.979176, exp_rate<=2 : 0.995352, exp_rate<=3 : 0.998695, loss: 0.001619, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19906 s, avg_samples: 56.0, ips: 281.32779 samples/s, eta: 3:52:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:59:24] ppocr INFO: epoch: [474/500], global_step: 474900, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.942048, exp_rate<=1 : 0.978701, exp_rate<=2 : 0.995132, exp_rate<=3 : 0.998556, loss: 0.001436, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18988 s, avg_samples: 56.0, ips: 294.92551 samples/s, eta: 3:51:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 03:59:44] ppocr INFO: epoch: [474/500], global_step: 475000, lr: 0.000100, edit distance: 0.001263, exp_rate: 0.941588, exp_rate<=1 : 0.978308, exp_rate<=2 : 0.994836, exp_rate<=3 : 0.998438, loss: 0.001392, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20065 s, avg_samples: 56.0, ips: 279.08840 samples/s, eta: 3:51:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:00:03] ppocr INFO: epoch: [474/500], global_step: 475100, lr: 0.000100, edit distance: 0.001249, exp_rate: 0.941730, exp_rate<=1 : 0.978037, exp_rate<=2 : 0.994846, exp_rate<=3 : 0.998456, loss: 0.001484, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19181 s, avg_samples: 56.0, ips: 291.96294 samples/s, eta: 3:51:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:00:21] ppocr INFO: epoch: [474/500], global_step: 475200, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.942137, exp_rate<=1 : 0.978110, exp_rate<=2 : 0.994797, exp_rate<=3 : 0.998398, loss: 0.001605, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18451 s, avg_samples: 56.0, ips: 303.50995 samples/s, eta: 3:50:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:00:39] ppocr INFO: epoch: [474/500], global_step: 475300, lr: 0.000100, edit distance: 0.001252, exp_rate: 0.942031, exp_rate<=1 : 0.978005, exp_rate<=2 : 0.994720, exp_rate<=3 : 0.998406, loss: 0.001815, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17951 s, avg_samples: 56.0, ips: 311.96054 samples/s, eta: 3:50:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:00:58] ppocr INFO: epoch: [474/500], global_step: 475400, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.941389, exp_rate<=1 : 0.977617, exp_rate<=2 : 0.994762, exp_rate<=3 : 0.998387, loss: 0.001486, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18890 s, avg_samples: 56.0, ips: 296.45169 samples/s, eta: 3:50:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:01:18] ppocr INFO: epoch: [474/500], global_step: 475500, lr: 0.000100, edit distance: 0.001280, exp_rate: 0.940982, exp_rate<=1 : 0.977462, exp_rate<=2 : 0.994824, exp_rate<=3 : 0.998387, loss: 0.001917, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19590 s, avg_samples: 56.0, ips: 285.85446 samples/s, eta: 3:49:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:01:37] ppocr INFO: epoch: [474/500], global_step: 475600, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.941033, exp_rate<=1 : 0.977555, exp_rate<=2 : 0.994843, exp_rate<=3 : 0.998338, loss: 0.001299, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19097 s, avg_samples: 56.0, ips: 293.24655 samples/s, eta: 3:49:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:01:55] ppocr INFO: epoch: [474/500], global_step: 475700, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.941233, exp_rate<=1 : 0.977643, exp_rate<=2 : 0.994922, exp_rate<=3 : 0.998345, loss: 0.001513, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18280 s, avg_samples: 56.0, ips: 306.34971 samples/s, eta: 3:49:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:02:14] ppocr INFO: epoch: [474/500], global_step: 475800, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.940919, exp_rate<=1 : 0.977562, exp_rate<=2 : 0.994937, exp_rate<=3 : 0.998396, loss: 0.001479, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19068 s, avg_samples: 56.0, ips: 293.69023 samples/s, eta: 3:48:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:02:34] ppocr INFO: epoch: [474/500], global_step: 475900, lr: 0.000100, edit distance: 0.001278, exp_rate: 0.940840, exp_rate<=1 : 0.977408, exp_rate<=2 : 0.994975, exp_rate<=3 : 0.998428, loss: 0.001130, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19119 s, avg_samples: 56.0, ips: 292.90349 samples/s, eta: 3:48:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:02:53] ppocr INFO: epoch: [474/500], global_step: 476000, lr: 0.000100, edit distance: 0.001290, exp_rate: 0.940190, exp_rate<=1 : 0.976958, exp_rate<=2 : 0.994871, exp_rate<=3 : 0.998337, loss: 0.001476, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19877 s, avg_samples: 56.0, ips: 281.73594 samples/s, eta: 3:48:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:03:11] ppocr INFO: epoch: [474/500], global_step: 476100, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.940038, exp_rate<=1 : 0.976820, exp_rate<=2 : 0.994902, exp_rate<=3 : 0.998343, loss: 0.001186, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17535 s, avg_samples: 56.0, ips: 319.35954 samples/s, eta: 3:47:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:03:31] ppocr INFO: epoch: [474/500], global_step: 476200, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.940158, exp_rate<=1 : 0.976834, exp_rate<=2 : 0.994917, exp_rate<=3 : 0.998342, loss: 0.002026, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19630 s, avg_samples: 56.0, ips: 285.27866 samples/s, eta: 3:47:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:03:49] ppocr INFO: epoch: [474/500], global_step: 476300, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.940454, exp_rate<=1 : 0.977008, exp_rate<=2 : 0.995010, exp_rate<=3 : 0.998376, loss: 0.001080, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18226 s, avg_samples: 56.0, ips: 307.24992 samples/s, eta: 3:47:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:04:07] ppocr INFO: epoch: [474/500], global_step: 476400, lr: 0.000100, edit distance: 0.001275, exp_rate: 0.940638, exp_rate<=1 : 0.977053, exp_rate<=2 : 0.994980, exp_rate<=3 : 0.998355, loss: 0.001029, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18402 s, avg_samples: 56.0, ips: 304.31516 samples/s, eta: 3:46:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:04:27] ppocr INFO: epoch: [474/500], global_step: 476500, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.940659, exp_rate<=1 : 0.977082, exp_rate<=2 : 0.995024, exp_rate<=3 : 0.998356, loss: 0.001763, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19431 s, avg_samples: 56.0, ips: 288.19414 samples/s, eta: 3:46:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:04:45] ppocr INFO: epoch: [474/500], global_step: 476600, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.940543, exp_rate<=1 : 0.977050, exp_rate<=2 : 0.995042, exp_rate<=3 : 0.998329, loss: 0.001071, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18508 s, avg_samples: 56.0, ips: 302.57602 samples/s, eta: 3:46:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:05:03] ppocr INFO: epoch: [474/500], global_step: 476700, lr: 0.000100, edit distance: 0.001274, exp_rate: 0.940920, exp_rate<=1 : 0.977190, exp_rate<=2 : 0.995073, exp_rate<=3 : 0.998345, loss: 0.001404, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17635 s, avg_samples: 56.0, ips: 317.55528 samples/s, eta: 3:45:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:05:14] ppocr INFO: epoch: [474/500], global_step: 476760, lr: 0.000100, edit distance: 0.001272, exp_rate: 0.940928, exp_rate<=1 : 0.977201, exp_rate<=2 : 0.995093, exp_rate<=3 : 0.998348, loss: 0.001602, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11024 s, avg_samples: 33.6, ips: 304.79880 samples/s, eta: 3:45:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:05:16] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 04:05:24] ppocr INFO: epoch: [475/500], global_step: 476800, lr: 0.000100, edit distance: 0.001586, exp_rate: 0.957663, exp_rate<=1 : 0.985485, exp_rate<=2 : 0.996610, exp_rate<=3 : 0.999326, loss: 0.001189, avg_reader_cost: 0.02514 s, avg_batch_cost: 0.09790 s, avg_samples: 22.4, ips: 228.80627 samples/s, eta: 3:45:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:05:42] ppocr INFO: epoch: [475/500], global_step: 476900, lr: 0.000100, edit distance: 0.001086, exp_rate: 0.957143, exp_rate<=1 : 0.985486, exp_rate<=2 : 0.996622, exp_rate<=3 : 0.998996, loss: 0.001077, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18639 s, avg_samples: 56.0, ips: 300.44656 samples/s, eta: 3:45:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:06:01] ppocr INFO: epoch: [475/500], global_step: 477000, lr: 0.000100, edit distance: 0.001124, exp_rate: 0.955222, exp_rate<=1 : 0.984067, exp_rate<=2 : 0.996611, exp_rate<=3 : 0.998988, loss: 0.001147, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18484 s, avg_samples: 56.0, ips: 302.95996 samples/s, eta: 3:45:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:06:21] ppocr INFO: epoch: [475/500], global_step: 477100, lr: 0.000100, edit distance: 0.001085, exp_rate: 0.952615, exp_rate<=1 : 0.982656, exp_rate<=2 : 0.996098, exp_rate<=3 : 0.998641, loss: 0.001142, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19724 s, avg_samples: 56.0, ips: 283.91095 samples/s, eta: 3:44:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:06:40] ppocr INFO: epoch: [475/500], global_step: 477200, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.949912, exp_rate<=1 : 0.981414, exp_rate<=2 : 0.995680, exp_rate<=3 : 0.998382, loss: 0.001207, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19266 s, avg_samples: 56.0, ips: 290.66869 samples/s, eta: 3:44:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:06:58] ppocr INFO: epoch: [475/500], global_step: 477300, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.950245, exp_rate<=1 : 0.981668, exp_rate<=2 : 0.995632, exp_rate<=3 : 0.998406, loss: 0.001744, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17822 s, avg_samples: 56.0, ips: 314.21995 samples/s, eta: 3:44:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:07:16] ppocr INFO: epoch: [475/500], global_step: 477400, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.950283, exp_rate<=1 : 0.981748, exp_rate<=2 : 0.995805, exp_rate<=3 : 0.998471, loss: 0.001392, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18675 s, avg_samples: 56.0, ips: 299.86780 samples/s, eta: 3:43:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:07:35] ppocr INFO: epoch: [475/500], global_step: 477500, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.950156, exp_rate<=1 : 0.981449, exp_rate<=2 : 0.995917, exp_rate<=3 : 0.998535, loss: 0.001574, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18449 s, avg_samples: 56.0, ips: 303.53142 samples/s, eta: 3:43:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:07:53] ppocr INFO: epoch: [475/500], global_step: 477600, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.950049, exp_rate<=1 : 0.981281, exp_rate<=2 : 0.995948, exp_rate<=3 : 0.998552, loss: 0.001251, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17743 s, avg_samples: 56.0, ips: 315.61939 samples/s, eta: 3:43:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:08:14] ppocr INFO: epoch: [475/500], global_step: 477700, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.948082, exp_rate<=1 : 0.980196, exp_rate<=2 : 0.995692, exp_rate<=3 : 0.998423, loss: 0.001187, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20924 s, avg_samples: 56.0, ips: 267.63773 samples/s, eta: 3:42:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:08:32] ppocr INFO: epoch: [475/500], global_step: 477800, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.947733, exp_rate<=1 : 0.980048, exp_rate<=2 : 0.995566, exp_rate<=3 : 0.998313, loss: 0.001107, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18690 s, avg_samples: 56.0, ips: 299.63148 samples/s, eta: 3:42:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:08:51] ppocr INFO: epoch: [475/500], global_step: 477900, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.947544, exp_rate<=1 : 0.980027, exp_rate<=2 : 0.995626, exp_rate<=3 : 0.998310, loss: 0.001585, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18722 s, avg_samples: 56.0, ips: 299.10930 samples/s, eta: 3:42:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:09:08] ppocr INFO: epoch: [475/500], global_step: 478000, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.947913, exp_rate<=1 : 0.980245, exp_rate<=2 : 0.995723, exp_rate<=3 : 0.998397, loss: 0.001807, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17304 s, avg_samples: 56.0, ips: 323.62028 samples/s, eta: 3:41:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:09:27] ppocr INFO: epoch: [475/500], global_step: 478100, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.947346, exp_rate<=1 : 0.980049, exp_rate<=2 : 0.995658, exp_rate<=3 : 0.998432, loss: 0.001864, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18920 s, avg_samples: 56.0, ips: 295.98779 samples/s, eta: 3:41:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:09:45] ppocr INFO: epoch: [475/500], global_step: 478200, lr: 0.000100, edit distance: 0.001255, exp_rate: 0.946864, exp_rate<=1 : 0.979908, exp_rate<=2 : 0.995559, exp_rate<=3 : 0.998402, loss: 0.001562, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17392 s, avg_samples: 56.0, ips: 321.99240 samples/s, eta: 3:41:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:10:04] ppocr INFO: epoch: [475/500], global_step: 478300, lr: 0.000100, edit distance: 0.001271, exp_rate: 0.946230, exp_rate<=1 : 0.979648, exp_rate<=2 : 0.995528, exp_rate<=3 : 0.998425, loss: 0.001399, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18699 s, avg_samples: 56.0, ips: 299.47843 samples/s, eta: 3:40:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:10:22] ppocr INFO: epoch: [475/500], global_step: 478400, lr: 0.000100, edit distance: 0.001266, exp_rate: 0.945930, exp_rate<=1 : 0.979621, exp_rate<=2 : 0.995499, exp_rate<=3 : 0.998423, loss: 0.001270, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17977 s, avg_samples: 56.0, ips: 311.50300 samples/s, eta: 3:40:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:10:41] ppocr INFO: epoch: [475/500], global_step: 478500, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.945040, exp_rate<=1 : 0.979278, exp_rate<=2 : 0.995448, exp_rate<=3 : 0.998371, loss: 0.001390, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19389 s, avg_samples: 56.0, ips: 288.81616 samples/s, eta: 3:40:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:10:59] ppocr INFO: epoch: [475/500], global_step: 478600, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.945010, exp_rate<=1 : 0.979277, exp_rate<=2 : 0.995382, exp_rate<=3 : 0.998374, loss: 0.001184, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17744 s, avg_samples: 56.0, ips: 315.59117 samples/s, eta: 3:39:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:11:19] ppocr INFO: epoch: [475/500], global_step: 478700, lr: 0.000100, edit distance: 0.001298, exp_rate: 0.944708, exp_rate<=1 : 0.979044, exp_rate<=2 : 0.995387, exp_rate<=3 : 0.998355, loss: 0.001623, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20236 s, avg_samples: 56.0, ips: 276.73527 samples/s, eta: 3:39:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:11:37] ppocr INFO: epoch: [475/500], global_step: 478800, lr: 0.000100, edit distance: 0.001320, exp_rate: 0.944123, exp_rate<=1 : 0.978893, exp_rate<=2 : 0.995344, exp_rate<=3 : 0.998365, loss: 0.002310, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18084 s, avg_samples: 56.0, ips: 309.67110 samples/s, eta: 3:39:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:11:56] ppocr INFO: epoch: [475/500], global_step: 478900, lr: 0.000100, edit distance: 0.001322, exp_rate: 0.943628, exp_rate<=1 : 0.978658, exp_rate<=2 : 0.995305, exp_rate<=3 : 0.998316, loss: 0.001527, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19048 s, avg_samples: 56.0, ips: 293.99921 samples/s, eta: 3:38:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:12:14] ppocr INFO: epoch: [475/500], global_step: 479000, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.943840, exp_rate<=1 : 0.978811, exp_rate<=2 : 0.995333, exp_rate<=3 : 0.998348, loss: 0.001375, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17747 s, avg_samples: 56.0, ips: 315.54810 samples/s, eta: 3:38:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:12:32] ppocr INFO: epoch: [475/500], global_step: 479100, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.943886, exp_rate<=1 : 0.978835, exp_rate<=2 : 0.995318, exp_rate<=3 : 0.998346, loss: 0.001609, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18439 s, avg_samples: 56.0, ips: 303.70136 samples/s, eta: 3:38:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:12:52] ppocr INFO: epoch: [475/500], global_step: 479200, lr: 0.000100, edit distance: 0.001317, exp_rate: 0.943274, exp_rate<=1 : 0.978576, exp_rate<=2 : 0.995246, exp_rate<=3 : 0.998312, loss: 0.001235, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20116 s, avg_samples: 56.0, ips: 278.38270 samples/s, eta: 3:38:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:13:11] ppocr INFO: epoch: [475/500], global_step: 479300, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.943125, exp_rate<=1 : 0.978556, exp_rate<=2 : 0.995218, exp_rate<=3 : 0.998286, loss: 0.001862, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18323 s, avg_samples: 56.0, ips: 305.63402 samples/s, eta: 3:37:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:13:31] ppocr INFO: epoch: [475/500], global_step: 479400, lr: 0.000100, edit distance: 0.001316, exp_rate: 0.942998, exp_rate<=1 : 0.978572, exp_rate<=2 : 0.995215, exp_rate<=3 : 0.998321, loss: 0.001389, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19786 s, avg_samples: 56.0, ips: 283.03058 samples/s, eta: 3:37:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:13:50] ppocr INFO: epoch: [475/500], global_step: 479500, lr: 0.000100, edit distance: 0.001328, exp_rate: 0.942870, exp_rate<=1 : 0.978512, exp_rate<=2 : 0.995191, exp_rate<=3 : 0.998322, loss: 0.001145, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19010 s, avg_samples: 56.0, ips: 294.57898 samples/s, eta: 3:37:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:13:52] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 04:13:52] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_475 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 04:14:12] ppocr INFO: epoch: [476/500], global_step: 479600, lr: 0.000100, edit distance: 0.000809, exp_rate: 0.952551, exp_rate<=1 : 0.982566, exp_rate<=2 : 0.995933, exp_rate<=3 : 0.998314, loss: 0.001023, avg_reader_cost: 0.03187 s, avg_batch_cost: 0.22097 s, avg_samples: 56.0, ips: 253.43200 samples/s, eta: 3:36:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:14:30] ppocr INFO: epoch: [476/500], global_step: 479700, lr: 0.000100, edit distance: 0.000867, exp_rate: 0.952650, exp_rate<=1 : 0.982612, exp_rate<=2 : 0.996596, exp_rate<=3 : 0.998781, loss: 0.001116, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18501 s, avg_samples: 56.0, ips: 302.68868 samples/s, eta: 3:36:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:14:48] ppocr INFO: epoch: [476/500], global_step: 479800, lr: 0.000100, edit distance: 0.000997, exp_rate: 0.953405, exp_rate<=1 : 0.983469, exp_rate<=2 : 0.996710, exp_rate<=3 : 0.998797, loss: 0.001178, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17734 s, avg_samples: 56.0, ips: 315.77905 samples/s, eta: 3:36:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:15:07] ppocr INFO: epoch: [476/500], global_step: 479900, lr: 0.000100, edit distance: 0.001086, exp_rate: 0.950992, exp_rate<=1 : 0.982374, exp_rate<=2 : 0.996341, exp_rate<=3 : 0.998599, loss: 0.001419, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19099 s, avg_samples: 56.0, ips: 293.20895 samples/s, eta: 3:35:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:15:25] ppocr INFO: epoch: [476/500], global_step: 480000, lr: 0.000100, edit distance: 0.001109, exp_rate: 0.948659, exp_rate<=1 : 0.981234, exp_rate<=2 : 0.995920, exp_rate<=3 : 0.998331, loss: 0.001155, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18156 s, avg_samples: 56.0, ips: 308.44336 samples/s, eta: 3:35:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:15:46] ppocr INFO: epoch: [476/500], global_step: 480100, lr: 0.000100, edit distance: 0.001099, exp_rate: 0.947825, exp_rate<=1 : 0.980269, exp_rate<=2 : 0.995471, exp_rate<=3 : 0.998081, loss: 0.001254, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20784 s, avg_samples: 56.0, ips: 269.43881 samples/s, eta: 3:35:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:16:04] ppocr INFO: epoch: [476/500], global_step: 480200, lr: 0.000100, edit distance: 0.001087, exp_rate: 0.948273, exp_rate<=1 : 0.980685, exp_rate<=2 : 0.995568, exp_rate<=3 : 0.998121, loss: 0.000855, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17506 s, avg_samples: 56.0, ips: 319.88995 samples/s, eta: 3:34:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:16:21] ppocr INFO: epoch: [476/500], global_step: 480300, lr: 0.000100, edit distance: 0.001096, exp_rate: 0.948878, exp_rate<=1 : 0.980994, exp_rate<=2 : 0.995762, exp_rate<=3 : 0.998261, loss: 0.001352, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17318 s, avg_samples: 56.0, ips: 323.36059 samples/s, eta: 3:34:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:16:41] ppocr INFO: epoch: [476/500], global_step: 480400, lr: 0.000100, edit distance: 0.001121, exp_rate: 0.947201, exp_rate<=1 : 0.980184, exp_rate<=2 : 0.995516, exp_rate<=3 : 0.998336, loss: 0.001385, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19732 s, avg_samples: 56.0, ips: 283.79669 samples/s, eta: 3:34:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:16:58] ppocr INFO: epoch: [476/500], global_step: 480500, lr: 0.000100, edit distance: 0.001132, exp_rate: 0.947939, exp_rate<=1 : 0.980611, exp_rate<=2 : 0.995655, exp_rate<=3 : 0.998434, loss: 0.001303, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16914 s, avg_samples: 56.0, ips: 331.08083 samples/s, eta: 3:33:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:17:17] ppocr INFO: epoch: [476/500], global_step: 480600, lr: 0.000100, edit distance: 0.001169, exp_rate: 0.947050, exp_rate<=1 : 0.980369, exp_rate<=2 : 0.995497, exp_rate<=3 : 0.998395, loss: 0.001097, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19272 s, avg_samples: 56.0, ips: 290.57709 samples/s, eta: 3:33:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:17:36] ppocr INFO: epoch: [476/500], global_step: 480700, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.946086, exp_rate<=1 : 0.980261, exp_rate<=2 : 0.995470, exp_rate<=3 : 0.998391, loss: 0.001023, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18581 s, avg_samples: 56.0, ips: 301.38240 samples/s, eta: 3:33:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:17:54] ppocr INFO: epoch: [476/500], global_step: 480800, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.945163, exp_rate<=1 : 0.979756, exp_rate<=2 : 0.995257, exp_rate<=3 : 0.998256, loss: 0.001476, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18875 s, avg_samples: 56.0, ips: 296.68330 samples/s, eta: 3:32:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:18:13] ppocr INFO: epoch: [476/500], global_step: 480900, lr: 0.000100, edit distance: 0.001226, exp_rate: 0.944255, exp_rate<=1 : 0.979394, exp_rate<=2 : 0.995312, exp_rate<=3 : 0.998249, loss: 0.001772, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18999 s, avg_samples: 56.0, ips: 294.74795 samples/s, eta: 3:32:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:18:32] ppocr INFO: epoch: [476/500], global_step: 481000, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.944028, exp_rate<=1 : 0.979419, exp_rate<=2 : 0.995399, exp_rate<=3 : 0.998275, loss: 0.001566, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18192 s, avg_samples: 56.0, ips: 307.83449 samples/s, eta: 3:32:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:18:51] ppocr INFO: epoch: [476/500], global_step: 481100, lr: 0.000100, edit distance: 0.001248, exp_rate: 0.943451, exp_rate<=1 : 0.979394, exp_rate<=2 : 0.995405, exp_rate<=3 : 0.998284, loss: 0.001698, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19629 s, avg_samples: 56.0, ips: 285.28577 samples/s, eta: 3:32:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:19:11] ppocr INFO: epoch: [476/500], global_step: 481200, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.942928, exp_rate<=1 : 0.979111, exp_rate<=2 : 0.995248, exp_rate<=3 : 0.998173, loss: 0.001251, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19962 s, avg_samples: 56.0, ips: 280.53245 samples/s, eta: 3:31:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:19:31] ppocr INFO: epoch: [476/500], global_step: 481300, lr: 0.000100, edit distance: 0.001257, exp_rate: 0.942871, exp_rate<=1 : 0.979035, exp_rate<=2 : 0.995216, exp_rate<=3 : 0.998195, loss: 0.002238, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19146 s, avg_samples: 56.0, ips: 292.49101 samples/s, eta: 3:31:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:19:50] ppocr INFO: epoch: [476/500], global_step: 481400, lr: 0.000100, edit distance: 0.001270, exp_rate: 0.942271, exp_rate<=1 : 0.978729, exp_rate<=2 : 0.995196, exp_rate<=3 : 0.998205, loss: 0.001594, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19725 s, avg_samples: 56.0, ips: 283.90909 samples/s, eta: 3:31:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:20:10] ppocr INFO: epoch: [476/500], global_step: 481500, lr: 0.000100, edit distance: 0.001283, exp_rate: 0.941794, exp_rate<=1 : 0.978440, exp_rate<=2 : 0.995140, exp_rate<=3 : 0.998202, loss: 0.001884, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19675 s, avg_samples: 56.0, ips: 284.61962 samples/s, eta: 3:30:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:20:28] ppocr INFO: epoch: [476/500], global_step: 481600, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.941810, exp_rate<=1 : 0.978422, exp_rate<=2 : 0.995125, exp_rate<=3 : 0.998201, loss: 0.001024, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18071 s, avg_samples: 56.0, ips: 309.88180 samples/s, eta: 3:30:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:20:47] ppocr INFO: epoch: [476/500], global_step: 481700, lr: 0.000100, edit distance: 0.001288, exp_rate: 0.941776, exp_rate<=1 : 0.978483, exp_rate<=2 : 0.995160, exp_rate<=3 : 0.998224, loss: 0.002223, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18974 s, avg_samples: 56.0, ips: 295.13873 samples/s, eta: 3:30:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:21:05] ppocr INFO: epoch: [476/500], global_step: 481800, lr: 0.000100, edit distance: 0.001289, exp_rate: 0.941561, exp_rate<=1 : 0.978129, exp_rate<=2 : 0.995076, exp_rate<=3 : 0.998215, loss: 0.001169, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.18253 s, avg_samples: 56.0, ips: 306.79323 samples/s, eta: 3:29:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:21:24] ppocr INFO: epoch: [476/500], global_step: 481900, lr: 0.000100, edit distance: 0.001291, exp_rate: 0.941395, exp_rate<=1 : 0.978071, exp_rate<=2 : 0.995062, exp_rate<=3 : 0.998222, loss: 0.001550, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18997 s, avg_samples: 56.0, ips: 294.79022 samples/s, eta: 3:29:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:21:42] ppocr INFO: epoch: [476/500], global_step: 482000, lr: 0.000100, edit distance: 0.001296, exp_rate: 0.941475, exp_rate<=1 : 0.978100, exp_rate<=2 : 0.995112, exp_rate<=3 : 0.998258, loss: 0.001120, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17864 s, avg_samples: 56.0, ips: 313.48550 samples/s, eta: 3:29:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:22:00] ppocr INFO: epoch: [476/500], global_step: 482100, lr: 0.000100, edit distance: 0.001295, exp_rate: 0.941564, exp_rate<=1 : 0.978126, exp_rate<=2 : 0.995125, exp_rate<=3 : 0.998276, loss: 0.001636, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18039 s, avg_samples: 56.0, ips: 310.43881 samples/s, eta: 3:28:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:22:20] ppocr INFO: epoch: [476/500], global_step: 482200, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.941433, exp_rate<=1 : 0.978067, exp_rate<=2 : 0.995108, exp_rate<=3 : 0.998255, loss: 0.001703, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19215 s, avg_samples: 56.0, ips: 291.44387 samples/s, eta: 3:28:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:22:28] ppocr INFO: epoch: [476/500], global_step: 482240, lr: 0.000100, edit distance: 0.001300, exp_rate: 0.941344, exp_rate<=1 : 0.978031, exp_rate<=2 : 0.995112, exp_rate<=3 : 0.998253, loss: 0.001499, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.08049 s, avg_samples: 22.4, ips: 278.28306 samples/s, eta: 3:28:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:22:30] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 04:22:41] ppocr INFO: epoch: [477/500], global_step: 482300, lr: 0.000100, edit distance: 0.000766, exp_rate: 0.953060, exp_rate<=1 : 0.985461, exp_rate<=2 : 0.997810, exp_rate<=3 : 0.999293, loss: 0.001165, avg_reader_cost: 0.02724 s, avg_batch_cost: 0.13837 s, avg_samples: 33.6, ips: 242.82585 samples/s, eta: 3:28:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:23:01] ppocr INFO: epoch: [477/500], global_step: 482400, lr: 0.000100, edit distance: 0.000940, exp_rate: 0.951316, exp_rate<=1 : 0.982518, exp_rate<=2 : 0.996693, exp_rate<=3 : 0.998737, loss: 0.001422, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19085 s, avg_samples: 56.0, ips: 293.42857 samples/s, eta: 3:27:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:23:19] ppocr INFO: epoch: [477/500], global_step: 482500, lr: 0.000100, edit distance: 0.000937, exp_rate: 0.954447, exp_rate<=1 : 0.983694, exp_rate<=2 : 0.997212, exp_rate<=3 : 0.999073, loss: 0.001252, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18175 s, avg_samples: 56.0, ips: 308.11311 samples/s, eta: 3:27:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:23:37] ppocr INFO: epoch: [477/500], global_step: 482600, lr: 0.000100, edit distance: 0.001021, exp_rate: 0.952985, exp_rate<=1 : 0.982470, exp_rate<=2 : 0.996774, exp_rate<=3 : 0.998930, loss: 0.000670, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17786 s, avg_samples: 56.0, ips: 314.84910 samples/s, eta: 3:27:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:23:57] ppocr INFO: epoch: [477/500], global_step: 482700, lr: 0.000100, edit distance: 0.001057, exp_rate: 0.950784, exp_rate<=1 : 0.981687, exp_rate<=2 : 0.996284, exp_rate<=3 : 0.998850, loss: 0.001138, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20303 s, avg_samples: 56.0, ips: 275.82394 samples/s, eta: 3:26:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:24:15] ppocr INFO: epoch: [477/500], global_step: 482800, lr: 0.000100, edit distance: 0.001056, exp_rate: 0.950274, exp_rate<=1 : 0.981082, exp_rate<=2 : 0.995972, exp_rate<=3 : 0.998643, loss: 0.001413, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18618 s, avg_samples: 56.0, ips: 300.78322 samples/s, eta: 3:26:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:24:34] ppocr INFO: epoch: [477/500], global_step: 482900, lr: 0.000100, edit distance: 0.001039, exp_rate: 0.950712, exp_rate<=1 : 0.981452, exp_rate<=2 : 0.995914, exp_rate<=3 : 0.998545, loss: 0.000893, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18795 s, avg_samples: 56.0, ips: 297.95228 samples/s, eta: 3:26:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:24:54] ppocr INFO: epoch: [477/500], global_step: 483000, lr: 0.000100, edit distance: 0.001054, exp_rate: 0.951273, exp_rate<=1 : 0.981502, exp_rate<=2 : 0.996139, exp_rate<=3 : 0.998652, loss: 0.001298, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19301 s, avg_samples: 56.0, ips: 290.13649 samples/s, eta: 3:25:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:25:12] ppocr INFO: epoch: [477/500], global_step: 483100, lr: 0.000100, edit distance: 0.001069, exp_rate: 0.951275, exp_rate<=1 : 0.981599, exp_rate<=2 : 0.996179, exp_rate<=3 : 0.998671, loss: 0.001308, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17901 s, avg_samples: 56.0, ips: 312.83390 samples/s, eta: 3:25:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:25:30] ppocr INFO: epoch: [477/500], global_step: 483200, lr: 0.000100, edit distance: 0.001084, exp_rate: 0.951135, exp_rate<=1 : 0.981505, exp_rate<=2 : 0.996076, exp_rate<=3 : 0.998685, loss: 0.001150, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18113 s, avg_samples: 56.0, ips: 309.17059 samples/s, eta: 3:25:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:25:48] ppocr INFO: epoch: [477/500], global_step: 483300, lr: 0.000100, edit distance: 0.001077, exp_rate: 0.950609, exp_rate<=1 : 0.981337, exp_rate<=2 : 0.996024, exp_rate<=3 : 0.998626, loss: 0.001441, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18425 s, avg_samples: 56.0, ips: 303.93025 samples/s, eta: 3:25:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:26:06] ppocr INFO: epoch: [477/500], global_step: 483400, lr: 0.000100, edit distance: 0.001096, exp_rate: 0.950051, exp_rate<=1 : 0.981188, exp_rate<=2 : 0.996040, exp_rate<=3 : 0.998660, loss: 0.001344, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17421 s, avg_samples: 56.0, ips: 321.44934 samples/s, eta: 3:24:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:26:25] ppocr INFO: epoch: [477/500], global_step: 483500, lr: 0.000100, edit distance: 0.001118, exp_rate: 0.948771, exp_rate<=1 : 0.980509, exp_rate<=2 : 0.995844, exp_rate<=3 : 0.998629, loss: 0.001315, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19534 s, avg_samples: 56.0, ips: 286.67687 samples/s, eta: 3:24:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:26:46] ppocr INFO: epoch: [477/500], global_step: 483600, lr: 0.000100, edit distance: 0.001118, exp_rate: 0.948164, exp_rate<=1 : 0.980272, exp_rate<=2 : 0.995819, exp_rate<=3 : 0.998640, loss: 0.000998, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20634 s, avg_samples: 56.0, ips: 271.39109 samples/s, eta: 3:24:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:27:05] ppocr INFO: epoch: [477/500], global_step: 483700, lr: 0.000100, edit distance: 0.001110, exp_rate: 0.948089, exp_rate<=1 : 0.980164, exp_rate<=2 : 0.995735, exp_rate<=3 : 0.998665, loss: 0.001281, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19561 s, avg_samples: 56.0, ips: 286.27795 samples/s, eta: 3:23:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:27:24] ppocr INFO: epoch: [477/500], global_step: 483800, lr: 0.000100, edit distance: 0.001099, exp_rate: 0.948297, exp_rate<=1 : 0.980145, exp_rate<=2 : 0.995768, exp_rate<=3 : 0.998641, loss: 0.001274, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18779 s, avg_samples: 56.0, ips: 298.20735 samples/s, eta: 3:23:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:27:42] ppocr INFO: epoch: [477/500], global_step: 483900, lr: 0.000100, edit distance: 0.001100, exp_rate: 0.948181, exp_rate<=1 : 0.980124, exp_rate<=2 : 0.995713, exp_rate<=3 : 0.998604, loss: 0.001246, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18257 s, avg_samples: 56.0, ips: 306.72532 samples/s, eta: 3:23:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:28:01] ppocr INFO: epoch: [477/500], global_step: 484000, lr: 0.000100, edit distance: 0.001115, exp_rate: 0.948193, exp_rate<=1 : 0.980163, exp_rate<=2 : 0.995808, exp_rate<=3 : 0.998657, loss: 0.001348, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18986 s, avg_samples: 56.0, ips: 294.94645 samples/s, eta: 3:22:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:28:20] ppocr INFO: epoch: [477/500], global_step: 484100, lr: 0.000100, edit distance: 0.001107, exp_rate: 0.948363, exp_rate<=1 : 0.980083, exp_rate<=2 : 0.995757, exp_rate<=3 : 0.998659, loss: 0.001064, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18413 s, avg_samples: 56.0, ips: 304.13430 samples/s, eta: 3:22:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:28:38] ppocr INFO: epoch: [477/500], global_step: 484200, lr: 0.000100, edit distance: 0.001105, exp_rate: 0.948640, exp_rate<=1 : 0.980337, exp_rate<=2 : 0.995774, exp_rate<=3 : 0.998658, loss: 0.001535, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17841 s, avg_samples: 56.0, ips: 313.88267 samples/s, eta: 3:22:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:28:56] ppocr INFO: epoch: [477/500], global_step: 484300, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.947984, exp_rate<=1 : 0.980220, exp_rate<=2 : 0.995743, exp_rate<=3 : 0.998643, loss: 0.001543, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18228 s, avg_samples: 56.0, ips: 307.22035 samples/s, eta: 3:21:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:29:14] ppocr INFO: epoch: [477/500], global_step: 484400, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.947783, exp_rate<=1 : 0.980208, exp_rate<=2 : 0.995798, exp_rate<=3 : 0.998655, loss: 0.001325, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17821 s, avg_samples: 56.0, ips: 314.23984 samples/s, eta: 3:21:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:29:32] ppocr INFO: epoch: [477/500], global_step: 484500, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.947397, exp_rate<=1 : 0.980006, exp_rate<=2 : 0.995750, exp_rate<=3 : 0.998614, loss: 0.001125, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18350 s, avg_samples: 56.0, ips: 305.18125 samples/s, eta: 3:21:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:29:52] ppocr INFO: epoch: [477/500], global_step: 484600, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.946455, exp_rate<=1 : 0.979675, exp_rate<=2 : 0.995634, exp_rate<=3 : 0.998538, loss: 0.001357, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20219 s, avg_samples: 56.0, ips: 276.96835 samples/s, eta: 3:20:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:30:12] ppocr INFO: epoch: [477/500], global_step: 484700, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.945505, exp_rate<=1 : 0.979336, exp_rate<=2 : 0.995589, exp_rate<=3 : 0.998528, loss: 0.001370, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19456 s, avg_samples: 56.0, ips: 287.83604 samples/s, eta: 3:20:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:30:31] ppocr INFO: epoch: [477/500], global_step: 484800, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.945059, exp_rate<=1 : 0.979026, exp_rate<=2 : 0.995502, exp_rate<=3 : 0.998471, loss: 0.001802, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18748 s, avg_samples: 56.0, ips: 298.69753 samples/s, eta: 3:20:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:30:50] ppocr INFO: epoch: [477/500], global_step: 484900, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.944497, exp_rate<=1 : 0.978782, exp_rate<=2 : 0.995384, exp_rate<=3 : 0.998434, loss: 0.000984, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18959 s, avg_samples: 56.0, ips: 295.36895 samples/s, eta: 3:19:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:31:04] ppocr INFO: epoch: [477/500], global_step: 484980, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.944620, exp_rate<=1 : 0.978808, exp_rate<=2 : 0.995393, exp_rate<=3 : 0.998436, loss: 0.002838, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14828 s, avg_samples: 44.8, ips: 302.12650 samples/s, eta: 3:19:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:31:07] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 04:31:11] ppocr INFO: epoch: [478/500], global_step: 485000, lr: 0.000100, edit distance: 0.001707, exp_rate: 0.928504, exp_rate<=1 : 0.971383, exp_rate<=2 : 0.992602, exp_rate<=3 : 0.995707, loss: 0.001134, avg_reader_cost: 0.02828 s, avg_batch_cost: 0.06984 s, avg_samples: 11.2, ips: 160.37659 samples/s, eta: 3:19:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:31:30] ppocr INFO: epoch: [478/500], global_step: 485100, lr: 0.000100, edit distance: 0.001338, exp_rate: 0.942875, exp_rate<=1 : 0.979220, exp_rate<=2 : 0.995313, exp_rate<=3 : 0.998384, loss: 0.001534, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18683 s, avg_samples: 56.0, ips: 299.73687 samples/s, eta: 3:19:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:31:50] ppocr INFO: epoch: [478/500], global_step: 485200, lr: 0.000100, edit distance: 0.001112, exp_rate: 0.946342, exp_rate<=1 : 0.980357, exp_rate<=2 : 0.995312, exp_rate<=3 : 0.998016, loss: 0.001090, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19696 s, avg_samples: 56.0, ips: 284.31675 samples/s, eta: 3:19:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:32:11] ppocr INFO: epoch: [478/500], global_step: 485300, lr: 0.000100, edit distance: 0.001086, exp_rate: 0.945929, exp_rate<=1 : 0.980406, exp_rate<=2 : 0.995478, exp_rate<=3 : 0.998369, loss: 0.001411, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.21044 s, avg_samples: 56.0, ips: 266.11047 samples/s, eta: 3:18:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:32:30] ppocr INFO: epoch: [478/500], global_step: 485400, lr: 0.000100, edit distance: 0.001102, exp_rate: 0.946189, exp_rate<=1 : 0.980741, exp_rate<=2 : 0.995945, exp_rate<=3 : 0.998477, loss: 0.001539, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18677 s, avg_samples: 56.0, ips: 299.82830 samples/s, eta: 3:18:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:32:48] ppocr INFO: epoch: [478/500], global_step: 485500, lr: 0.000100, edit distance: 0.001042, exp_rate: 0.948315, exp_rate<=1 : 0.981278, exp_rate<=2 : 0.996330, exp_rate<=3 : 0.998636, loss: 0.001100, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17849 s, avg_samples: 56.0, ips: 313.73813 samples/s, eta: 3:18:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:33:06] ppocr INFO: epoch: [478/500], global_step: 485600, lr: 0.000100, edit distance: 0.001068, exp_rate: 0.947983, exp_rate<=1 : 0.981317, exp_rate<=2 : 0.996572, exp_rate<=3 : 0.998801, loss: 0.001483, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18325 s, avg_samples: 56.0, ips: 305.58584 samples/s, eta: 3:17:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:33:26] ppocr INFO: epoch: [478/500], global_step: 485700, lr: 0.000100, edit distance: 0.001055, exp_rate: 0.948103, exp_rate<=1 : 0.981074, exp_rate<=2 : 0.996421, exp_rate<=3 : 0.998826, loss: 0.001897, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20070 s, avg_samples: 56.0, ips: 279.01859 samples/s, eta: 3:17:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:33:46] ppocr INFO: epoch: [478/500], global_step: 485800, lr: 0.000100, edit distance: 0.001119, exp_rate: 0.944231, exp_rate<=1 : 0.979331, exp_rate<=2 : 0.995892, exp_rate<=3 : 0.998307, loss: 0.001651, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20008 s, avg_samples: 56.0, ips: 279.89131 samples/s, eta: 3:17:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:34:06] ppocr INFO: epoch: [478/500], global_step: 485900, lr: 0.000100, edit distance: 0.001124, exp_rate: 0.943454, exp_rate<=1 : 0.979172, exp_rate<=2 : 0.995710, exp_rate<=3 : 0.998238, loss: 0.001054, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19716 s, avg_samples: 56.0, ips: 284.03416 samples/s, eta: 3:16:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:34:26] ppocr INFO: epoch: [478/500], global_step: 486000, lr: 0.000100, edit distance: 0.001127, exp_rate: 0.943273, exp_rate<=1 : 0.978912, exp_rate<=2 : 0.995493, exp_rate<=3 : 0.998217, loss: 0.000989, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19925 s, avg_samples: 56.0, ips: 281.05950 samples/s, eta: 3:16:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:34:44] ppocr INFO: epoch: [478/500], global_step: 486100, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.943967, exp_rate<=1 : 0.979248, exp_rate<=2 : 0.995520, exp_rate<=3 : 0.998207, loss: 0.001067, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18253 s, avg_samples: 56.0, ips: 306.79513 samples/s, eta: 3:16:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:35:02] ppocr INFO: epoch: [478/500], global_step: 486200, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.944459, exp_rate<=1 : 0.979545, exp_rate<=2 : 0.995622, exp_rate<=3 : 0.998281, loss: 0.000814, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18448 s, avg_samples: 56.0, ips: 303.56030 samples/s, eta: 3:15:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:35:21] ppocr INFO: epoch: [478/500], global_step: 486300, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.944345, exp_rate<=1 : 0.979471, exp_rate<=2 : 0.995573, exp_rate<=3 : 0.998287, loss: 0.000969, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18191 s, avg_samples: 56.0, ips: 307.84312 samples/s, eta: 3:15:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:35:39] ppocr INFO: epoch: [478/500], global_step: 486400, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.944040, exp_rate<=1 : 0.979507, exp_rate<=2 : 0.995683, exp_rate<=3 : 0.998350, loss: 0.001323, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18192 s, avg_samples: 56.0, ips: 307.82926 samples/s, eta: 3:15:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:35:59] ppocr INFO: epoch: [478/500], global_step: 486500, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.943617, exp_rate<=1 : 0.979295, exp_rate<=2 : 0.995645, exp_rate<=3 : 0.998345, loss: 0.001444, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19632 s, avg_samples: 56.0, ips: 285.25469 samples/s, eta: 3:14:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:36:17] ppocr INFO: epoch: [478/500], global_step: 486600, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.943749, exp_rate<=1 : 0.979443, exp_rate<=2 : 0.995631, exp_rate<=3 : 0.998328, loss: 0.001660, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18198 s, avg_samples: 56.0, ips: 307.72687 samples/s, eta: 3:14:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:36:35] ppocr INFO: epoch: [478/500], global_step: 486700, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.943836, exp_rate<=1 : 0.979427, exp_rate<=2 : 0.995624, exp_rate<=3 : 0.998330, loss: 0.001136, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17997 s, avg_samples: 56.0, ips: 311.15757 samples/s, eta: 3:14:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:36:53] ppocr INFO: epoch: [478/500], global_step: 486800, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.943802, exp_rate<=1 : 0.979372, exp_rate<=2 : 0.995676, exp_rate<=3 : 0.998356, loss: 0.001878, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18634 s, avg_samples: 56.0, ips: 300.53351 samples/s, eta: 3:13:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:37:12] ppocr INFO: epoch: [478/500], global_step: 486900, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.943503, exp_rate<=1 : 0.979244, exp_rate<=2 : 0.995599, exp_rate<=3 : 0.998333, loss: 0.001353, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18816 s, avg_samples: 56.0, ips: 297.61998 samples/s, eta: 3:13:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:37:30] ppocr INFO: epoch: [478/500], global_step: 487000, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.943968, exp_rate<=1 : 0.979471, exp_rate<=2 : 0.995665, exp_rate<=3 : 0.998359, loss: 0.001531, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17980 s, avg_samples: 56.0, ips: 311.44914 samples/s, eta: 3:13:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:37:49] ppocr INFO: epoch: [478/500], global_step: 487100, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.943787, exp_rate<=1 : 0.979336, exp_rate<=2 : 0.995617, exp_rate<=3 : 0.998344, loss: 0.001779, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18580 s, avg_samples: 56.0, ips: 301.39976 samples/s, eta: 3:12:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:38:08] ppocr INFO: epoch: [478/500], global_step: 487200, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.943164, exp_rate<=1 : 0.979055, exp_rate<=2 : 0.995544, exp_rate<=3 : 0.998339, loss: 0.001307, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19420 s, avg_samples: 56.0, ips: 288.36690 samples/s, eta: 3:12:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:38:26] ppocr INFO: epoch: [478/500], global_step: 487300, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.942981, exp_rate<=1 : 0.979004, exp_rate<=2 : 0.995511, exp_rate<=3 : 0.998335, loss: 0.001216, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17364 s, avg_samples: 56.0, ips: 322.50008 samples/s, eta: 3:12:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:38:45] ppocr INFO: epoch: [478/500], global_step: 487400, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.942743, exp_rate<=1 : 0.979014, exp_rate<=2 : 0.995480, exp_rate<=3 : 0.998293, loss: 0.001488, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19047 s, avg_samples: 56.0, ips: 294.00446 samples/s, eta: 3:12:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:39:03] ppocr INFO: epoch: [478/500], global_step: 487500, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.942900, exp_rate<=1 : 0.978915, exp_rate<=2 : 0.995521, exp_rate<=3 : 0.998303, loss: 0.001460, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17803 s, avg_samples: 56.0, ips: 314.55585 samples/s, eta: 3:11:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:39:21] ppocr INFO: epoch: [478/500], global_step: 487600, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.943129, exp_rate<=1 : 0.979057, exp_rate<=2 : 0.995532, exp_rate<=3 : 0.998298, loss: 0.001568, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18622 s, avg_samples: 56.0, ips: 300.71187 samples/s, eta: 3:11:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:39:40] ppocr INFO: epoch: [478/500], global_step: 487700, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.942644, exp_rate<=1 : 0.978839, exp_rate<=2 : 0.995425, exp_rate<=3 : 0.998261, loss: 0.001269, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18875 s, avg_samples: 56.0, ips: 296.68246 samples/s, eta: 3:11:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:39:43] ppocr INFO: epoch: [478/500], global_step: 487720, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.942704, exp_rate<=1 : 0.978840, exp_rate<=2 : 0.995414, exp_rate<=3 : 0.998260, loss: 0.001396, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03387 s, avg_samples: 11.2, ips: 330.65169 samples/s, eta: 3:11:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:39:46] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 04:40:01] ppocr INFO: epoch: [479/500], global_step: 487800, lr: 0.000100, edit distance: 0.000969, exp_rate: 0.952850, exp_rate<=1 : 0.981629, exp_rate<=2 : 0.997661, exp_rate<=3 : 0.999182, loss: 0.001036, avg_reader_cost: 0.02553 s, avg_batch_cost: 0.17745 s, avg_samples: 44.8, ips: 252.47188 samples/s, eta: 3:10:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:40:20] ppocr INFO: epoch: [479/500], global_step: 487900, lr: 0.000100, edit distance: 0.001093, exp_rate: 0.948062, exp_rate<=1 : 0.978665, exp_rate<=2 : 0.995549, exp_rate<=3 : 0.998697, loss: 0.001372, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18516 s, avg_samples: 56.0, ips: 302.43608 samples/s, eta: 3:10:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:40:38] ppocr INFO: epoch: [479/500], global_step: 488000, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.947570, exp_rate<=1 : 0.978858, exp_rate<=2 : 0.995181, exp_rate<=3 : 0.998335, loss: 0.001924, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18632 s, avg_samples: 56.0, ips: 300.55580 samples/s, eta: 3:10:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:40:59] ppocr INFO: epoch: [479/500], global_step: 488100, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.946574, exp_rate<=1 : 0.978432, exp_rate<=2 : 0.995099, exp_rate<=3 : 0.998099, loss: 0.001088, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20331 s, avg_samples: 56.0, ips: 275.43949 samples/s, eta: 3:09:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:41:18] ppocr INFO: epoch: [479/500], global_step: 488200, lr: 0.000100, edit distance: 0.001151, exp_rate: 0.946410, exp_rate<=1 : 0.978196, exp_rate<=2 : 0.994717, exp_rate<=3 : 0.998003, loss: 0.001272, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18797 s, avg_samples: 56.0, ips: 297.91621 samples/s, eta: 3:09:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:41:36] ppocr INFO: epoch: [479/500], global_step: 488300, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.948116, exp_rate<=1 : 0.979459, exp_rate<=2 : 0.995131, exp_rate<=3 : 0.998222, loss: 0.001275, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18297 s, avg_samples: 56.0, ips: 306.06843 samples/s, eta: 3:09:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:41:54] ppocr INFO: epoch: [479/500], global_step: 488400, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.947947, exp_rate<=1 : 0.979332, exp_rate<=2 : 0.995117, exp_rate<=3 : 0.998231, loss: 0.001230, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18282 s, avg_samples: 56.0, ips: 306.31611 samples/s, eta: 3:08:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:42:12] ppocr INFO: epoch: [479/500], global_step: 488500, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.947835, exp_rate<=1 : 0.979356, exp_rate<=2 : 0.995094, exp_rate<=3 : 0.998305, loss: 0.000793, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17917 s, avg_samples: 56.0, ips: 312.55204 samples/s, eta: 3:08:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:42:31] ppocr INFO: epoch: [479/500], global_step: 488600, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.946817, exp_rate<=1 : 0.979315, exp_rate<=2 : 0.995338, exp_rate<=3 : 0.998439, loss: 0.001884, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18450 s, avg_samples: 56.0, ips: 303.52199 samples/s, eta: 3:08:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:42:48] ppocr INFO: epoch: [479/500], global_step: 488700, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.946198, exp_rate<=1 : 0.979057, exp_rate<=2 : 0.995252, exp_rate<=3 : 0.998381, loss: 0.001034, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17770 s, avg_samples: 56.0, ips: 315.14536 samples/s, eta: 3:07:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:43:07] ppocr INFO: epoch: [479/500], global_step: 488800, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.945627, exp_rate<=1 : 0.978788, exp_rate<=2 : 0.995248, exp_rate<=3 : 0.998405, loss: 0.001733, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19033 s, avg_samples: 56.0, ips: 294.23017 samples/s, eta: 3:07:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:43:27] ppocr INFO: epoch: [479/500], global_step: 488900, lr: 0.000100, edit distance: 0.001177, exp_rate: 0.944720, exp_rate<=1 : 0.978416, exp_rate<=2 : 0.995205, exp_rate<=3 : 0.998398, loss: 0.001347, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19763 s, avg_samples: 56.0, ips: 283.35294 samples/s, eta: 3:07:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:43:47] ppocr INFO: epoch: [479/500], global_step: 489000, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.944325, exp_rate<=1 : 0.978281, exp_rate<=2 : 0.995084, exp_rate<=3 : 0.998309, loss: 0.001706, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19511 s, avg_samples: 56.0, ips: 287.01756 samples/s, eta: 3:06:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:44:05] ppocr INFO: epoch: [479/500], global_step: 489100, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.944272, exp_rate<=1 : 0.978498, exp_rate<=2 : 0.995202, exp_rate<=3 : 0.998371, loss: 0.001302, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18627 s, avg_samples: 56.0, ips: 300.63652 samples/s, eta: 3:06:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:44:25] ppocr INFO: epoch: [479/500], global_step: 489200, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.944189, exp_rate<=1 : 0.978622, exp_rate<=2 : 0.995215, exp_rate<=3 : 0.998396, loss: 0.001348, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19649 s, avg_samples: 56.0, ips: 285.00135 samples/s, eta: 3:06:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:44:43] ppocr INFO: epoch: [479/500], global_step: 489300, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.944505, exp_rate<=1 : 0.978678, exp_rate<=2 : 0.995205, exp_rate<=3 : 0.998326, loss: 0.001498, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18229 s, avg_samples: 56.0, ips: 307.19609 samples/s, eta: 3:06:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:45:02] ppocr INFO: epoch: [479/500], global_step: 489400, lr: 0.000100, edit distance: 0.001221, exp_rate: 0.944395, exp_rate<=1 : 0.978643, exp_rate<=2 : 0.995062, exp_rate<=3 : 0.998258, loss: 0.001426, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18882 s, avg_samples: 56.0, ips: 296.57258 samples/s, eta: 3:05:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:45:22] ppocr INFO: epoch: [479/500], global_step: 489500, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.944180, exp_rate<=1 : 0.978556, exp_rate<=2 : 0.994956, exp_rate<=3 : 0.998169, loss: 0.001595, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19875 s, avg_samples: 56.0, ips: 281.76328 samples/s, eta: 3:05:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:45:41] ppocr INFO: epoch: [479/500], global_step: 489600, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.943627, exp_rate<=1 : 0.978441, exp_rate<=2 : 0.994959, exp_rate<=3 : 0.998148, loss: 0.001506, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18965 s, avg_samples: 56.0, ips: 295.28154 samples/s, eta: 3:05:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:46:00] ppocr INFO: epoch: [479/500], global_step: 489700, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.943497, exp_rate<=1 : 0.978489, exp_rate<=2 : 0.994977, exp_rate<=3 : 0.998203, loss: 0.001284, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18472 s, avg_samples: 56.0, ips: 303.16454 samples/s, eta: 3:04:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:46:18] ppocr INFO: epoch: [479/500], global_step: 489800, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.943303, exp_rate<=1 : 0.978524, exp_rate<=2 : 0.995051, exp_rate<=3 : 0.998245, loss: 0.001173, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18063 s, avg_samples: 56.0, ips: 310.03163 samples/s, eta: 3:04:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:46:35] ppocr INFO: epoch: [479/500], global_step: 489900, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.943479, exp_rate<=1 : 0.978500, exp_rate<=2 : 0.995011, exp_rate<=3 : 0.998192, loss: 0.000817, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17568 s, avg_samples: 56.0, ips: 318.75983 samples/s, eta: 3:04:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:46:52] ppocr INFO: epoch: [479/500], global_step: 490000, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.943880, exp_rate<=1 : 0.978681, exp_rate<=2 : 0.995068, exp_rate<=3 : 0.998215, loss: 0.001438, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16972 s, avg_samples: 56.0, ips: 329.95825 samples/s, eta: 3:03:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:47:11] ppocr INFO: epoch: [479/500], global_step: 490100, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.943835, exp_rate<=1 : 0.978649, exp_rate<=2 : 0.995113, exp_rate<=3 : 0.998238, loss: 0.001378, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19152 s, avg_samples: 56.0, ips: 292.39270 samples/s, eta: 3:03:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:47:32] ppocr INFO: epoch: [479/500], global_step: 490200, lr: 0.000100, edit distance: 0.001245, exp_rate: 0.943083, exp_rate<=1 : 0.978240, exp_rate<=2 : 0.994976, exp_rate<=3 : 0.998193, loss: 0.001792, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20827 s, avg_samples: 56.0, ips: 268.88816 samples/s, eta: 3:03:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:47:51] ppocr INFO: epoch: [479/500], global_step: 490300, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.943206, exp_rate<=1 : 0.978269, exp_rate<=2 : 0.994991, exp_rate<=3 : 0.998222, loss: 0.001005, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18553 s, avg_samples: 56.0, ips: 301.83563 samples/s, eta: 3:02:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:48:09] ppocr INFO: epoch: [479/500], global_step: 490400, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.943548, exp_rate<=1 : 0.978427, exp_rate<=2 : 0.995075, exp_rate<=3 : 0.998274, loss: 0.001195, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17985 s, avg_samples: 56.0, ips: 311.36586 samples/s, eta: 3:02:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:48:21] ppocr INFO: epoch: [479/500], global_step: 490460, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.943379, exp_rate<=1 : 0.978374, exp_rate<=2 : 0.995024, exp_rate<=3 : 0.998284, loss: 0.001644, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.11682 s, avg_samples: 33.6, ips: 287.63084 samples/s, eta: 3:02:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:48:23] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 04:48:32] ppocr INFO: epoch: [480/500], global_step: 490500, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.931500, exp_rate<=1 : 0.967790, exp_rate<=2 : 0.989918, exp_rate<=3 : 0.995901, loss: 0.001294, avg_reader_cost: 0.02881 s, avg_batch_cost: 0.11045 s, avg_samples: 22.4, ips: 202.80876 samples/s, eta: 3:02:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:48:52] ppocr INFO: epoch: [480/500], global_step: 490600, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.933837, exp_rate<=1 : 0.971673, exp_rate<=2 : 0.992572, exp_rate<=3 : 0.996895, loss: 0.001706, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20234 s, avg_samples: 56.0, ips: 276.75919 samples/s, eta: 3:01:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:49:11] ppocr INFO: epoch: [480/500], global_step: 490700, lr: 0.000100, edit distance: 0.001500, exp_rate: 0.921870, exp_rate<=1 : 0.966920, exp_rate<=2 : 0.989464, exp_rate<=3 : 0.995197, loss: 0.000869, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19198 s, avg_samples: 56.0, ips: 291.70228 samples/s, eta: 3:01:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:49:29] ppocr INFO: epoch: [480/500], global_step: 490800, lr: 0.000100, edit distance: 0.001376, exp_rate: 0.929799, exp_rate<=1 : 0.971244, exp_rate<=2 : 0.991416, exp_rate<=3 : 0.996218, loss: 0.001287, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17873 s, avg_samples: 56.0, ips: 313.31894 samples/s, eta: 3:01:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:49:50] ppocr INFO: epoch: [480/500], global_step: 490900, lr: 0.000100, edit distance: 0.001363, exp_rate: 0.930645, exp_rate<=1 : 0.970861, exp_rate<=2 : 0.991082, exp_rate<=3 : 0.995935, loss: 0.001568, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21172 s, avg_samples: 56.0, ips: 264.49798 samples/s, eta: 3:00:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:50:09] ppocr INFO: epoch: [480/500], global_step: 491000, lr: 0.000100, edit distance: 0.001323, exp_rate: 0.932628, exp_rate<=1 : 0.972112, exp_rate<=2 : 0.991809, exp_rate<=3 : 0.996485, loss: 0.000958, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18991 s, avg_samples: 56.0, ips: 294.88047 samples/s, eta: 3:00:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:50:27] ppocr INFO: epoch: [480/500], global_step: 491100, lr: 0.000100, edit distance: 0.001258, exp_rate: 0.935485, exp_rate<=1 : 0.973788, exp_rate<=2 : 0.992665, exp_rate<=3 : 0.996913, loss: 0.000961, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18160 s, avg_samples: 56.0, ips: 308.37344 samples/s, eta: 3:00:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:50:46] ppocr INFO: epoch: [480/500], global_step: 491200, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.937946, exp_rate<=1 : 0.974773, exp_rate<=2 : 0.992992, exp_rate<=3 : 0.997164, loss: 0.000848, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18756 s, avg_samples: 56.0, ips: 298.57794 samples/s, eta: 2:59:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:51:05] ppocr INFO: epoch: [480/500], global_step: 491300, lr: 0.000100, edit distance: 0.001183, exp_rate: 0.939896, exp_rate<=1 : 0.975628, exp_rate<=2 : 0.993355, exp_rate<=3 : 0.997232, loss: 0.002002, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18391 s, avg_samples: 56.0, ips: 304.50030 samples/s, eta: 2:59:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:51:22] ppocr INFO: epoch: [480/500], global_step: 491400, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.941880, exp_rate<=1 : 0.976520, exp_rate<=2 : 0.993677, exp_rate<=3 : 0.997453, loss: 0.001135, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17679 s, avg_samples: 56.0, ips: 316.75818 samples/s, eta: 2:59:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:51:41] ppocr INFO: epoch: [480/500], global_step: 491500, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.942053, exp_rate<=1 : 0.976787, exp_rate<=2 : 0.993848, exp_rate<=3 : 0.997591, loss: 0.000971, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18910 s, avg_samples: 56.0, ips: 296.14608 samples/s, eta: 2:59:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:52:00] ppocr INFO: epoch: [480/500], global_step: 491600, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.942464, exp_rate<=1 : 0.977201, exp_rate<=2 : 0.994055, exp_rate<=3 : 0.997711, loss: 0.001224, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18594 s, avg_samples: 56.0, ips: 301.17554 samples/s, eta: 2:58:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:52:18] ppocr INFO: epoch: [480/500], global_step: 491700, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.942548, exp_rate<=1 : 0.977339, exp_rate<=2 : 0.994202, exp_rate<=3 : 0.997794, loss: 0.001354, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18622 s, avg_samples: 56.0, ips: 300.72200 samples/s, eta: 2:58:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:52:37] ppocr INFO: epoch: [480/500], global_step: 491800, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.942344, exp_rate<=1 : 0.977300, exp_rate<=2 : 0.994149, exp_rate<=3 : 0.997774, loss: 0.001169, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18941 s, avg_samples: 56.0, ips: 295.65308 samples/s, eta: 2:58:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:52:58] ppocr INFO: epoch: [480/500], global_step: 491900, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.942186, exp_rate<=1 : 0.977248, exp_rate<=2 : 0.994129, exp_rate<=3 : 0.997752, loss: 0.001705, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21037 s, avg_samples: 56.0, ips: 266.20344 samples/s, eta: 2:57:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:53:17] ppocr INFO: epoch: [480/500], global_step: 492000, lr: 0.000100, edit distance: 0.001181, exp_rate: 0.942977, exp_rate<=1 : 0.977591, exp_rate<=2 : 0.994273, exp_rate<=3 : 0.997834, loss: 0.001090, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18098 s, avg_samples: 56.0, ips: 309.42270 samples/s, eta: 2:57:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:53:36] ppocr INFO: epoch: [480/500], global_step: 492100, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.943191, exp_rate<=1 : 0.977748, exp_rate<=2 : 0.994254, exp_rate<=3 : 0.997887, loss: 0.001180, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18986 s, avg_samples: 56.0, ips: 294.95914 samples/s, eta: 2:57:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:53:55] ppocr INFO: epoch: [480/500], global_step: 492200, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.943047, exp_rate<=1 : 0.977778, exp_rate<=2 : 0.994354, exp_rate<=3 : 0.997896, loss: 0.000994, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18947 s, avg_samples: 56.0, ips: 295.55525 samples/s, eta: 2:56:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:54:13] ppocr INFO: epoch: [480/500], global_step: 492300, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.942543, exp_rate<=1 : 0.977536, exp_rate<=2 : 0.994309, exp_rate<=3 : 0.997924, loss: 0.001710, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18916 s, avg_samples: 56.0, ips: 296.04152 samples/s, eta: 2:56:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:54:32] ppocr INFO: epoch: [480/500], global_step: 492400, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.942326, exp_rate<=1 : 0.977542, exp_rate<=2 : 0.994293, exp_rate<=3 : 0.997938, loss: 0.001234, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18843 s, avg_samples: 56.0, ips: 297.19562 samples/s, eta: 2:56:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:54:51] ppocr INFO: epoch: [480/500], global_step: 492500, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.942562, exp_rate<=1 : 0.977583, exp_rate<=2 : 0.994392, exp_rate<=3 : 0.997987, loss: 0.001691, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18487 s, avg_samples: 56.0, ips: 302.91372 samples/s, eta: 2:55:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:55:09] ppocr INFO: epoch: [480/500], global_step: 492600, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.942610, exp_rate<=1 : 0.977746, exp_rate<=2 : 0.994527, exp_rate<=3 : 0.998064, loss: 0.001463, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18479 s, avg_samples: 56.0, ips: 303.04132 samples/s, eta: 2:55:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:55:29] ppocr INFO: epoch: [480/500], global_step: 492700, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.942410, exp_rate<=1 : 0.977759, exp_rate<=2 : 0.994550, exp_rate<=3 : 0.998068, loss: 0.001667, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19628 s, avg_samples: 56.0, ips: 285.31081 samples/s, eta: 2:55:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:55:47] ppocr INFO: epoch: [480/500], global_step: 492800, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.942666, exp_rate<=1 : 0.977843, exp_rate<=2 : 0.994595, exp_rate<=3 : 0.998100, loss: 0.001214, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18102 s, avg_samples: 56.0, ips: 309.35468 samples/s, eta: 2:54:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:56:05] ppocr INFO: epoch: [480/500], global_step: 492900, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.942497, exp_rate<=1 : 0.977765, exp_rate<=2 : 0.994599, exp_rate<=3 : 0.998095, loss: 0.001368, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18122 s, avg_samples: 56.0, ips: 309.02497 samples/s, eta: 2:54:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:56:23] ppocr INFO: epoch: [480/500], global_step: 493000, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.942515, exp_rate<=1 : 0.977789, exp_rate<=2 : 0.994651, exp_rate<=3 : 0.998121, loss: 0.001563, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17891 s, avg_samples: 56.0, ips: 313.01452 samples/s, eta: 2:54:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:56:43] ppocr INFO: epoch: [480/500], global_step: 493100, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.942154, exp_rate<=1 : 0.977721, exp_rate<=2 : 0.994712, exp_rate<=3 : 0.998148, loss: 0.001174, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19397 s, avg_samples: 56.0, ips: 288.70793 samples/s, eta: 2:53:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:57:01] ppocr INFO: epoch: [480/500], global_step: 493200, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.942223, exp_rate<=1 : 0.977663, exp_rate<=2 : 0.994660, exp_rate<=3 : 0.998146, loss: 0.001479, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17891 s, avg_samples: 56.0, ips: 313.00432 samples/s, eta: 2:53:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:57:03] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 04:57:03] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_480 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 04:57:22] ppocr INFO: epoch: [481/500], global_step: 493300, lr: 0.000100, edit distance: 0.001037, exp_rate: 0.957170, exp_rate<=1 : 0.985071, exp_rate<=2 : 0.997407, exp_rate<=3 : 0.998421, loss: 0.000898, avg_reader_cost: 0.03455 s, avg_batch_cost: 0.21068 s, avg_samples: 56.0, ips: 265.80245 samples/s, eta: 2:53:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:57:40] ppocr INFO: epoch: [481/500], global_step: 493400, lr: 0.000100, edit distance: 0.001009, exp_rate: 0.958005, exp_rate<=1 : 0.985752, exp_rate<=2 : 0.997000, exp_rate<=3 : 0.999068, loss: 0.000884, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18455 s, avg_samples: 56.0, ips: 303.43276 samples/s, eta: 2:53:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:58:00] ppocr INFO: epoch: [481/500], global_step: 493500, lr: 0.000100, edit distance: 0.001002, exp_rate: 0.955902, exp_rate<=1 : 0.984352, exp_rate<=2 : 0.996630, exp_rate<=3 : 0.998916, loss: 0.001083, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19702 s, avg_samples: 56.0, ips: 284.23033 samples/s, eta: 2:52:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:58:19] ppocr INFO: epoch: [481/500], global_step: 493600, lr: 0.000100, edit distance: 0.001043, exp_rate: 0.954075, exp_rate<=1 : 0.983597, exp_rate<=2 : 0.996565, exp_rate<=3 : 0.998857, loss: 0.001084, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19356 s, avg_samples: 56.0, ips: 289.31655 samples/s, eta: 2:52:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:58:38] ppocr INFO: epoch: [481/500], global_step: 493700, lr: 0.000100, edit distance: 0.001057, exp_rate: 0.952868, exp_rate<=1 : 0.983028, exp_rate<=2 : 0.996349, exp_rate<=3 : 0.998653, loss: 0.001442, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18544 s, avg_samples: 56.0, ips: 301.98613 samples/s, eta: 2:52:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:58:58] ppocr INFO: epoch: [481/500], global_step: 493800, lr: 0.000100, edit distance: 0.001075, exp_rate: 0.950462, exp_rate<=1 : 0.981780, exp_rate<=2 : 0.996172, exp_rate<=3 : 0.998657, loss: 0.001119, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19850 s, avg_samples: 56.0, ips: 282.12073 samples/s, eta: 2:51:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:59:16] ppocr INFO: epoch: [481/500], global_step: 493900, lr: 0.000100, edit distance: 0.001098, exp_rate: 0.950388, exp_rate<=1 : 0.981897, exp_rate<=2 : 0.996145, exp_rate<=3 : 0.998655, loss: 0.001643, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18020 s, avg_samples: 56.0, ips: 310.76777 samples/s, eta: 2:51:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:59:32] ppocr INFO: epoch: [481/500], global_step: 494000, lr: 0.000100, edit distance: 0.001118, exp_rate: 0.950754, exp_rate<=1 : 0.982334, exp_rate<=2 : 0.996173, exp_rate<=3 : 0.998666, loss: 0.001453, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16465 s, avg_samples: 56.0, ips: 340.11566 samples/s, eta: 2:51:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 04:59:52] ppocr INFO: epoch: [481/500], global_step: 494100, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.949335, exp_rate<=1 : 0.981872, exp_rate<=2 : 0.996009, exp_rate<=3 : 0.998547, loss: 0.001710, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19614 s, avg_samples: 56.0, ips: 285.50417 samples/s, eta: 2:50:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:00:11] ppocr INFO: epoch: [481/500], global_step: 494200, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.948583, exp_rate<=1 : 0.981566, exp_rate<=2 : 0.995942, exp_rate<=3 : 0.998463, loss: 0.001046, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19195 s, avg_samples: 56.0, ips: 291.74881 samples/s, eta: 2:50:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:00:29] ppocr INFO: epoch: [481/500], global_step: 494300, lr: 0.000100, edit distance: 0.001126, exp_rate: 0.949049, exp_rate<=1 : 0.981725, exp_rate<=2 : 0.996050, exp_rate<=3 : 0.998521, loss: 0.001641, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17680 s, avg_samples: 56.0, ips: 316.74293 samples/s, eta: 2:50:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:00:47] ppocr INFO: epoch: [481/500], global_step: 494400, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.948274, exp_rate<=1 : 0.981348, exp_rate<=2 : 0.996085, exp_rate<=3 : 0.998560, loss: 0.001192, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18257 s, avg_samples: 56.0, ips: 306.73602 samples/s, eta: 2:49:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:01:05] ppocr INFO: epoch: [481/500], global_step: 494500, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.948032, exp_rate<=1 : 0.981203, exp_rate<=2 : 0.996034, exp_rate<=3 : 0.998475, loss: 0.001355, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18410 s, avg_samples: 56.0, ips: 304.17993 samples/s, eta: 2:49:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:01:23] ppocr INFO: epoch: [481/500], global_step: 494600, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.948143, exp_rate<=1 : 0.981364, exp_rate<=2 : 0.996094, exp_rate<=3 : 0.998527, loss: 0.001175, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17146 s, avg_samples: 56.0, ips: 326.59897 samples/s, eta: 2:49:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:01:41] ppocr INFO: epoch: [481/500], global_step: 494700, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.948146, exp_rate<=1 : 0.981424, exp_rate<=2 : 0.996107, exp_rate<=3 : 0.998578, loss: 0.001375, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18433 s, avg_samples: 56.0, ips: 303.80803 samples/s, eta: 2:48:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:02:00] ppocr INFO: epoch: [481/500], global_step: 494800, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.947922, exp_rate<=1 : 0.981094, exp_rate<=2 : 0.996067, exp_rate<=3 : 0.998563, loss: 0.001554, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18629 s, avg_samples: 56.0, ips: 300.61176 samples/s, eta: 2:48:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:02:19] ppocr INFO: epoch: [481/500], global_step: 494900, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.947650, exp_rate<=1 : 0.980788, exp_rate<=2 : 0.995991, exp_rate<=3 : 0.998545, loss: 0.001217, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19003 s, avg_samples: 56.0, ips: 294.68350 samples/s, eta: 2:48:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:02:37] ppocr INFO: epoch: [481/500], global_step: 495000, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.946847, exp_rate<=1 : 0.980428, exp_rate<=2 : 0.995895, exp_rate<=3 : 0.998586, loss: 0.001574, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17824 s, avg_samples: 56.0, ips: 314.18310 samples/s, eta: 2:47:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:02:57] ppocr INFO: epoch: [481/500], global_step: 495100, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.946240, exp_rate<=1 : 0.980241, exp_rate<=2 : 0.995781, exp_rate<=3 : 0.998539, loss: 0.001413, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20204 s, avg_samples: 56.0, ips: 277.17178 samples/s, eta: 2:47:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:03:18] ppocr INFO: epoch: [481/500], global_step: 495200, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.945540, exp_rate<=1 : 0.979853, exp_rate<=2 : 0.995667, exp_rate<=3 : 0.998531, loss: 0.001192, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.21535 s, avg_samples: 56.0, ips: 260.03724 samples/s, eta: 2:47:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:03:38] ppocr INFO: epoch: [481/500], global_step: 495300, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.945050, exp_rate<=1 : 0.979560, exp_rate<=2 : 0.995599, exp_rate<=3 : 0.998508, loss: 0.001723, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19208 s, avg_samples: 56.0, ips: 291.54249 samples/s, eta: 2:46:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:03:56] ppocr INFO: epoch: [481/500], global_step: 495400, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.944844, exp_rate<=1 : 0.979420, exp_rate<=2 : 0.995572, exp_rate<=3 : 0.998519, loss: 0.001252, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18265 s, avg_samples: 56.0, ips: 306.60117 samples/s, eta: 2:46:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:04:15] ppocr INFO: epoch: [481/500], global_step: 495500, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.944515, exp_rate<=1 : 0.979215, exp_rate<=2 : 0.995529, exp_rate<=3 : 0.998528, loss: 0.001701, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18773 s, avg_samples: 56.0, ips: 298.29910 samples/s, eta: 2:46:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:04:35] ppocr INFO: epoch: [481/500], global_step: 495600, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.944047, exp_rate<=1 : 0.978944, exp_rate<=2 : 0.995458, exp_rate<=3 : 0.998500, loss: 0.001459, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20073 s, avg_samples: 56.0, ips: 278.98578 samples/s, eta: 2:46:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:04:54] ppocr INFO: epoch: [481/500], global_step: 495700, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.943811, exp_rate<=1 : 0.978742, exp_rate<=2 : 0.995421, exp_rate<=3 : 0.998471, loss: 0.001184, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19556 s, avg_samples: 56.0, ips: 286.35488 samples/s, eta: 2:45:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:05:13] ppocr INFO: epoch: [481/500], global_step: 495800, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.943747, exp_rate<=1 : 0.978686, exp_rate<=2 : 0.995386, exp_rate<=3 : 0.998455, loss: 0.001582, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18390 s, avg_samples: 56.0, ips: 304.51015 samples/s, eta: 2:45:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:05:32] ppocr INFO: epoch: [481/500], global_step: 495900, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.943492, exp_rate<=1 : 0.978637, exp_rate<=2 : 0.995361, exp_rate<=3 : 0.998414, loss: 0.001289, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19004 s, avg_samples: 56.0, ips: 294.67526 samples/s, eta: 2:45:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:05:40] ppocr INFO: epoch: [481/500], global_step: 495940, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.943442, exp_rate<=1 : 0.978511, exp_rate<=2 : 0.995325, exp_rate<=3 : 0.998408, loss: 0.001201, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.08002 s, avg_samples: 22.4, ips: 279.94102 samples/s, eta: 2:44:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:05:42] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 05:05:53] ppocr INFO: epoch: [482/500], global_step: 496000, lr: 0.000100, edit distance: 0.000981, exp_rate: 0.949783, exp_rate<=1 : 0.982781, exp_rate<=2 : 0.996398, exp_rate<=3 : 0.999084, loss: 0.001316, avg_reader_cost: 0.02534 s, avg_batch_cost: 0.13545 s, avg_samples: 33.6, ips: 248.07077 samples/s, eta: 2:44:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:06:13] ppocr INFO: epoch: [482/500], global_step: 496100, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.944051, exp_rate<=1 : 0.977533, exp_rate<=2 : 0.995250, exp_rate<=3 : 0.998458, loss: 0.001630, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19763 s, avg_samples: 56.0, ips: 283.35751 samples/s, eta: 2:44:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:06:32] ppocr INFO: epoch: [482/500], global_step: 496200, lr: 0.000100, edit distance: 0.001084, exp_rate: 0.945894, exp_rate<=1 : 0.979019, exp_rate<=2 : 0.995081, exp_rate<=3 : 0.998303, loss: 0.000982, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19183 s, avg_samples: 56.0, ips: 291.92492 samples/s, eta: 2:44:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:06:51] ppocr INFO: epoch: [482/500], global_step: 496300, lr: 0.000100, edit distance: 0.001040, exp_rate: 0.948339, exp_rate<=1 : 0.980316, exp_rate<=2 : 0.995384, exp_rate<=3 : 0.998605, loss: 0.001352, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18610 s, avg_samples: 56.0, ips: 300.91917 samples/s, eta: 2:43:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:07:10] ppocr INFO: epoch: [482/500], global_step: 496400, lr: 0.000100, edit distance: 0.001031, exp_rate: 0.949000, exp_rate<=1 : 0.980827, exp_rate<=2 : 0.995595, exp_rate<=3 : 0.998835, loss: 0.000817, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18892 s, avg_samples: 56.0, ips: 296.42464 samples/s, eta: 2:43:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:07:27] ppocr INFO: epoch: [482/500], global_step: 496500, lr: 0.000100, edit distance: 0.001005, exp_rate: 0.950286, exp_rate<=1 : 0.981563, exp_rate<=2 : 0.995706, exp_rate<=3 : 0.998846, loss: 0.000757, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17320 s, avg_samples: 56.0, ips: 323.32683 samples/s, eta: 2:43:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:07:46] ppocr INFO: epoch: [482/500], global_step: 496600, lr: 0.000100, edit distance: 0.001026, exp_rate: 0.950324, exp_rate<=1 : 0.981163, exp_rate<=2 : 0.995587, exp_rate<=3 : 0.998802, loss: 0.001580, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18822 s, avg_samples: 56.0, ips: 297.52597 samples/s, eta: 2:42:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:08:05] ppocr INFO: epoch: [482/500], global_step: 496700, lr: 0.000100, edit distance: 0.001065, exp_rate: 0.949245, exp_rate<=1 : 0.981286, exp_rate<=2 : 0.995601, exp_rate<=3 : 0.998754, loss: 0.001326, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18516 s, avg_samples: 56.0, ips: 302.44806 samples/s, eta: 2:42:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:08:23] ppocr INFO: epoch: [482/500], global_step: 496800, lr: 0.000100, edit distance: 0.001057, exp_rate: 0.949373, exp_rate<=1 : 0.981241, exp_rate<=2 : 0.995549, exp_rate<=3 : 0.998800, loss: 0.001690, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18678 s, avg_samples: 56.0, ips: 299.81809 samples/s, eta: 2:42:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:08:42] ppocr INFO: epoch: [482/500], global_step: 496900, lr: 0.000100, edit distance: 0.001069, exp_rate: 0.948915, exp_rate<=1 : 0.981000, exp_rate<=2 : 0.995684, exp_rate<=3 : 0.998820, loss: 0.001312, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18850 s, avg_samples: 56.0, ips: 297.08146 samples/s, eta: 2:41:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:09:01] ppocr INFO: epoch: [482/500], global_step: 497000, lr: 0.000100, edit distance: 0.001081, exp_rate: 0.948403, exp_rate<=1 : 0.980474, exp_rate<=2 : 0.995621, exp_rate<=3 : 0.998770, loss: 0.001040, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18957 s, avg_samples: 56.0, ips: 295.40360 samples/s, eta: 2:41:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:09:20] ppocr INFO: epoch: [482/500], global_step: 497100, lr: 0.000100, edit distance: 0.001095, exp_rate: 0.948120, exp_rate<=1 : 0.980161, exp_rate<=2 : 0.995615, exp_rate<=3 : 0.998789, loss: 0.001346, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18479 s, avg_samples: 56.0, ips: 303.03871 samples/s, eta: 2:41:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:09:40] ppocr INFO: epoch: [482/500], global_step: 497200, lr: 0.000100, edit distance: 0.001101, exp_rate: 0.947164, exp_rate<=1 : 0.979736, exp_rate<=2 : 0.995435, exp_rate<=3 : 0.998686, loss: 0.000986, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20111 s, avg_samples: 56.0, ips: 278.45372 samples/s, eta: 2:40:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:10:00] ppocr INFO: epoch: [482/500], global_step: 497300, lr: 0.000100, edit distance: 0.001114, exp_rate: 0.946396, exp_rate<=1 : 0.979406, exp_rate<=2 : 0.995366, exp_rate<=3 : 0.998587, loss: 0.001540, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19953 s, avg_samples: 56.0, ips: 280.66153 samples/s, eta: 2:40:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:10:18] ppocr INFO: epoch: [482/500], global_step: 497400, lr: 0.000100, edit distance: 0.001109, exp_rate: 0.946700, exp_rate<=1 : 0.979691, exp_rate<=2 : 0.995450, exp_rate<=3 : 0.998562, loss: 0.001514, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18231 s, avg_samples: 56.0, ips: 307.17576 samples/s, eta: 2:40:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:10:38] ppocr INFO: epoch: [482/500], global_step: 497500, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.945928, exp_rate<=1 : 0.979435, exp_rate<=2 : 0.995285, exp_rate<=3 : 0.998448, loss: 0.001734, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19995 s, avg_samples: 56.0, ips: 280.06650 samples/s, eta: 2:40:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:10:57] ppocr INFO: epoch: [482/500], global_step: 497600, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.945516, exp_rate<=1 : 0.979271, exp_rate<=2 : 0.995363, exp_rate<=3 : 0.998472, loss: 0.001499, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19180 s, avg_samples: 56.0, ips: 291.97142 samples/s, eta: 2:39:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:11:16] ppocr INFO: epoch: [482/500], global_step: 497700, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.944984, exp_rate<=1 : 0.979169, exp_rate<=2 : 0.995398, exp_rate<=3 : 0.998464, loss: 0.001726, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19245 s, avg_samples: 56.0, ips: 290.98214 samples/s, eta: 2:39:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:11:34] ppocr INFO: epoch: [482/500], global_step: 497800, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.945462, exp_rate<=1 : 0.979515, exp_rate<=2 : 0.995488, exp_rate<=3 : 0.998493, loss: 0.001473, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17786 s, avg_samples: 56.0, ips: 314.84921 samples/s, eta: 2:39:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:11:53] ppocr INFO: epoch: [482/500], global_step: 497900, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.944928, exp_rate<=1 : 0.979313, exp_rate<=2 : 0.995356, exp_rate<=3 : 0.998504, loss: 0.001275, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18935 s, avg_samples: 56.0, ips: 295.75440 samples/s, eta: 2:38:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:12:11] ppocr INFO: epoch: [482/500], global_step: 498000, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.945058, exp_rate<=1 : 0.979486, exp_rate<=2 : 0.995444, exp_rate<=3 : 0.998541, loss: 0.001767, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17601 s, avg_samples: 56.0, ips: 318.16555 samples/s, eta: 2:38:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:12:30] ppocr INFO: epoch: [482/500], global_step: 498100, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.944392, exp_rate<=1 : 0.979185, exp_rate<=2 : 0.995393, exp_rate<=3 : 0.998502, loss: 0.000830, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19207 s, avg_samples: 56.0, ips: 291.56212 samples/s, eta: 2:38:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:12:48] ppocr INFO: epoch: [482/500], global_step: 498200, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.943928, exp_rate<=1 : 0.978935, exp_rate<=2 : 0.995290, exp_rate<=3 : 0.998467, loss: 0.001860, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18355 s, avg_samples: 56.0, ips: 305.09361 samples/s, eta: 2:37:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:13:07] ppocr INFO: epoch: [482/500], global_step: 498300, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.943726, exp_rate<=1 : 0.978846, exp_rate<=2 : 0.995274, exp_rate<=3 : 0.998468, loss: 0.001072, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18515 s, avg_samples: 56.0, ips: 302.45108 samples/s, eta: 2:37:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:13:25] ppocr INFO: epoch: [482/500], global_step: 498400, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.943988, exp_rate<=1 : 0.978952, exp_rate<=2 : 0.995269, exp_rate<=3 : 0.998453, loss: 0.001435, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18288 s, avg_samples: 56.0, ips: 306.21359 samples/s, eta: 2:37:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:13:45] ppocr INFO: epoch: [482/500], global_step: 498500, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.943607, exp_rate<=1 : 0.978717, exp_rate<=2 : 0.995161, exp_rate<=3 : 0.998346, loss: 0.001163, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.19274 s, avg_samples: 56.0, ips: 290.53966 samples/s, eta: 2:36:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:14:03] ppocr INFO: epoch: [482/500], global_step: 498600, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.943113, exp_rate<=1 : 0.978648, exp_rate<=2 : 0.995138, exp_rate<=3 : 0.998336, loss: 0.001359, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18647 s, avg_samples: 56.0, ips: 300.31711 samples/s, eta: 2:36:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:14:18] ppocr INFO: epoch: [482/500], global_step: 498680, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.943235, exp_rate<=1 : 0.978684, exp_rate<=2 : 0.995168, exp_rate<=3 : 0.998344, loss: 0.001313, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15229 s, avg_samples: 44.8, ips: 294.16922 samples/s, eta: 2:36:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:14:21] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 05:14:25] ppocr INFO: epoch: [483/500], global_step: 498700, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.940476, exp_rate<=1 : 0.978237, exp_rate<=2 : 0.998016, exp_rate<=3 : 0.998016, loss: 0.001614, avg_reader_cost: 0.02527 s, avg_batch_cost: 0.06561 s, avg_samples: 11.2, ips: 170.70703 samples/s, eta: 2:36:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:14:44] ppocr INFO: epoch: [483/500], global_step: 498800, lr: 0.000100, edit distance: 0.001131, exp_rate: 0.947464, exp_rate<=1 : 0.981022, exp_rate<=2 : 0.997326, exp_rate<=3 : 0.998950, loss: 0.001119, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18516 s, avg_samples: 56.0, ips: 302.43821 samples/s, eta: 2:35:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:15:02] ppocr INFO: epoch: [483/500], global_step: 498900, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.945586, exp_rate<=1 : 0.980074, exp_rate<=2 : 0.996195, exp_rate<=3 : 0.998897, loss: 0.001121, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18719 s, avg_samples: 56.0, ips: 299.16113 samples/s, eta: 2:35:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:15:21] ppocr INFO: epoch: [483/500], global_step: 499000, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.946925, exp_rate<=1 : 0.980403, exp_rate<=2 : 0.996376, exp_rate<=3 : 0.998958, loss: 0.001346, avg_reader_cost: 0.00019 s, avg_batch_cost: 0.18455 s, avg_samples: 56.0, ips: 303.43473 samples/s, eta: 2:35:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:15:38] ppocr INFO: epoch: [483/500], global_step: 499100, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.948693, exp_rate<=1 : 0.981158, exp_rate<=2 : 0.996320, exp_rate<=3 : 0.998976, loss: 0.001278, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17265 s, avg_samples: 56.0, ips: 324.36078 samples/s, eta: 2:34:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:15:56] ppocr INFO: epoch: [483/500], global_step: 499200, lr: 0.000100, edit distance: 0.001145, exp_rate: 0.950264, exp_rate<=1 : 0.981794, exp_rate<=2 : 0.996408, exp_rate<=3 : 0.998896, loss: 0.001396, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17570 s, avg_samples: 56.0, ips: 318.72513 samples/s, eta: 2:34:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:16:15] ppocr INFO: epoch: [483/500], global_step: 499300, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.948703, exp_rate<=1 : 0.980882, exp_rate<=2 : 0.995782, exp_rate<=3 : 0.998523, loss: 0.001623, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18895 s, avg_samples: 56.0, ips: 296.37600 samples/s, eta: 2:34:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:16:32] ppocr INFO: epoch: [483/500], global_step: 499400, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.947889, exp_rate<=1 : 0.980593, exp_rate<=2 : 0.995898, exp_rate<=3 : 0.998593, loss: 0.001594, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17846 s, avg_samples: 56.0, ips: 313.78772 samples/s, eta: 2:33:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:16:51] ppocr INFO: epoch: [483/500], global_step: 499500, lr: 0.000100, edit distance: 0.001183, exp_rate: 0.947469, exp_rate<=1 : 0.980586, exp_rate<=2 : 0.995951, exp_rate<=3 : 0.998599, loss: 0.001466, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18658 s, avg_samples: 56.0, ips: 300.14096 samples/s, eta: 2:33:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:17:10] ppocr INFO: epoch: [483/500], global_step: 499600, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.947907, exp_rate<=1 : 0.980723, exp_rate<=2 : 0.996069, exp_rate<=3 : 0.998620, loss: 0.001040, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18557 s, avg_samples: 56.0, ips: 301.77777 samples/s, eta: 2:33:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:17:29] ppocr INFO: epoch: [483/500], global_step: 499700, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.947484, exp_rate<=1 : 0.980507, exp_rate<=2 : 0.996050, exp_rate<=3 : 0.998581, loss: 0.001476, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19103 s, avg_samples: 56.0, ips: 293.14440 samples/s, eta: 2:33:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:17:48] ppocr INFO: epoch: [483/500], global_step: 499800, lr: 0.000100, edit distance: 0.001165, exp_rate: 0.946663, exp_rate<=1 : 0.980151, exp_rate<=2 : 0.995895, exp_rate<=3 : 0.998495, loss: 0.001658, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18742 s, avg_samples: 56.0, ips: 298.80151 samples/s, eta: 2:32:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:18:07] ppocr INFO: epoch: [483/500], global_step: 499900, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.946480, exp_rate<=1 : 0.980003, exp_rate<=2 : 0.995970, exp_rate<=3 : 0.998491, loss: 0.001572, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19490 s, avg_samples: 56.0, ips: 287.32116 samples/s, eta: 2:32:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:18:26] ppocr INFO: epoch: [483/500], global_step: 500000, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.946449, exp_rate<=1 : 0.979949, exp_rate<=2 : 0.995964, exp_rate<=3 : 0.998522, loss: 0.001585, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.74124 samples/s, eta: 2:32:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:18:47] ppocr INFO: epoch: [483/500], global_step: 500100, lr: 0.000100, edit distance: 0.001157, exp_rate: 0.945933, exp_rate<=1 : 0.979622, exp_rate<=2 : 0.995882, exp_rate<=3 : 0.998516, loss: 0.001161, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20259 s, avg_samples: 56.0, ips: 276.41897 samples/s, eta: 2:31:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:19:05] ppocr INFO: epoch: [483/500], global_step: 500200, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.946558, exp_rate<=1 : 0.979979, exp_rate<=2 : 0.995921, exp_rate<=3 : 0.998570, loss: 0.000814, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18565 s, avg_samples: 56.0, ips: 301.64956 samples/s, eta: 2:31:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:19:25] ppocr INFO: epoch: [483/500], global_step: 500300, lr: 0.000100, edit distance: 0.001144, exp_rate: 0.946495, exp_rate<=1 : 0.980027, exp_rate<=2 : 0.995965, exp_rate<=3 : 0.998606, loss: 0.001342, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19762 s, avg_samples: 56.0, ips: 283.36878 samples/s, eta: 2:31:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:19:44] ppocr INFO: epoch: [483/500], global_step: 500400, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.946575, exp_rate<=1 : 0.980003, exp_rate<=2 : 0.995900, exp_rate<=3 : 0.998631, loss: 0.001193, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19022 s, avg_samples: 56.0, ips: 294.38928 samples/s, eta: 2:30:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:20:03] ppocr INFO: epoch: [483/500], global_step: 500500, lr: 0.000100, edit distance: 0.001167, exp_rate: 0.946182, exp_rate<=1 : 0.979869, exp_rate<=2 : 0.995767, exp_rate<=3 : 0.998556, loss: 0.001477, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19148 s, avg_samples: 56.0, ips: 292.45454 samples/s, eta: 2:30:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:20:23] ppocr INFO: epoch: [483/500], global_step: 500600, lr: 0.000100, edit distance: 0.001175, exp_rate: 0.945339, exp_rate<=1 : 0.979363, exp_rate<=2 : 0.995704, exp_rate<=3 : 0.998522, loss: 0.001519, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19608 s, avg_samples: 56.0, ips: 285.60353 samples/s, eta: 2:30:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:20:41] ppocr INFO: epoch: [483/500], global_step: 500700, lr: 0.000100, edit distance: 0.001178, exp_rate: 0.945230, exp_rate<=1 : 0.979283, exp_rate<=2 : 0.995710, exp_rate<=3 : 0.998517, loss: 0.001309, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18310 s, avg_samples: 56.0, ips: 305.84168 samples/s, eta: 2:29:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:21:00] ppocr INFO: epoch: [483/500], global_step: 500800, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.944878, exp_rate<=1 : 0.979220, exp_rate<=2 : 0.995643, exp_rate<=3 : 0.998512, loss: 0.001291, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18580 s, avg_samples: 56.0, ips: 301.40436 samples/s, eta: 2:29:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:21:19] ppocr INFO: epoch: [483/500], global_step: 500900, lr: 0.000100, edit distance: 0.001183, exp_rate: 0.944664, exp_rate<=1 : 0.979122, exp_rate<=2 : 0.995667, exp_rate<=3 : 0.998506, loss: 0.001988, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19200 s, avg_samples: 56.0, ips: 291.66667 samples/s, eta: 2:29:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:21:37] ppocr INFO: epoch: [483/500], global_step: 501000, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.944281, exp_rate<=1 : 0.979060, exp_rate<=2 : 0.995659, exp_rate<=3 : 0.998496, loss: 0.001854, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18075 s, avg_samples: 56.0, ips: 309.81249 samples/s, eta: 2:28:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:21:58] ppocr INFO: epoch: [483/500], global_step: 501100, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.944076, exp_rate<=1 : 0.978961, exp_rate<=2 : 0.995563, exp_rate<=3 : 0.998405, loss: 0.001402, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.21068 s, avg_samples: 56.0, ips: 265.80302 samples/s, eta: 2:28:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:22:17] ppocr INFO: epoch: [483/500], global_step: 501200, lr: 0.000100, edit distance: 0.001189, exp_rate: 0.944087, exp_rate<=1 : 0.978982, exp_rate<=2 : 0.995532, exp_rate<=3 : 0.998400, loss: 0.000799, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18944 s, avg_samples: 56.0, ips: 295.60576 samples/s, eta: 2:28:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:22:36] ppocr INFO: epoch: [483/500], global_step: 501300, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.944355, exp_rate<=1 : 0.979181, exp_rate<=2 : 0.995576, exp_rate<=3 : 0.998422, loss: 0.000936, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18745 s, avg_samples: 56.0, ips: 298.74377 samples/s, eta: 2:27:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:22:55] ppocr INFO: epoch: [483/500], global_step: 501400, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.944332, exp_rate<=1 : 0.979113, exp_rate<=2 : 0.995555, exp_rate<=3 : 0.998421, loss: 0.001478, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19035 s, avg_samples: 56.0, ips: 294.19027 samples/s, eta: 2:27:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:22:58] ppocr INFO: epoch: [483/500], global_step: 501420, lr: 0.000100, edit distance: 0.001175, exp_rate: 0.944288, exp_rate<=1 : 0.979038, exp_rate<=2 : 0.995524, exp_rate<=3 : 0.998421, loss: 0.001290, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03473 s, avg_samples: 11.2, ips: 322.44318 samples/s, eta: 2:27:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:23:01] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 05:23:17] ppocr INFO: epoch: [484/500], global_step: 501500, lr: 0.000100, edit distance: 0.000989, exp_rate: 0.952247, exp_rate<=1 : 0.977837, exp_rate<=2 : 0.994681, exp_rate<=3 : 0.998165, loss: 0.001328, avg_reader_cost: 0.03108 s, avg_batch_cost: 0.18661 s, avg_samples: 44.8, ips: 240.07784 samples/s, eta: 2:27:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:23:37] ppocr INFO: epoch: [484/500], global_step: 501600, lr: 0.000100, edit distance: 0.001031, exp_rate: 0.949593, exp_rate<=1 : 0.979079, exp_rate<=2 : 0.995207, exp_rate<=3 : 0.998112, loss: 0.000852, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20023 s, avg_samples: 56.0, ips: 279.67509 samples/s, eta: 2:27:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:23:56] ppocr INFO: epoch: [484/500], global_step: 501700, lr: 0.000100, edit distance: 0.000949, exp_rate: 0.953488, exp_rate<=1 : 0.981582, exp_rate<=2 : 0.995828, exp_rate<=3 : 0.998523, loss: 0.001171, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19110 s, avg_samples: 56.0, ips: 293.04596 samples/s, eta: 2:26:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:24:14] ppocr INFO: epoch: [484/500], global_step: 501800, lr: 0.000100, edit distance: 0.000968, exp_rate: 0.955010, exp_rate<=1 : 0.982508, exp_rate<=2 : 0.995997, exp_rate<=3 : 0.998598, loss: 0.001413, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17991 s, avg_samples: 56.0, ips: 311.26043 samples/s, eta: 2:26:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:24:33] ppocr INFO: epoch: [484/500], global_step: 501900, lr: 0.000100, edit distance: 0.000986, exp_rate: 0.953411, exp_rate<=1 : 0.981685, exp_rate<=2 : 0.995892, exp_rate<=3 : 0.998460, loss: 0.001431, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18791 s, avg_samples: 56.0, ips: 298.01167 samples/s, eta: 2:26:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:24:51] ppocr INFO: epoch: [484/500], global_step: 502000, lr: 0.000100, edit distance: 0.000980, exp_rate: 0.953084, exp_rate<=1 : 0.982455, exp_rate<=2 : 0.996032, exp_rate<=3 : 0.998498, loss: 0.000910, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17485 s, avg_samples: 56.0, ips: 320.26704 samples/s, eta: 2:25:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:25:09] ppocr INFO: epoch: [484/500], global_step: 502100, lr: 0.000100, edit distance: 0.000975, exp_rate: 0.953114, exp_rate<=1 : 0.982423, exp_rate<=2 : 0.996220, exp_rate<=3 : 0.998554, loss: 0.001082, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18622 s, avg_samples: 56.0, ips: 300.71883 samples/s, eta: 2:25:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:25:28] ppocr INFO: epoch: [484/500], global_step: 502200, lr: 0.000100, edit distance: 0.000977, exp_rate: 0.952093, exp_rate<=1 : 0.982166, exp_rate<=2 : 0.996131, exp_rate<=3 : 0.998586, loss: 0.000647, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18728 s, avg_samples: 56.0, ips: 299.02055 samples/s, eta: 2:25:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:25:47] ppocr INFO: epoch: [484/500], global_step: 502300, lr: 0.000100, edit distance: 0.000969, exp_rate: 0.952128, exp_rate<=1 : 0.982451, exp_rate<=2 : 0.996255, exp_rate<=3 : 0.998667, loss: 0.001332, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18561 s, avg_samples: 56.0, ips: 301.70610 samples/s, eta: 2:24:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:26:06] ppocr INFO: epoch: [484/500], global_step: 502400, lr: 0.000100, edit distance: 0.000975, exp_rate: 0.951489, exp_rate<=1 : 0.981959, exp_rate<=2 : 0.996061, exp_rate<=3 : 0.998531, loss: 0.002157, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19763 s, avg_samples: 56.0, ips: 283.35333 samples/s, eta: 2:24:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:26:24] ppocr INFO: epoch: [484/500], global_step: 502500, lr: 0.000100, edit distance: 0.000992, exp_rate: 0.950623, exp_rate<=1 : 0.981425, exp_rate<=2 : 0.995835, exp_rate<=3 : 0.998476, loss: 0.002004, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17228 s, avg_samples: 56.0, ips: 325.04541 samples/s, eta: 2:24:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:26:42] ppocr INFO: epoch: [484/500], global_step: 502600, lr: 0.000100, edit distance: 0.001029, exp_rate: 0.949906, exp_rate<=1 : 0.981258, exp_rate<=2 : 0.995797, exp_rate<=3 : 0.998473, loss: 0.001263, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18300 s, avg_samples: 56.0, ips: 306.01329 samples/s, eta: 2:23:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:27:00] ppocr INFO: epoch: [484/500], global_step: 502700, lr: 0.000100, edit distance: 0.001049, exp_rate: 0.949365, exp_rate<=1 : 0.980782, exp_rate<=2 : 0.995679, exp_rate<=3 : 0.998461, loss: 0.000778, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18368 s, avg_samples: 56.0, ips: 304.88069 samples/s, eta: 2:23:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:27:18] ppocr INFO: epoch: [484/500], global_step: 502800, lr: 0.000100, edit distance: 0.001067, exp_rate: 0.949251, exp_rate<=1 : 0.980806, exp_rate<=2 : 0.995673, exp_rate<=3 : 0.998502, loss: 0.001312, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17165 s, avg_samples: 56.0, ips: 326.25459 samples/s, eta: 2:23:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:27:36] ppocr INFO: epoch: [484/500], global_step: 502900, lr: 0.000100, edit distance: 0.001087, exp_rate: 0.948253, exp_rate<=1 : 0.980524, exp_rate<=2 : 0.995586, exp_rate<=3 : 0.998459, loss: 0.001430, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18392 s, avg_samples: 56.0, ips: 304.47762 samples/s, eta: 2:22:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:27:54] ppocr INFO: epoch: [484/500], global_step: 503000, lr: 0.000100, edit distance: 0.001096, exp_rate: 0.947776, exp_rate<=1 : 0.980444, exp_rate<=2 : 0.995594, exp_rate<=3 : 0.998492, loss: 0.001598, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18151 s, avg_samples: 56.0, ips: 308.52741 samples/s, eta: 2:22:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:28:15] ppocr INFO: epoch: [484/500], global_step: 503100, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.946194, exp_rate<=1 : 0.979744, exp_rate<=2 : 0.995423, exp_rate<=3 : 0.998399, loss: 0.001590, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20859 s, avg_samples: 56.0, ips: 268.46316 samples/s, eta: 2:22:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:28:34] ppocr INFO: epoch: [484/500], global_step: 503200, lr: 0.000100, edit distance: 0.001127, exp_rate: 0.946121, exp_rate<=1 : 0.979760, exp_rate<=2 : 0.995432, exp_rate<=3 : 0.998412, loss: 0.001263, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18821 s, avg_samples: 56.0, ips: 297.54062 samples/s, eta: 2:21:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:28:53] ppocr INFO: epoch: [484/500], global_step: 503300, lr: 0.000100, edit distance: 0.001133, exp_rate: 0.945737, exp_rate<=1 : 0.979726, exp_rate<=2 : 0.995458, exp_rate<=3 : 0.998434, loss: 0.000823, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19181 s, avg_samples: 56.0, ips: 291.95584 samples/s, eta: 2:21:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:29:13] ppocr INFO: epoch: [484/500], global_step: 503400, lr: 0.000100, edit distance: 0.001151, exp_rate: 0.945101, exp_rate<=1 : 0.979508, exp_rate<=2 : 0.995412, exp_rate<=3 : 0.998423, loss: 0.001488, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19464 s, avg_samples: 56.0, ips: 287.70422 samples/s, eta: 2:21:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:29:29] ppocr INFO: epoch: [484/500], global_step: 503500, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.945314, exp_rate<=1 : 0.979528, exp_rate<=2 : 0.995410, exp_rate<=3 : 0.998445, loss: 0.001652, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16576 s, avg_samples: 56.0, ips: 337.83421 samples/s, eta: 2:20:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:29:49] ppocr INFO: epoch: [484/500], global_step: 503600, lr: 0.000100, edit distance: 0.001156, exp_rate: 0.944680, exp_rate<=1 : 0.979280, exp_rate<=2 : 0.995425, exp_rate<=3 : 0.998468, loss: 0.001442, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20234 s, avg_samples: 56.0, ips: 276.75647 samples/s, eta: 2:20:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:30:08] ppocr INFO: epoch: [484/500], global_step: 503700, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.944421, exp_rate<=1 : 0.979156, exp_rate<=2 : 0.995419, exp_rate<=3 : 0.998468, loss: 0.001311, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18873 s, avg_samples: 56.0, ips: 296.71331 samples/s, eta: 2:20:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:30:26] ppocr INFO: epoch: [484/500], global_step: 503800, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.944729, exp_rate<=1 : 0.979283, exp_rate<=2 : 0.995467, exp_rate<=3 : 0.998510, loss: 0.001230, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17383 s, avg_samples: 56.0, ips: 322.15052 samples/s, eta: 2:20:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:30:45] ppocr INFO: epoch: [484/500], global_step: 503900, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.944875, exp_rate<=1 : 0.979335, exp_rate<=2 : 0.995454, exp_rate<=3 : 0.998533, loss: 0.001689, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18961 s, avg_samples: 56.0, ips: 295.33943 samples/s, eta: 2:19:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:31:05] ppocr INFO: epoch: [484/500], global_step: 504000, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.944597, exp_rate<=1 : 0.979166, exp_rate<=2 : 0.995431, exp_rate<=3 : 0.998499, loss: 0.001118, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20142 s, avg_samples: 56.0, ips: 278.02607 samples/s, eta: 2:19:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:31:25] ppocr INFO: epoch: [484/500], global_step: 504100, lr: 0.000100, edit distance: 0.001167, exp_rate: 0.944296, exp_rate<=1 : 0.979076, exp_rate<=2 : 0.995427, exp_rate<=3 : 0.998489, loss: 0.001295, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19938 s, avg_samples: 56.0, ips: 280.87330 samples/s, eta: 2:19:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:31:36] ppocr INFO: epoch: [484/500], global_step: 504160, lr: 0.000100, edit distance: 0.001167, exp_rate: 0.944201, exp_rate<=1 : 0.979000, exp_rate<=2 : 0.995468, exp_rate<=3 : 0.998512, loss: 0.001520, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.11638 s, avg_samples: 33.6, ips: 288.71669 samples/s, eta: 2:18:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:31:38] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 05:31:46] ppocr INFO: epoch: [485/500], global_step: 504200, lr: 0.000100, edit distance: 0.001018, exp_rate: 0.966602, exp_rate<=1 : 0.985124, exp_rate<=2 : 0.997231, exp_rate<=3 : 1.000000, loss: 0.001077, avg_reader_cost: 0.02287 s, avg_batch_cost: 0.09490 s, avg_samples: 22.4, ips: 236.02737 samples/s, eta: 2:18:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:32:05] ppocr INFO: epoch: [485/500], global_step: 504300, lr: 0.000100, edit distance: 0.001009, exp_rate: 0.956556, exp_rate<=1 : 0.982493, exp_rate<=2 : 0.996442, exp_rate<=3 : 0.998495, loss: 0.001361, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19382 s, avg_samples: 56.0, ips: 288.92321 samples/s, eta: 2:18:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:32:25] ppocr INFO: epoch: [485/500], global_step: 504400, lr: 0.000100, edit distance: 0.001026, exp_rate: 0.952239, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.996347, exp_rate<=3 : 0.998486, loss: 0.001541, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20093 s, avg_samples: 56.0, ips: 278.71039 samples/s, eta: 2:18:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:32:46] ppocr INFO: epoch: [485/500], global_step: 504500, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.950081, exp_rate<=1 : 0.980719, exp_rate<=2 : 0.995955, exp_rate<=3 : 0.998546, loss: 0.001110, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20384 s, avg_samples: 56.0, ips: 274.72596 samples/s, eta: 2:17:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:33:04] ppocr INFO: epoch: [485/500], global_step: 504600, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.950586, exp_rate<=1 : 0.981078, exp_rate<=2 : 0.996022, exp_rate<=3 : 0.998756, loss: 0.001410, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17951 s, avg_samples: 56.0, ips: 311.96550 samples/s, eta: 2:17:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:33:22] ppocr INFO: epoch: [485/500], global_step: 504700, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.949950, exp_rate<=1 : 0.980940, exp_rate<=2 : 0.995700, exp_rate<=3 : 0.998804, loss: 0.001073, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17921 s, avg_samples: 56.0, ips: 312.48223 samples/s, eta: 2:17:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:33:40] ppocr INFO: epoch: [485/500], global_step: 504800, lr: 0.000100, edit distance: 0.001169, exp_rate: 0.949312, exp_rate<=1 : 0.980759, exp_rate<=2 : 0.995518, exp_rate<=3 : 0.998647, loss: 0.001332, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17994 s, avg_samples: 56.0, ips: 311.22245 samples/s, eta: 2:16:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:33:59] ppocr INFO: epoch: [485/500], global_step: 504900, lr: 0.000100, edit distance: 0.001206, exp_rate: 0.947337, exp_rate<=1 : 0.979770, exp_rate<=2 : 0.995185, exp_rate<=3 : 0.998558, loss: 0.001430, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19702 s, avg_samples: 56.0, ips: 284.23755 samples/s, eta: 2:16:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:34:18] ppocr INFO: epoch: [485/500], global_step: 505000, lr: 0.000100, edit distance: 0.001177, exp_rate: 0.948164, exp_rate<=1 : 0.980039, exp_rate<=2 : 0.995438, exp_rate<=3 : 0.998710, loss: 0.001428, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18388 s, avg_samples: 56.0, ips: 304.54251 samples/s, eta: 2:16:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:34:37] ppocr INFO: epoch: [485/500], global_step: 505100, lr: 0.000100, edit distance: 0.001180, exp_rate: 0.948250, exp_rate<=1 : 0.979977, exp_rate<=2 : 0.995408, exp_rate<=3 : 0.998676, loss: 0.001223, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18662 s, avg_samples: 56.0, ips: 300.07850 samples/s, eta: 2:15:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:34:55] ppocr INFO: epoch: [485/500], global_step: 505200, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.948135, exp_rate<=1 : 0.979828, exp_rate<=2 : 0.995404, exp_rate<=3 : 0.998670, loss: 0.001973, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18483 s, avg_samples: 56.0, ips: 302.97924 samples/s, eta: 2:15:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:35:14] ppocr INFO: epoch: [485/500], global_step: 505300, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.948106, exp_rate<=1 : 0.979800, exp_rate<=2 : 0.995477, exp_rate<=3 : 0.998701, loss: 0.001277, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18959 s, avg_samples: 56.0, ips: 295.36820 samples/s, eta: 2:15:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:35:33] ppocr INFO: epoch: [485/500], global_step: 505400, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.946981, exp_rate<=1 : 0.979525, exp_rate<=2 : 0.995308, exp_rate<=3 : 0.998657, loss: 0.001304, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18763 s, avg_samples: 56.0, ips: 298.45439 samples/s, eta: 2:14:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:35:53] ppocr INFO: epoch: [485/500], global_step: 505500, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.946617, exp_rate<=1 : 0.979199, exp_rate<=2 : 0.995211, exp_rate<=3 : 0.998618, loss: 0.001691, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19727 s, avg_samples: 56.0, ips: 283.87864 samples/s, eta: 2:14:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:36:12] ppocr INFO: epoch: [485/500], global_step: 505600, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.945968, exp_rate<=1 : 0.979061, exp_rate<=2 : 0.995245, exp_rate<=3 : 0.998652, loss: 0.001085, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19376 s, avg_samples: 56.0, ips: 289.01271 samples/s, eta: 2:14:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:36:30] ppocr INFO: epoch: [485/500], global_step: 505700, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.946323, exp_rate<=1 : 0.979283, exp_rate<=2 : 0.995365, exp_rate<=3 : 0.998643, loss: 0.001008, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17754 s, avg_samples: 56.0, ips: 315.42161 samples/s, eta: 2:14:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:36:49] ppocr INFO: epoch: [485/500], global_step: 505800, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.945772, exp_rate<=1 : 0.979082, exp_rate<=2 : 0.995346, exp_rate<=3 : 0.998644, loss: 0.001001, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19043 s, avg_samples: 56.0, ips: 294.07589 samples/s, eta: 2:13:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:37:07] ppocr INFO: epoch: [485/500], global_step: 505900, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.945568, exp_rate<=1 : 0.979071, exp_rate<=2 : 0.995302, exp_rate<=3 : 0.998575, loss: 0.001423, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18481 s, avg_samples: 56.0, ips: 303.00856 samples/s, eta: 2:13:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:37:26] ppocr INFO: epoch: [485/500], global_step: 506000, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.945295, exp_rate<=1 : 0.978901, exp_rate<=2 : 0.995308, exp_rate<=3 : 0.998559, loss: 0.001191, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18406 s, avg_samples: 56.0, ips: 304.24128 samples/s, eta: 2:13:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:37:45] ppocr INFO: epoch: [485/500], global_step: 506100, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.944947, exp_rate<=1 : 0.979032, exp_rate<=2 : 0.995302, exp_rate<=3 : 0.998615, loss: 0.001382, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19092 s, avg_samples: 56.0, ips: 293.31249 samples/s, eta: 2:12:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:38:05] ppocr INFO: epoch: [485/500], global_step: 506200, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.944097, exp_rate<=1 : 0.978718, exp_rate<=2 : 0.995233, exp_rate<=3 : 0.998593, loss: 0.001176, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20400 s, avg_samples: 56.0, ips: 274.51208 samples/s, eta: 2:12:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:38:23] ppocr INFO: epoch: [485/500], global_step: 506300, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.944358, exp_rate<=1 : 0.978822, exp_rate<=2 : 0.995259, exp_rate<=3 : 0.998614, loss: 0.001646, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17604 s, avg_samples: 56.0, ips: 318.11135 samples/s, eta: 2:12:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:38:41] ppocr INFO: epoch: [485/500], global_step: 506400, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.944287, exp_rate<=1 : 0.978796, exp_rate<=2 : 0.995349, exp_rate<=3 : 0.998623, loss: 0.001359, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17910 s, avg_samples: 56.0, ips: 312.67140 samples/s, eta: 2:11:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:39:01] ppocr INFO: epoch: [485/500], global_step: 506500, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.943899, exp_rate<=1 : 0.978614, exp_rate<=2 : 0.995381, exp_rate<=3 : 0.998642, loss: 0.001001, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20393 s, avg_samples: 56.0, ips: 274.60189 samples/s, eta: 2:11:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:39:20] ppocr INFO: epoch: [485/500], global_step: 506600, lr: 0.000100, edit distance: 0.001228, exp_rate: 0.944095, exp_rate<=1 : 0.978760, exp_rate<=2 : 0.995426, exp_rate<=3 : 0.998642, loss: 0.000924, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18366 s, avg_samples: 56.0, ips: 304.90791 samples/s, eta: 2:11:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:39:37] ppocr INFO: epoch: [485/500], global_step: 506700, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.944525, exp_rate<=1 : 0.979011, exp_rate<=2 : 0.995454, exp_rate<=3 : 0.998641, loss: 0.001617, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17098 s, avg_samples: 56.0, ips: 327.51702 samples/s, eta: 2:10:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:39:57] ppocr INFO: epoch: [485/500], global_step: 506800, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.944192, exp_rate<=1 : 0.978792, exp_rate<=2 : 0.995376, exp_rate<=3 : 0.998584, loss: 0.001261, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19831 s, avg_samples: 56.0, ips: 282.38380 samples/s, eta: 2:10:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:40:16] ppocr INFO: epoch: [485/500], global_step: 506900, lr: 0.000100, edit distance: 0.001261, exp_rate: 0.943521, exp_rate<=1 : 0.978463, exp_rate<=2 : 0.995297, exp_rate<=3 : 0.998530, loss: 0.001645, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19598 s, avg_samples: 56.0, ips: 285.74085 samples/s, eta: 2:10:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:40:18] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 05:40:19] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_485 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 05:40:39] ppocr INFO: epoch: [486/500], global_step: 507000, lr: 0.000100, edit distance: 0.001518, exp_rate: 0.941623, exp_rate<=1 : 0.977946, exp_rate<=2 : 0.996410, exp_rate<=3 : 0.998588, loss: 0.001258, avg_reader_cost: 0.03048 s, avg_batch_cost: 0.22529 s, avg_samples: 56.0, ips: 248.57206 samples/s, eta: 2:09:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:40:58] ppocr INFO: epoch: [486/500], global_step: 507100, lr: 0.000100, edit distance: 0.001307, exp_rate: 0.944788, exp_rate<=1 : 0.979533, exp_rate<=2 : 0.996533, exp_rate<=3 : 0.998806, loss: 0.001030, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18824 s, avg_samples: 56.0, ips: 297.48773 samples/s, eta: 2:09:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:41:17] ppocr INFO: epoch: [486/500], global_step: 507200, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.943249, exp_rate<=1 : 0.979413, exp_rate<=2 : 0.995981, exp_rate<=3 : 0.998424, loss: 0.001539, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19106 s, avg_samples: 56.0, ips: 293.10410 samples/s, eta: 2:09:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:41:36] ppocr INFO: epoch: [486/500], global_step: 507300, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.944246, exp_rate<=1 : 0.979240, exp_rate<=2 : 0.995611, exp_rate<=3 : 0.998184, loss: 0.001205, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18870 s, avg_samples: 56.0, ips: 296.77145 samples/s, eta: 2:08:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:41:54] ppocr INFO: epoch: [486/500], global_step: 507400, lr: 0.000100, edit distance: 0.001183, exp_rate: 0.945557, exp_rate<=1 : 0.979016, exp_rate<=2 : 0.995753, exp_rate<=3 : 0.998278, loss: 0.001250, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18357 s, avg_samples: 56.0, ips: 305.05832 samples/s, eta: 2:08:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:42:13] ppocr INFO: epoch: [486/500], global_step: 507500, lr: 0.000100, edit distance: 0.001109, exp_rate: 0.947546, exp_rate<=1 : 0.979920, exp_rate<=2 : 0.995831, exp_rate<=3 : 0.998260, loss: 0.000992, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18601 s, avg_samples: 56.0, ips: 301.05511 samples/s, eta: 2:08:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:42:33] ppocr INFO: epoch: [486/500], global_step: 507600, lr: 0.000100, edit distance: 0.001082, exp_rate: 0.948253, exp_rate<=1 : 0.980121, exp_rate<=2 : 0.995878, exp_rate<=3 : 0.998387, loss: 0.001061, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20131 s, avg_samples: 56.0, ips: 278.17690 samples/s, eta: 2:08:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:42:52] ppocr INFO: epoch: [486/500], global_step: 507700, lr: 0.000100, edit distance: 0.001058, exp_rate: 0.949072, exp_rate<=1 : 0.980471, exp_rate<=2 : 0.995918, exp_rate<=3 : 0.998469, loss: 0.001429, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19203 s, avg_samples: 56.0, ips: 291.62172 samples/s, eta: 2:07:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:43:11] ppocr INFO: epoch: [486/500], global_step: 507800, lr: 0.000100, edit distance: 0.001074, exp_rate: 0.948635, exp_rate<=1 : 0.980186, exp_rate<=2 : 0.995967, exp_rate<=3 : 0.998451, loss: 0.001124, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19013 s, avg_samples: 56.0, ips: 294.53723 samples/s, eta: 2:07:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:43:29] ppocr INFO: epoch: [486/500], global_step: 507900, lr: 0.000100, edit distance: 0.001073, exp_rate: 0.948746, exp_rate<=1 : 0.980393, exp_rate<=2 : 0.995915, exp_rate<=3 : 0.998450, loss: 0.001262, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17594 s, avg_samples: 56.0, ips: 318.28869 samples/s, eta: 2:07:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:43:47] ppocr INFO: epoch: [486/500], global_step: 508000, lr: 0.000100, edit distance: 0.001064, exp_rate: 0.949376, exp_rate<=1 : 0.980747, exp_rate<=2 : 0.995929, exp_rate<=3 : 0.998461, loss: 0.001310, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18606 s, avg_samples: 56.0, ips: 300.98153 samples/s, eta: 2:06:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:44:06] ppocr INFO: epoch: [486/500], global_step: 508100, lr: 0.000100, edit distance: 0.001088, exp_rate: 0.949604, exp_rate<=1 : 0.980626, exp_rate<=2 : 0.995864, exp_rate<=3 : 0.998390, loss: 0.001826, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18539 s, avg_samples: 56.0, ips: 302.07283 samples/s, eta: 2:06:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:44:23] ppocr INFO: epoch: [486/500], global_step: 508200, lr: 0.000100, edit distance: 0.001101, exp_rate: 0.949378, exp_rate<=1 : 0.980480, exp_rate<=2 : 0.995843, exp_rate<=3 : 0.998414, loss: 0.001372, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17445 s, avg_samples: 56.0, ips: 321.01521 samples/s, eta: 2:06:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:44:42] ppocr INFO: epoch: [486/500], global_step: 508300, lr: 0.000100, edit distance: 0.001106, exp_rate: 0.948918, exp_rate<=1 : 0.980308, exp_rate<=2 : 0.995900, exp_rate<=3 : 0.998429, loss: 0.001589, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18581 s, avg_samples: 56.0, ips: 301.38885 samples/s, eta: 2:05:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:45:01] ppocr INFO: epoch: [486/500], global_step: 508400, lr: 0.000100, edit distance: 0.001110, exp_rate: 0.948256, exp_rate<=1 : 0.980007, exp_rate<=2 : 0.995867, exp_rate<=3 : 0.998419, loss: 0.000783, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19026 s, avg_samples: 56.0, ips: 294.33499 samples/s, eta: 2:05:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:45:21] ppocr INFO: epoch: [486/500], global_step: 508500, lr: 0.000100, edit distance: 0.001114, exp_rate: 0.947624, exp_rate<=1 : 0.979575, exp_rate<=2 : 0.995754, exp_rate<=3 : 0.998380, loss: 0.000986, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19726 s, avg_samples: 56.0, ips: 283.88989 samples/s, eta: 2:05:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:45:40] ppocr INFO: epoch: [486/500], global_step: 508600, lr: 0.000100, edit distance: 0.001112, exp_rate: 0.947262, exp_rate<=1 : 0.979374, exp_rate<=2 : 0.995714, exp_rate<=3 : 0.998392, loss: 0.001571, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19649 s, avg_samples: 56.0, ips: 285.00800 samples/s, eta: 2:04:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:46:00] ppocr INFO: epoch: [486/500], global_step: 508700, lr: 0.000100, edit distance: 0.001121, exp_rate: 0.946903, exp_rate<=1 : 0.979268, exp_rate<=2 : 0.995632, exp_rate<=3 : 0.998342, loss: 0.001387, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19579 s, avg_samples: 56.0, ips: 286.01355 samples/s, eta: 2:04:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:46:19] ppocr INFO: epoch: [486/500], global_step: 508800, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.946159, exp_rate<=1 : 0.978973, exp_rate<=2 : 0.995532, exp_rate<=3 : 0.998308, loss: 0.001748, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18922 s, avg_samples: 56.0, ips: 295.95027 samples/s, eta: 2:04:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:46:38] ppocr INFO: epoch: [486/500], global_step: 508900, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.945804, exp_rate<=1 : 0.978804, exp_rate<=2 : 0.995398, exp_rate<=3 : 0.998292, loss: 0.001524, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19165 s, avg_samples: 56.0, ips: 292.19733 samples/s, eta: 2:03:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:46:58] ppocr INFO: epoch: [486/500], global_step: 509000, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.945327, exp_rate<=1 : 0.978558, exp_rate<=2 : 0.995321, exp_rate<=3 : 0.998256, loss: 0.001599, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19372 s, avg_samples: 56.0, ips: 289.07169 samples/s, eta: 2:03:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:47:15] ppocr INFO: epoch: [486/500], global_step: 509100, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.945434, exp_rate<=1 : 0.978654, exp_rate<=2 : 0.995410, exp_rate<=3 : 0.998272, loss: 0.001077, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17820 s, avg_samples: 56.0, ips: 314.25015 samples/s, eta: 2:03:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:47:34] ppocr INFO: epoch: [486/500], global_step: 509200, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.945567, exp_rate<=1 : 0.978742, exp_rate<=2 : 0.995420, exp_rate<=3 : 0.998270, loss: 0.000979, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18526 s, avg_samples: 56.0, ips: 302.27137 samples/s, eta: 2:02:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:47:54] ppocr INFO: epoch: [486/500], global_step: 509300, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.945348, exp_rate<=1 : 0.978681, exp_rate<=2 : 0.995372, exp_rate<=3 : 0.998274, loss: 0.001072, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20195 s, avg_samples: 56.0, ips: 277.30311 samples/s, eta: 2:02:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:48:13] ppocr INFO: epoch: [486/500], global_step: 509400, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.945251, exp_rate<=1 : 0.978696, exp_rate<=2 : 0.995451, exp_rate<=3 : 0.998306, loss: 0.001604, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18729 s, avg_samples: 56.0, ips: 299.00875 samples/s, eta: 2:02:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:48:31] ppocr INFO: epoch: [486/500], global_step: 509500, lr: 0.000100, edit distance: 0.001208, exp_rate: 0.945140, exp_rate<=1 : 0.978682, exp_rate<=2 : 0.995465, exp_rate<=3 : 0.998304, loss: 0.001528, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18239 s, avg_samples: 56.0, ips: 307.03020 samples/s, eta: 2:01:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:48:51] ppocr INFO: epoch: [486/500], global_step: 509600, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.944855, exp_rate<=1 : 0.978506, exp_rate<=2 : 0.995429, exp_rate<=3 : 0.998314, loss: 0.001791, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19380 s, avg_samples: 56.0, ips: 288.95991 samples/s, eta: 2:01:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:48:56] ppocr INFO: epoch: [486/500], global_step: 509640, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.945114, exp_rate<=1 : 0.978615, exp_rate<=2 : 0.995455, exp_rate<=3 : 0.998339, loss: 0.001569, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.05927 s, avg_samples: 22.4, ips: 377.91303 samples/s, eta: 2:01:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:48:59] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 05:49:11] ppocr INFO: epoch: [487/500], global_step: 509700, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.939007, exp_rate<=1 : 0.976492, exp_rate<=2 : 0.996362, exp_rate<=3 : 0.999197, loss: 0.001467, avg_reader_cost: 0.02534 s, avg_batch_cost: 0.14401 s, avg_samples: 33.6, ips: 233.31346 samples/s, eta: 2:01:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:49:29] ppocr INFO: epoch: [487/500], global_step: 509800, lr: 0.000100, edit distance: 0.001212, exp_rate: 0.943799, exp_rate<=1 : 0.979002, exp_rate<=2 : 0.996136, exp_rate<=3 : 0.998918, loss: 0.001602, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17616 s, avg_samples: 56.0, ips: 317.89375 samples/s, eta: 2:01:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:49:47] ppocr INFO: epoch: [487/500], global_step: 509900, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.944949, exp_rate<=1 : 0.979827, exp_rate<=2 : 0.996004, exp_rate<=3 : 0.998855, loss: 0.001727, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18614 s, avg_samples: 56.0, ips: 300.84307 samples/s, eta: 2:00:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:50:06] ppocr INFO: epoch: [487/500], global_step: 510000, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.944936, exp_rate<=1 : 0.978732, exp_rate<=2 : 0.995899, exp_rate<=3 : 0.998701, loss: 0.000914, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18593 s, avg_samples: 56.0, ips: 301.18804 samples/s, eta: 2:00:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:50:24] ppocr INFO: epoch: [487/500], global_step: 510100, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.946926, exp_rate<=1 : 0.979411, exp_rate<=2 : 0.995759, exp_rate<=3 : 0.998613, loss: 0.000943, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18341 s, avg_samples: 56.0, ips: 305.33034 samples/s, eta: 2:00:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:50:44] ppocr INFO: epoch: [487/500], global_step: 510200, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.947142, exp_rate<=1 : 0.980272, exp_rate<=2 : 0.995978, exp_rate<=3 : 0.998670, loss: 0.000858, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19789 s, avg_samples: 56.0, ips: 282.98811 samples/s, eta: 1:59:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:51:03] ppocr INFO: epoch: [487/500], global_step: 510300, lr: 0.000100, edit distance: 0.001102, exp_rate: 0.947986, exp_rate<=1 : 0.980354, exp_rate<=2 : 0.996055, exp_rate<=3 : 0.998787, loss: 0.001064, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19166 s, avg_samples: 56.0, ips: 292.17831 samples/s, eta: 1:59:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:51:22] ppocr INFO: epoch: [487/500], global_step: 510400, lr: 0.000100, edit distance: 0.001123, exp_rate: 0.947463, exp_rate<=1 : 0.979932, exp_rate<=2 : 0.995729, exp_rate<=3 : 0.998702, loss: 0.001502, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19004 s, avg_samples: 56.0, ips: 294.67968 samples/s, eta: 1:59:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:51:42] ppocr INFO: epoch: [487/500], global_step: 510500, lr: 0.000100, edit distance: 0.001117, exp_rate: 0.947496, exp_rate<=1 : 0.979813, exp_rate<=2 : 0.995801, exp_rate<=3 : 0.998627, loss: 0.001590, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19682 s, avg_samples: 56.0, ips: 284.52348 samples/s, eta: 1:58:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:52:03] ppocr INFO: epoch: [487/500], global_step: 510600, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.947139, exp_rate<=1 : 0.979882, exp_rate<=2 : 0.995699, exp_rate<=3 : 0.998583, loss: 0.001182, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20657 s, avg_samples: 56.0, ips: 271.09948 samples/s, eta: 1:58:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:52:22] ppocr INFO: epoch: [487/500], global_step: 510700, lr: 0.000100, edit distance: 0.001107, exp_rate: 0.947383, exp_rate<=1 : 0.980070, exp_rate<=2 : 0.995832, exp_rate<=3 : 0.998582, loss: 0.001032, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19765 s, avg_samples: 56.0, ips: 283.32773 samples/s, eta: 1:58:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:52:39] ppocr INFO: epoch: [487/500], global_step: 510800, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.947315, exp_rate<=1 : 0.979914, exp_rate<=2 : 0.995856, exp_rate<=3 : 0.998640, loss: 0.001405, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.17169 s, avg_samples: 56.0, ips: 326.17274 samples/s, eta: 1:57:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:52:58] ppocr INFO: epoch: [487/500], global_step: 510900, lr: 0.000100, edit distance: 0.001126, exp_rate: 0.947093, exp_rate<=1 : 0.979873, exp_rate<=2 : 0.995696, exp_rate<=3 : 0.998677, loss: 0.001234, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18925 s, avg_samples: 56.0, ips: 295.90266 samples/s, eta: 1:57:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:53:18] ppocr INFO: epoch: [487/500], global_step: 511000, lr: 0.000100, edit distance: 0.001146, exp_rate: 0.946686, exp_rate<=1 : 0.979630, exp_rate<=2 : 0.995582, exp_rate<=3 : 0.998586, loss: 0.001546, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19199 s, avg_samples: 56.0, ips: 291.67903 samples/s, eta: 1:57:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:53:37] ppocr INFO: epoch: [487/500], global_step: 511100, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.944899, exp_rate<=1 : 0.978951, exp_rate<=2 : 0.995399, exp_rate<=3 : 0.998532, loss: 0.001524, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19540 s, avg_samples: 56.0, ips: 286.58545 samples/s, eta: 1:56:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:53:55] ppocr INFO: epoch: [487/500], global_step: 511200, lr: 0.000100, edit distance: 0.001179, exp_rate: 0.944781, exp_rate<=1 : 0.978919, exp_rate<=2 : 0.995373, exp_rate<=3 : 0.998457, loss: 0.001270, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17721 s, avg_samples: 56.0, ips: 316.00339 samples/s, eta: 1:56:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:54:13] ppocr INFO: epoch: [487/500], global_step: 511300, lr: 0.000100, edit distance: 0.001172, exp_rate: 0.945082, exp_rate<=1 : 0.979076, exp_rate<=2 : 0.995380, exp_rate<=3 : 0.998473, loss: 0.000841, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17902 s, avg_samples: 56.0, ips: 312.80730 samples/s, eta: 1:56:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:54:32] ppocr INFO: epoch: [487/500], global_step: 511400, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.944959, exp_rate<=1 : 0.978924, exp_rate<=2 : 0.995406, exp_rate<=3 : 0.998488, loss: 0.001328, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18785 s, avg_samples: 56.0, ips: 298.11415 samples/s, eta: 1:55:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:54:51] ppocr INFO: epoch: [487/500], global_step: 511500, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.945036, exp_rate<=1 : 0.978830, exp_rate<=2 : 0.995317, exp_rate<=3 : 0.998475, loss: 0.001027, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19184 s, avg_samples: 56.0, ips: 291.90950 samples/s, eta: 1:55:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:55:09] ppocr INFO: epoch: [487/500], global_step: 511600, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.944955, exp_rate<=1 : 0.978746, exp_rate<=2 : 0.995271, exp_rate<=3 : 0.998485, loss: 0.000663, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18287 s, avg_samples: 56.0, ips: 306.23399 samples/s, eta: 1:55:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:55:29] ppocr INFO: epoch: [487/500], global_step: 511700, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.943718, exp_rate<=1 : 0.978207, exp_rate<=2 : 0.995016, exp_rate<=3 : 0.998311, loss: 0.001573, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19703 s, avg_samples: 56.0, ips: 284.22130 samples/s, eta: 1:55:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:55:48] ppocr INFO: epoch: [487/500], global_step: 511800, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.943299, exp_rate<=1 : 0.978021, exp_rate<=2 : 0.995013, exp_rate<=3 : 0.998346, loss: 0.001533, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19431 s, avg_samples: 56.0, ips: 288.20590 samples/s, eta: 1:54:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:56:08] ppocr INFO: epoch: [487/500], global_step: 511900, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.942669, exp_rate<=1 : 0.977664, exp_rate<=2 : 0.994991, exp_rate<=3 : 0.998360, loss: 0.001401, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19653 s, avg_samples: 56.0, ips: 284.93672 samples/s, eta: 1:54:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:56:26] ppocr INFO: epoch: [487/500], global_step: 512000, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.942569, exp_rate<=1 : 0.977777, exp_rate<=2 : 0.995047, exp_rate<=3 : 0.998389, loss: 0.001418, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18181 s, avg_samples: 56.0, ips: 308.01708 samples/s, eta: 1:54:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:56:45] ppocr INFO: epoch: [487/500], global_step: 512100, lr: 0.000100, edit distance: 0.001198, exp_rate: 0.942522, exp_rate<=1 : 0.977814, exp_rate<=2 : 0.995088, exp_rate<=3 : 0.998404, loss: 0.001158, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18681 s, avg_samples: 56.0, ips: 299.77133 samples/s, eta: 1:53:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:57:02] ppocr INFO: epoch: [487/500], global_step: 512200, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.942756, exp_rate<=1 : 0.977798, exp_rate<=2 : 0.995169, exp_rate<=3 : 0.998439, loss: 0.001056, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17310 s, avg_samples: 56.0, ips: 323.51167 samples/s, eta: 1:53:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:57:20] ppocr INFO: epoch: [487/500], global_step: 512300, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.942979, exp_rate<=1 : 0.977872, exp_rate<=2 : 0.995190, exp_rate<=3 : 0.998417, loss: 0.001098, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18085 s, avg_samples: 56.0, ips: 309.65325 samples/s, eta: 1:53:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:57:36] ppocr INFO: epoch: [487/500], global_step: 512380, lr: 0.000100, edit distance: 0.001195, exp_rate: 0.943126, exp_rate<=1 : 0.978019, exp_rate<=2 : 0.995244, exp_rate<=3 : 0.998437, loss: 0.000803, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15342 s, avg_samples: 44.8, ips: 292.00210 samples/s, eta: 1:52:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:57:38] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 05:57:42] ppocr INFO: epoch: [488/500], global_step: 512400, lr: 0.000100, edit distance: 0.000914, exp_rate: 0.955763, exp_rate<=1 : 0.985913, exp_rate<=2 : 0.998295, exp_rate<=3 : 1.000000, loss: 0.001379, avg_reader_cost: 0.02506 s, avg_batch_cost: 0.05994 s, avg_samples: 11.2, ips: 186.86670 samples/s, eta: 1:52:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:58:01] ppocr INFO: epoch: [488/500], global_step: 512500, lr: 0.000100, edit distance: 0.000797, exp_rate: 0.957460, exp_rate<=1 : 0.983738, exp_rate<=2 : 0.996429, exp_rate<=3 : 0.999240, loss: 0.001496, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19377 s, avg_samples: 56.0, ips: 289.00050 samples/s, eta: 1:52:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:58:20] ppocr INFO: epoch: [488/500], global_step: 512600, lr: 0.000100, edit distance: 0.000932, exp_rate: 0.952618, exp_rate<=1 : 0.982395, exp_rate<=2 : 0.996276, exp_rate<=3 : 0.999101, loss: 0.000964, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19054 s, avg_samples: 56.0, ips: 293.89936 samples/s, eta: 1:52:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:58:39] ppocr INFO: epoch: [488/500], global_step: 512700, lr: 0.000100, edit distance: 0.000979, exp_rate: 0.954423, exp_rate<=1 : 0.982967, exp_rate<=2 : 0.996262, exp_rate<=3 : 0.998975, loss: 0.000671, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18825 s, avg_samples: 56.0, ips: 297.47562 samples/s, eta: 1:51:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:58:57] ppocr INFO: epoch: [488/500], global_step: 512800, lr: 0.000100, edit distance: 0.000942, exp_rate: 0.956118, exp_rate<=1 : 0.983980, exp_rate<=2 : 0.996572, exp_rate<=3 : 0.998999, loss: 0.001311, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18054 s, avg_samples: 56.0, ips: 310.17360 samples/s, eta: 1:51:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:59:16] ppocr INFO: epoch: [488/500], global_step: 512900, lr: 0.000100, edit distance: 0.000940, exp_rate: 0.955098, exp_rate<=1 : 0.983956, exp_rate<=2 : 0.996369, exp_rate<=3 : 0.998776, loss: 0.000967, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19245 s, avg_samples: 56.0, ips: 290.97719 samples/s, eta: 1:51:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:59:35] ppocr INFO: epoch: [488/500], global_step: 513000, lr: 0.000100, edit distance: 0.000954, exp_rate: 0.954688, exp_rate<=1 : 0.983553, exp_rate<=2 : 0.996431, exp_rate<=3 : 0.998850, loss: 0.001215, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18202 s, avg_samples: 56.0, ips: 307.66440 samples/s, eta: 1:50:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 05:59:54] ppocr INFO: epoch: [488/500], global_step: 513100, lr: 0.000100, edit distance: 0.000975, exp_rate: 0.952435, exp_rate<=1 : 0.982740, exp_rate<=2 : 0.996197, exp_rate<=3 : 0.998794, loss: 0.001521, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19154 s, avg_samples: 56.0, ips: 292.37265 samples/s, eta: 1:50:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:00:11] ppocr INFO: epoch: [488/500], global_step: 513200, lr: 0.000100, edit distance: 0.001004, exp_rate: 0.952255, exp_rate<=1 : 0.982911, exp_rate<=2 : 0.996333, exp_rate<=3 : 0.998841, loss: 0.001388, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17135 s, avg_samples: 56.0, ips: 326.80877 samples/s, eta: 1:50:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:00:32] ppocr INFO: epoch: [488/500], global_step: 513300, lr: 0.000100, edit distance: 0.001036, exp_rate: 0.949730, exp_rate<=1 : 0.981977, exp_rate<=2 : 0.995977, exp_rate<=3 : 0.998706, loss: 0.001468, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20778 s, avg_samples: 56.0, ips: 269.51893 samples/s, eta: 1:49:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:00:50] ppocr INFO: epoch: [488/500], global_step: 513400, lr: 0.000100, edit distance: 0.001043, exp_rate: 0.949503, exp_rate<=1 : 0.981878, exp_rate<=2 : 0.995787, exp_rate<=3 : 0.998667, loss: 0.001117, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18001 s, avg_samples: 56.0, ips: 311.08862 samples/s, eta: 1:49:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:01:10] ppocr INFO: epoch: [488/500], global_step: 513500, lr: 0.000100, edit distance: 0.001060, exp_rate: 0.948811, exp_rate<=1 : 0.981727, exp_rate<=2 : 0.995767, exp_rate<=3 : 0.998617, loss: 0.001250, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19912 s, avg_samples: 56.0, ips: 281.23578 samples/s, eta: 1:49:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:01:28] ppocr INFO: epoch: [488/500], global_step: 513600, lr: 0.000100, edit distance: 0.001062, exp_rate: 0.949141, exp_rate<=1 : 0.981607, exp_rate<=2 : 0.995819, exp_rate<=3 : 0.998653, loss: 0.001917, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18580 s, avg_samples: 56.0, ips: 301.40378 samples/s, eta: 1:48:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:01:46] ppocr INFO: epoch: [488/500], global_step: 513700, lr: 0.000100, edit distance: 0.001061, exp_rate: 0.949358, exp_rate<=1 : 0.981663, exp_rate<=2 : 0.995820, exp_rate<=3 : 0.998621, loss: 0.001212, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17769 s, avg_samples: 56.0, ips: 315.16102 samples/s, eta: 1:48:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:02:06] ppocr INFO: epoch: [488/500], global_step: 513800, lr: 0.000100, edit distance: 0.001082, exp_rate: 0.948555, exp_rate<=1 : 0.981351, exp_rate<=2 : 0.995685, exp_rate<=3 : 0.998549, loss: 0.001553, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19720 s, avg_samples: 56.0, ips: 283.97877 samples/s, eta: 1:48:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:02:25] ppocr INFO: epoch: [488/500], global_step: 513900, lr: 0.000100, edit distance: 0.001087, exp_rate: 0.948198, exp_rate<=1 : 0.981127, exp_rate<=2 : 0.995677, exp_rate<=3 : 0.998537, loss: 0.001272, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19124 s, avg_samples: 56.0, ips: 292.82109 samples/s, eta: 1:48:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:02:44] ppocr INFO: epoch: [488/500], global_step: 514000, lr: 0.000100, edit distance: 0.001089, exp_rate: 0.947897, exp_rate<=1 : 0.980906, exp_rate<=2 : 0.995644, exp_rate<=3 : 0.998496, loss: 0.001660, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19398 s, avg_samples: 56.0, ips: 288.68281 samples/s, eta: 1:47:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:03:03] ppocr INFO: epoch: [488/500], global_step: 514100, lr: 0.000100, edit distance: 0.001100, exp_rate: 0.947067, exp_rate<=1 : 0.980431, exp_rate<=2 : 0.995534, exp_rate<=3 : 0.998442, loss: 0.001204, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18732 s, avg_samples: 56.0, ips: 298.95053 samples/s, eta: 1:47:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:03:22] ppocr INFO: epoch: [488/500], global_step: 514200, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.944195, exp_rate<=1 : 0.978954, exp_rate<=2 : 0.994826, exp_rate<=3 : 0.998070, loss: 0.004026, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18635 s, avg_samples: 56.0, ips: 300.50796 samples/s, eta: 1:47:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:03:41] ppocr INFO: epoch: [488/500], global_step: 514300, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.942280, exp_rate<=1 : 0.978018, exp_rate<=2 : 0.994506, exp_rate<=3 : 0.997918, loss: 0.002262, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19120 s, avg_samples: 56.0, ips: 292.88993 samples/s, eta: 1:46:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:04:01] ppocr INFO: epoch: [488/500], global_step: 514400, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.940973, exp_rate<=1 : 0.977617, exp_rate<=2 : 0.994400, exp_rate<=3 : 0.997899, loss: 0.001386, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19599 s, avg_samples: 56.0, ips: 285.72910 samples/s, eta: 1:46:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:04:18] ppocr INFO: epoch: [488/500], global_step: 514500, lr: 0.000100, edit distance: 0.001229, exp_rate: 0.941092, exp_rate<=1 : 0.977860, exp_rate<=2 : 0.994476, exp_rate<=3 : 0.997927, loss: 0.001609, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17874 s, avg_samples: 56.0, ips: 313.29788 samples/s, eta: 1:46:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:04:37] ppocr INFO: epoch: [488/500], global_step: 514600, lr: 0.000100, edit distance: 0.001224, exp_rate: 0.941385, exp_rate<=1 : 0.977895, exp_rate<=2 : 0.994519, exp_rate<=3 : 0.997935, loss: 0.001718, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18566 s, avg_samples: 56.0, ips: 301.63430 samples/s, eta: 1:45:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:04:55] ppocr INFO: epoch: [488/500], global_step: 514700, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.941404, exp_rate<=1 : 0.977956, exp_rate<=2 : 0.994598, exp_rate<=3 : 0.997983, loss: 0.001228, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17879 s, avg_samples: 56.0, ips: 313.21687 samples/s, eta: 1:45:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:05:13] ppocr INFO: epoch: [488/500], global_step: 514800, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.941354, exp_rate<=1 : 0.977992, exp_rate<=2 : 0.994637, exp_rate<=3 : 0.998015, loss: 0.001217, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18289 s, avg_samples: 56.0, ips: 306.20070 samples/s, eta: 1:45:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:05:34] ppocr INFO: epoch: [488/500], global_step: 514900, lr: 0.000100, edit distance: 0.001230, exp_rate: 0.941320, exp_rate<=1 : 0.978052, exp_rate<=2 : 0.994660, exp_rate<=3 : 0.998056, loss: 0.001051, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21182 s, avg_samples: 56.0, ips: 264.37934 samples/s, eta: 1:44:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:05:53] ppocr INFO: epoch: [488/500], global_step: 515000, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.941575, exp_rate<=1 : 0.978181, exp_rate<=2 : 0.994702, exp_rate<=3 : 0.998057, loss: 0.000916, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18425 s, avg_samples: 56.0, ips: 303.93542 samples/s, eta: 1:44:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:06:11] ppocr INFO: epoch: [488/500], global_step: 515100, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.941863, exp_rate<=1 : 0.978278, exp_rate<=2 : 0.994718, exp_rate<=3 : 0.998099, loss: 0.001197, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18456 s, avg_samples: 56.0, ips: 303.41804 samples/s, eta: 1:44:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:06:15] ppocr INFO: epoch: [488/500], global_step: 515120, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.941745, exp_rate<=1 : 0.978258, exp_rate<=2 : 0.994731, exp_rate<=3 : 0.998107, loss: 0.001429, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03688 s, avg_samples: 11.2, ips: 303.71646 samples/s, eta: 1:44:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:06:18] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 06:06:32] ppocr INFO: epoch: [489/500], global_step: 515200, lr: 0.000100, edit distance: 0.001093, exp_rate: 0.955364, exp_rate<=1 : 0.984609, exp_rate<=2 : 0.996873, exp_rate<=3 : 0.999747, loss: 0.001370, avg_reader_cost: 0.03044 s, avg_batch_cost: 0.16552 s, avg_samples: 44.8, ips: 270.65829 samples/s, eta: 1:43:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:06:51] ppocr INFO: epoch: [489/500], global_step: 515300, lr: 0.000100, edit distance: 0.001107, exp_rate: 0.950285, exp_rate<=1 : 0.981932, exp_rate<=2 : 0.996282, exp_rate<=3 : 0.999057, loss: 0.001569, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19017 s, avg_samples: 56.0, ips: 294.46774 samples/s, eta: 1:43:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:07:10] ppocr INFO: epoch: [489/500], global_step: 515400, lr: 0.000100, edit distance: 0.001122, exp_rate: 0.951240, exp_rate<=1 : 0.982011, exp_rate<=2 : 0.996312, exp_rate<=3 : 0.998976, loss: 0.000665, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19046 s, avg_samples: 56.0, ips: 294.02527 samples/s, eta: 1:43:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:07:28] ppocr INFO: epoch: [489/500], global_step: 515500, lr: 0.000100, edit distance: 0.001076, exp_rate: 0.951742, exp_rate<=1 : 0.982192, exp_rate<=2 : 0.996069, exp_rate<=3 : 0.998520, loss: 0.000961, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18489 s, avg_samples: 56.0, ips: 302.88786 samples/s, eta: 1:42:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:07:46] ppocr INFO: epoch: [489/500], global_step: 515600, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.951180, exp_rate<=1 : 0.982369, exp_rate<=2 : 0.996379, exp_rate<=3 : 0.998768, loss: 0.001646, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17769 s, avg_samples: 56.0, ips: 315.14881 samples/s, eta: 1:42:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:08:05] ppocr INFO: epoch: [489/500], global_step: 515700, lr: 0.000100, edit distance: 0.001133, exp_rate: 0.951095, exp_rate<=1 : 0.982206, exp_rate<=2 : 0.996400, exp_rate<=3 : 0.998717, loss: 0.001307, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19392 s, avg_samples: 56.0, ips: 288.78563 samples/s, eta: 1:42:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:08:25] ppocr INFO: epoch: [489/500], global_step: 515800, lr: 0.000100, edit distance: 0.001109, exp_rate: 0.951342, exp_rate<=1 : 0.982278, exp_rate<=2 : 0.996577, exp_rate<=3 : 0.998775, loss: 0.001447, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19396 s, avg_samples: 56.0, ips: 288.72452 samples/s, eta: 1:42:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:08:44] ppocr INFO: epoch: [489/500], global_step: 515900, lr: 0.000100, edit distance: 0.001125, exp_rate: 0.949652, exp_rate<=1 : 0.981610, exp_rate<=2 : 0.996299, exp_rate<=3 : 0.998643, loss: 0.000580, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.73781 samples/s, eta: 1:41:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:09:02] ppocr INFO: epoch: [489/500], global_step: 516000, lr: 0.000100, edit distance: 0.001093, exp_rate: 0.950777, exp_rate<=1 : 0.981999, exp_rate<=2 : 0.996375, exp_rate<=3 : 0.998703, loss: 0.001303, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18191 s, avg_samples: 56.0, ips: 307.84200 samples/s, eta: 1:41:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:09:21] ppocr INFO: epoch: [489/500], global_step: 516100, lr: 0.000100, edit distance: 0.001090, exp_rate: 0.951167, exp_rate<=1 : 0.981942, exp_rate<=2 : 0.996491, exp_rate<=3 : 0.998723, loss: 0.001358, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18289 s, avg_samples: 56.0, ips: 306.20329 samples/s, eta: 1:41:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:09:41] ppocr INFO: epoch: [489/500], global_step: 516200, lr: 0.000100, edit distance: 0.001085, exp_rate: 0.950697, exp_rate<=1 : 0.981709, exp_rate<=2 : 0.996365, exp_rate<=3 : 0.998724, loss: 0.000992, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20129 s, avg_samples: 56.0, ips: 278.21233 samples/s, eta: 1:40:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:09:58] ppocr INFO: epoch: [489/500], global_step: 516300, lr: 0.000100, edit distance: 0.001086, exp_rate: 0.950858, exp_rate<=1 : 0.981785, exp_rate<=2 : 0.996305, exp_rate<=3 : 0.998708, loss: 0.001448, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17260 s, avg_samples: 56.0, ips: 324.45422 samples/s, eta: 1:40:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:10:18] ppocr INFO: epoch: [489/500], global_step: 516400, lr: 0.000100, edit distance: 0.001108, exp_rate: 0.949879, exp_rate<=1 : 0.981244, exp_rate<=2 : 0.995982, exp_rate<=3 : 0.998549, loss: 0.001108, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19432 s, avg_samples: 56.0, ips: 288.18560 samples/s, eta: 1:40:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:10:36] ppocr INFO: epoch: [489/500], global_step: 516500, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.949595, exp_rate<=1 : 0.981348, exp_rate<=2 : 0.996014, exp_rate<=3 : 0.998510, loss: 0.002664, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18235 s, avg_samples: 56.0, ips: 307.10393 samples/s, eta: 1:39:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:10:55] ppocr INFO: epoch: [489/500], global_step: 516600, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.948564, exp_rate<=1 : 0.980797, exp_rate<=2 : 0.995739, exp_rate<=3 : 0.998440, loss: 0.001284, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18800 s, avg_samples: 56.0, ips: 297.86867 samples/s, eta: 1:39:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:11:14] ppocr INFO: epoch: [489/500], global_step: 516700, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.947838, exp_rate<=1 : 0.980552, exp_rate<=2 : 0.995644, exp_rate<=3 : 0.998454, loss: 0.001709, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18981 s, avg_samples: 56.0, ips: 295.03123 samples/s, eta: 1:39:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:11:32] ppocr INFO: epoch: [489/500], global_step: 516800, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.947571, exp_rate<=1 : 0.980577, exp_rate<=2 : 0.995679, exp_rate<=3 : 0.998501, loss: 0.001410, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18166 s, avg_samples: 56.0, ips: 308.26640 samples/s, eta: 1:38:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:11:51] ppocr INFO: epoch: [489/500], global_step: 516900, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.947064, exp_rate<=1 : 0.980469, exp_rate<=2 : 0.995739, exp_rate<=3 : 0.998466, loss: 0.001389, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19538 s, avg_samples: 56.0, ips: 286.62379 samples/s, eta: 1:38:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:12:11] ppocr INFO: epoch: [489/500], global_step: 517000, lr: 0.000100, edit distance: 0.001207, exp_rate: 0.946528, exp_rate<=1 : 0.980229, exp_rate<=2 : 0.995699, exp_rate<=3 : 0.998449, loss: 0.001259, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19720 s, avg_samples: 56.0, ips: 283.98072 samples/s, eta: 1:38:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:12:29] ppocr INFO: epoch: [489/500], global_step: 517100, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.946365, exp_rate<=1 : 0.980188, exp_rate<=2 : 0.995739, exp_rate<=3 : 0.998475, loss: 0.001437, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18273 s, avg_samples: 56.0, ips: 306.45471 samples/s, eta: 1:37:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:12:48] ppocr INFO: epoch: [489/500], global_step: 517200, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.946114, exp_rate<=1 : 0.980017, exp_rate<=2 : 0.995714, exp_rate<=3 : 0.998440, loss: 0.001002, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18530 s, avg_samples: 56.0, ips: 302.21982 samples/s, eta: 1:37:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:13:08] ppocr INFO: epoch: [489/500], global_step: 517300, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.945744, exp_rate<=1 : 0.979840, exp_rate<=2 : 0.995706, exp_rate<=3 : 0.998414, loss: 0.001521, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19537 s, avg_samples: 56.0, ips: 286.63777 samples/s, eta: 1:37:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:13:27] ppocr INFO: epoch: [489/500], global_step: 517400, lr: 0.000100, edit distance: 0.001223, exp_rate: 0.945350, exp_rate<=1 : 0.979734, exp_rate<=2 : 0.995621, exp_rate<=3 : 0.998372, loss: 0.001768, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19205 s, avg_samples: 56.0, ips: 291.59479 samples/s, eta: 1:36:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:13:47] ppocr INFO: epoch: [489/500], global_step: 517500, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.944756, exp_rate<=1 : 0.979560, exp_rate<=2 : 0.995547, exp_rate<=3 : 0.998332, loss: 0.002132, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20453 s, avg_samples: 56.0, ips: 273.80087 samples/s, eta: 1:36:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:14:06] ppocr INFO: epoch: [489/500], global_step: 517600, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.944618, exp_rate<=1 : 0.979469, exp_rate<=2 : 0.995540, exp_rate<=3 : 0.998345, loss: 0.001471, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18899 s, avg_samples: 56.0, ips: 296.30931 samples/s, eta: 1:36:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:14:25] ppocr INFO: epoch: [489/500], global_step: 517700, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.944383, exp_rate<=1 : 0.979258, exp_rate<=2 : 0.995480, exp_rate<=3 : 0.998347, loss: 0.001675, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18393 s, avg_samples: 56.0, ips: 304.46357 samples/s, eta: 1:35:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:14:43] ppocr INFO: epoch: [489/500], global_step: 517800, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.944406, exp_rate<=1 : 0.979314, exp_rate<=2 : 0.995515, exp_rate<=3 : 0.998351, loss: 0.001446, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18262 s, avg_samples: 56.0, ips: 306.65464 samples/s, eta: 1:35:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:14:54] ppocr INFO: epoch: [489/500], global_step: 517860, lr: 0.000100, edit distance: 0.001227, exp_rate: 0.944434, exp_rate<=1 : 0.979300, exp_rate<=2 : 0.995507, exp_rate<=3 : 0.998337, loss: 0.001215, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10689 s, avg_samples: 33.6, ips: 314.32730 samples/s, eta: 1:35:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:14:56] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 06:15:04] ppocr INFO: epoch: [490/500], global_step: 517900, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.947586, exp_rate<=1 : 0.980355, exp_rate<=2 : 0.996259, exp_rate<=3 : 0.999326, loss: 0.001157, avg_reader_cost: 0.02594 s, avg_batch_cost: 0.10697 s, avg_samples: 22.4, ips: 209.40513 samples/s, eta: 1:35:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:15:24] ppocr INFO: epoch: [490/500], global_step: 518000, lr: 0.000100, edit distance: 0.001078, exp_rate: 0.947861, exp_rate<=1 : 0.979859, exp_rate<=2 : 0.995891, exp_rate<=3 : 0.998433, loss: 0.001303, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19964 s, avg_samples: 56.0, ips: 280.50811 samples/s, eta: 1:35:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:15:45] ppocr INFO: epoch: [490/500], global_step: 518100, lr: 0.000100, edit distance: 0.001092, exp_rate: 0.945872, exp_rate<=1 : 0.978994, exp_rate<=2 : 0.995692, exp_rate<=3 : 0.998233, loss: 0.000712, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20671 s, avg_samples: 56.0, ips: 270.91106 samples/s, eta: 1:34:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:16:05] ppocr INFO: epoch: [490/500], global_step: 518200, lr: 0.000100, edit distance: 0.001080, exp_rate: 0.947488, exp_rate<=1 : 0.979326, exp_rate<=2 : 0.995387, exp_rate<=3 : 0.998185, loss: 0.001377, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20009 s, avg_samples: 56.0, ips: 279.87522 samples/s, eta: 1:34:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:16:24] ppocr INFO: epoch: [490/500], global_step: 518300, lr: 0.000100, edit distance: 0.001069, exp_rate: 0.948526, exp_rate<=1 : 0.980065, exp_rate<=2 : 0.995479, exp_rate<=3 : 0.998430, loss: 0.001044, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18835 s, avg_samples: 56.0, ips: 297.31488 samples/s, eta: 1:34:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:16:43] ppocr INFO: epoch: [490/500], global_step: 518400, lr: 0.000100, edit distance: 0.001072, exp_rate: 0.948799, exp_rate<=1 : 0.979888, exp_rate<=2 : 0.995355, exp_rate<=3 : 0.998317, loss: 0.001229, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18788 s, avg_samples: 56.0, ips: 298.05923 samples/s, eta: 1:33:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:17:02] ppocr INFO: epoch: [490/500], global_step: 518500, lr: 0.000100, edit distance: 0.001104, exp_rate: 0.949083, exp_rate<=1 : 0.980108, exp_rate<=2 : 0.995486, exp_rate<=3 : 0.998301, loss: 0.001342, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18946 s, avg_samples: 56.0, ips: 295.57194 samples/s, eta: 1:33:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:17:19] ppocr INFO: epoch: [490/500], global_step: 518600, lr: 0.000100, edit distance: 0.001132, exp_rate: 0.948788, exp_rate<=1 : 0.980386, exp_rate<=2 : 0.995600, exp_rate<=3 : 0.998245, loss: 0.001205, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17321 s, avg_samples: 56.0, ips: 323.30746 samples/s, eta: 1:33:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:17:37] ppocr INFO: epoch: [490/500], global_step: 518700, lr: 0.000100, edit distance: 0.001174, exp_rate: 0.948022, exp_rate<=1 : 0.979993, exp_rate<=2 : 0.995412, exp_rate<=3 : 0.998258, loss: 0.001318, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18458 s, avg_samples: 56.0, ips: 303.38516 samples/s, eta: 1:32:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:17:57] ppocr INFO: epoch: [490/500], global_step: 518800, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.947925, exp_rate<=1 : 0.980061, exp_rate<=2 : 0.995552, exp_rate<=3 : 0.998316, loss: 0.001185, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19416 s, avg_samples: 56.0, ips: 288.41584 samples/s, eta: 1:32:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:18:16] ppocr INFO: epoch: [490/500], global_step: 518900, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.947988, exp_rate<=1 : 0.980042, exp_rate<=2 : 0.995505, exp_rate<=3 : 0.998310, loss: 0.001434, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18745 s, avg_samples: 56.0, ips: 298.75319 samples/s, eta: 1:32:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:18:35] ppocr INFO: epoch: [490/500], global_step: 519000, lr: 0.000100, edit distance: 0.001168, exp_rate: 0.947387, exp_rate<=1 : 0.980041, exp_rate<=2 : 0.995516, exp_rate<=3 : 0.998340, loss: 0.000988, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19448 s, avg_samples: 56.0, ips: 287.94817 samples/s, eta: 1:31:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:18:54] ppocr INFO: epoch: [490/500], global_step: 519100, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.946763, exp_rate<=1 : 0.979921, exp_rate<=2 : 0.995518, exp_rate<=3 : 0.998286, loss: 0.001567, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18677 s, avg_samples: 56.0, ips: 299.83224 samples/s, eta: 1:31:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:19:12] ppocr INFO: epoch: [490/500], global_step: 519200, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.946723, exp_rate<=1 : 0.980031, exp_rate<=2 : 0.995547, exp_rate<=3 : 0.998350, loss: 0.001158, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17785 s, avg_samples: 56.0, ips: 314.87932 samples/s, eta: 1:31:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:19:30] ppocr INFO: epoch: [490/500], global_step: 519300, lr: 0.000100, edit distance: 0.001206, exp_rate: 0.947290, exp_rate<=1 : 0.980265, exp_rate<=2 : 0.995589, exp_rate<=3 : 0.998391, loss: 0.001604, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18620 s, avg_samples: 56.0, ips: 300.75596 samples/s, eta: 1:30:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:19:49] ppocr INFO: epoch: [490/500], global_step: 519400, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.947154, exp_rate<=1 : 0.980237, exp_rate<=2 : 0.995707, exp_rate<=3 : 0.998392, loss: 0.001391, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18922 s, avg_samples: 56.0, ips: 295.95524 samples/s, eta: 1:30:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:20:08] ppocr INFO: epoch: [490/500], global_step: 519500, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.947058, exp_rate<=1 : 0.980176, exp_rate<=2 : 0.995668, exp_rate<=3 : 0.998375, loss: 0.001626, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19166 s, avg_samples: 56.0, ips: 292.19127 samples/s, eta: 1:30:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:20:28] ppocr INFO: epoch: [490/500], global_step: 519600, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.946579, exp_rate<=1 : 0.979955, exp_rate<=2 : 0.995611, exp_rate<=3 : 0.998386, loss: 0.001406, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19264 s, avg_samples: 56.0, ips: 290.69140 samples/s, eta: 1:29:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:20:45] ppocr INFO: epoch: [490/500], global_step: 519700, lr: 0.000100, edit distance: 0.001193, exp_rate: 0.946862, exp_rate<=1 : 0.979991, exp_rate<=2 : 0.995652, exp_rate<=3 : 0.998408, loss: 0.001051, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16962 s, avg_samples: 56.0, ips: 330.15220 samples/s, eta: 1:29:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:21:02] ppocr INFO: epoch: [490/500], global_step: 519800, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.946947, exp_rate<=1 : 0.980026, exp_rate<=2 : 0.995664, exp_rate<=3 : 0.998422, loss: 0.001470, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17078 s, avg_samples: 56.0, ips: 327.90428 samples/s, eta: 1:29:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:21:19] ppocr INFO: epoch: [490/500], global_step: 519900, lr: 0.000100, edit distance: 0.001202, exp_rate: 0.947043, exp_rate<=1 : 0.980101, exp_rate<=2 : 0.995682, exp_rate<=3 : 0.998435, loss: 0.001004, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17594 s, avg_samples: 56.0, ips: 318.28752 samples/s, eta: 1:29:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:21:39] ppocr INFO: epoch: [490/500], global_step: 520000, lr: 0.000100, edit distance: 0.001201, exp_rate: 0.946500, exp_rate<=1 : 0.979916, exp_rate<=2 : 0.995639, exp_rate<=3 : 0.998417, loss: 0.001636, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19386 s, avg_samples: 56.0, ips: 288.86609 samples/s, eta: 1:28:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:21:58] ppocr INFO: epoch: [490/500], global_step: 520100, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.946686, exp_rate<=1 : 0.980009, exp_rate<=2 : 0.995687, exp_rate<=3 : 0.998482, loss: 0.001654, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18922 s, avg_samples: 56.0, ips: 295.94908 samples/s, eta: 1:28:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:22:16] ppocr INFO: epoch: [490/500], global_step: 520200, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.946486, exp_rate<=1 : 0.979970, exp_rate<=2 : 0.995708, exp_rate<=3 : 0.998481, loss: 0.001676, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17862 s, avg_samples: 56.0, ips: 313.51115 samples/s, eta: 1:28:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:22:34] ppocr INFO: epoch: [490/500], global_step: 520300, lr: 0.000100, edit distance: 0.001204, exp_rate: 0.946065, exp_rate<=1 : 0.979771, exp_rate<=2 : 0.995667, exp_rate<=3 : 0.998472, loss: 0.000962, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17978 s, avg_samples: 56.0, ips: 311.49525 samples/s, eta: 1:27:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:22:53] ppocr INFO: epoch: [490/500], global_step: 520400, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.945606, exp_rate<=1 : 0.979540, exp_rate<=2 : 0.995589, exp_rate<=3 : 0.998433, loss: 0.001677, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19673 s, avg_samples: 56.0, ips: 284.66090 samples/s, eta: 1:27:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:23:13] ppocr INFO: epoch: [490/500], global_step: 520500, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.944779, exp_rate<=1 : 0.979400, exp_rate<=2 : 0.995528, exp_rate<=3 : 0.998417, loss: 0.001986, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20035 s, avg_samples: 56.0, ips: 279.50894 samples/s, eta: 1:27:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:23:33] ppocr INFO: epoch: [490/500], global_step: 520600, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.944415, exp_rate<=1 : 0.979258, exp_rate<=2 : 0.995449, exp_rate<=3 : 0.998411, loss: 0.001905, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19351 s, avg_samples: 56.0, ips: 289.39448 samples/s, eta: 1:26:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:23:35] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 06:23:36] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_490 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 06:23:56] ppocr INFO: epoch: [491/500], global_step: 520700, lr: 0.000100, edit distance: 0.001424, exp_rate: 0.933801, exp_rate<=1 : 0.973955, exp_rate<=2 : 0.992838, exp_rate<=3 : 0.997216, loss: 0.001124, avg_reader_cost: 0.03370 s, avg_batch_cost: 0.23002 s, avg_samples: 56.0, ips: 243.46070 samples/s, eta: 1:26:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:24:13] ppocr INFO: epoch: [491/500], global_step: 520800, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.944314, exp_rate<=1 : 0.977831, exp_rate<=2 : 0.994770, exp_rate<=3 : 0.997880, loss: 0.001325, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17684 s, avg_samples: 56.0, ips: 316.67712 samples/s, eta: 1:26:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:24:32] ppocr INFO: epoch: [491/500], global_step: 520900, lr: 0.000100, edit distance: 0.001339, exp_rate: 0.944132, exp_rate<=1 : 0.977941, exp_rate<=2 : 0.994689, exp_rate<=3 : 0.997912, loss: 0.001318, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18331 s, avg_samples: 56.0, ips: 305.49361 samples/s, eta: 1:25:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:24:51] ppocr INFO: epoch: [491/500], global_step: 521000, lr: 0.000100, edit distance: 0.001309, exp_rate: 0.944705, exp_rate<=1 : 0.978636, exp_rate<=2 : 0.994878, exp_rate<=3 : 0.998125, loss: 0.001261, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19455 s, avg_samples: 56.0, ips: 287.84618 samples/s, eta: 1:25:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:25:10] ppocr INFO: epoch: [491/500], global_step: 521100, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.946170, exp_rate<=1 : 0.979750, exp_rate<=2 : 0.995287, exp_rate<=3 : 0.998328, loss: 0.001561, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18567 s, avg_samples: 56.0, ips: 301.60486 samples/s, eta: 1:25:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:25:30] ppocr INFO: epoch: [491/500], global_step: 521200, lr: 0.000100, edit distance: 0.001279, exp_rate: 0.945410, exp_rate<=1 : 0.979623, exp_rate<=2 : 0.995120, exp_rate<=3 : 0.998305, loss: 0.001590, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20231 s, avg_samples: 56.0, ips: 276.79954 samples/s, eta: 1:24:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:25:50] ppocr INFO: epoch: [491/500], global_step: 521300, lr: 0.000100, edit distance: 0.001311, exp_rate: 0.943674, exp_rate<=1 : 0.979096, exp_rate<=2 : 0.995164, exp_rate<=3 : 0.998267, loss: 0.001405, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19915 s, avg_samples: 56.0, ips: 281.19793 samples/s, eta: 1:24:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:26:10] ppocr INFO: epoch: [491/500], global_step: 521400, lr: 0.000100, edit distance: 0.001314, exp_rate: 0.942370, exp_rate<=1 : 0.978119, exp_rate<=2 : 0.994827, exp_rate<=3 : 0.998080, loss: 0.001163, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19459 s, avg_samples: 56.0, ips: 287.79000 samples/s, eta: 1:24:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:26:28] ppocr INFO: epoch: [491/500], global_step: 521500, lr: 0.000100, edit distance: 0.001292, exp_rate: 0.943280, exp_rate<=1 : 0.978577, exp_rate<=2 : 0.994967, exp_rate<=3 : 0.998096, loss: 0.000768, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18591 s, avg_samples: 56.0, ips: 301.22114 samples/s, eta: 1:23:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:26:47] ppocr INFO: epoch: [491/500], global_step: 521600, lr: 0.000100, edit distance: 0.001253, exp_rate: 0.944159, exp_rate<=1 : 0.978629, exp_rate<=2 : 0.995114, exp_rate<=3 : 0.998143, loss: 0.001352, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19160 s, avg_samples: 56.0, ips: 292.28308 samples/s, eta: 1:23:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:27:08] ppocr INFO: epoch: [491/500], global_step: 521700, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.944828, exp_rate<=1 : 0.978736, exp_rate<=2 : 0.995021, exp_rate<=3 : 0.998116, loss: 0.000901, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20478 s, avg_samples: 56.0, ips: 273.46254 samples/s, eta: 1:23:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:27:26] ppocr INFO: epoch: [491/500], global_step: 521800, lr: 0.000100, edit distance: 0.001222, exp_rate: 0.945506, exp_rate<=1 : 0.978910, exp_rate<=2 : 0.995050, exp_rate<=3 : 0.998125, loss: 0.001145, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17988 s, avg_samples: 56.0, ips: 311.31844 samples/s, eta: 1:23:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:27:45] ppocr INFO: epoch: [491/500], global_step: 521900, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.945744, exp_rate<=1 : 0.978988, exp_rate<=2 : 0.995069, exp_rate<=3 : 0.998132, loss: 0.001257, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19389 s, avg_samples: 56.0, ips: 288.83097 samples/s, eta: 1:22:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:28:04] ppocr INFO: epoch: [491/500], global_step: 522000, lr: 0.000100, edit distance: 0.001199, exp_rate: 0.945736, exp_rate<=1 : 0.979089, exp_rate<=2 : 0.995171, exp_rate<=3 : 0.998228, loss: 0.001094, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19115 s, avg_samples: 56.0, ips: 292.95796 samples/s, eta: 1:22:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:28:23] ppocr INFO: epoch: [491/500], global_step: 522100, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.945359, exp_rate<=1 : 0.978866, exp_rate<=2 : 0.995191, exp_rate<=3 : 0.998245, loss: 0.001800, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19011 s, avg_samples: 56.0, ips: 294.56052 samples/s, eta: 1:22:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:28:42] ppocr INFO: epoch: [491/500], global_step: 522200, lr: 0.000100, edit distance: 0.001209, exp_rate: 0.945059, exp_rate<=1 : 0.978875, exp_rate<=2 : 0.995159, exp_rate<=3 : 0.998263, loss: 0.001324, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18693 s, avg_samples: 56.0, ips: 299.57113 samples/s, eta: 1:21:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:29:02] ppocr INFO: epoch: [491/500], global_step: 522300, lr: 0.000100, edit distance: 0.001211, exp_rate: 0.944770, exp_rate<=1 : 0.978850, exp_rate<=2 : 0.995223, exp_rate<=3 : 0.998255, loss: 0.001475, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19476 s, avg_samples: 56.0, ips: 287.53848 samples/s, eta: 1:21:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:29:21] ppocr INFO: epoch: [491/500], global_step: 522400, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.944845, exp_rate<=1 : 0.978827, exp_rate<=2 : 0.995160, exp_rate<=3 : 0.998239, loss: 0.001262, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19281 s, avg_samples: 56.0, ips: 290.44527 samples/s, eta: 1:21:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:29:39] ppocr INFO: epoch: [491/500], global_step: 522500, lr: 0.000100, edit distance: 0.001216, exp_rate: 0.944946, exp_rate<=1 : 0.978863, exp_rate<=2 : 0.995183, exp_rate<=3 : 0.998224, loss: 0.000786, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17884 s, avg_samples: 56.0, ips: 313.12507 samples/s, eta: 1:20:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:29:57] ppocr INFO: epoch: [491/500], global_step: 522600, lr: 0.000100, edit distance: 0.001213, exp_rate: 0.945204, exp_rate<=1 : 0.978963, exp_rate<=2 : 0.995246, exp_rate<=3 : 0.998251, loss: 0.001449, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18282 s, avg_samples: 56.0, ips: 306.30806 samples/s, eta: 1:20:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:30:16] ppocr INFO: epoch: [491/500], global_step: 522700, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.944787, exp_rate<=1 : 0.978770, exp_rate<=2 : 0.995183, exp_rate<=3 : 0.998190, loss: 0.001648, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19359 s, avg_samples: 56.0, ips: 289.26451 samples/s, eta: 1:20:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:30:35] ppocr INFO: epoch: [491/500], global_step: 522800, lr: 0.000100, edit distance: 0.001225, exp_rate: 0.944271, exp_rate<=1 : 0.978620, exp_rate<=2 : 0.995166, exp_rate<=3 : 0.998124, loss: 0.001839, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18140 s, avg_samples: 56.0, ips: 308.70777 samples/s, eta: 1:19:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:30:53] ppocr INFO: epoch: [491/500], global_step: 522900, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.944154, exp_rate<=1 : 0.978575, exp_rate<=2 : 0.995130, exp_rate<=3 : 0.998095, loss: 0.001982, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18022 s, avg_samples: 56.0, ips: 310.72645 samples/s, eta: 1:19:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:31:11] ppocr INFO: epoch: [491/500], global_step: 523000, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.943668, exp_rate<=1 : 0.978403, exp_rate<=2 : 0.995094, exp_rate<=3 : 0.998078, loss: 0.001354, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18712 s, avg_samples: 56.0, ips: 299.27117 samples/s, eta: 1:19:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:31:30] ppocr INFO: epoch: [491/500], global_step: 523100, lr: 0.000100, edit distance: 0.001239, exp_rate: 0.943655, exp_rate<=1 : 0.978404, exp_rate<=2 : 0.995054, exp_rate<=3 : 0.998082, loss: 0.001518, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18534 s, avg_samples: 56.0, ips: 302.13968 samples/s, eta: 1:18:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:31:48] ppocr INFO: epoch: [491/500], global_step: 523200, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.943570, exp_rate<=1 : 0.978394, exp_rate<=2 : 0.995034, exp_rate<=3 : 0.998057, loss: 0.002060, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18281 s, avg_samples: 56.0, ips: 306.33070 samples/s, eta: 1:18:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:32:06] ppocr INFO: epoch: [491/500], global_step: 523300, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.943601, exp_rate<=1 : 0.978454, exp_rate<=2 : 0.995089, exp_rate<=3 : 0.998082, loss: 0.001552, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17944 s, avg_samples: 56.0, ips: 312.07425 samples/s, eta: 1:18:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:32:13] ppocr INFO: epoch: [491/500], global_step: 523340, lr: 0.000100, edit distance: 0.001246, exp_rate: 0.943623, exp_rate<=1 : 0.978488, exp_rate<=2 : 0.995083, exp_rate<=3 : 0.998095, loss: 0.001134, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07019 s, avg_samples: 22.4, ips: 319.13771 samples/s, eta: 1:18:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:32:15] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 06:32:27] ppocr INFO: epoch: [492/500], global_step: 523400, lr: 0.000100, edit distance: 0.001084, exp_rate: 0.952704, exp_rate<=1 : 0.979074, exp_rate<=2 : 0.995069, exp_rate<=3 : 0.998639, loss: 0.000958, avg_reader_cost: 0.02521 s, avg_batch_cost: 0.13478 s, avg_samples: 33.6, ips: 249.29613 samples/s, eta: 1:17:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:32:46] ppocr INFO: epoch: [492/500], global_step: 523500, lr: 0.000100, edit distance: 0.001129, exp_rate: 0.949780, exp_rate<=1 : 0.979940, exp_rate<=2 : 0.994787, exp_rate<=3 : 0.998285, loss: 0.000955, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19272 s, avg_samples: 56.0, ips: 290.57330 samples/s, eta: 1:17:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:33:06] ppocr INFO: epoch: [492/500], global_step: 523600, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.948163, exp_rate<=1 : 0.980224, exp_rate<=2 : 0.995214, exp_rate<=3 : 0.998306, loss: 0.001213, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19468 s, avg_samples: 56.0, ips: 287.64836 samples/s, eta: 1:17:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:33:24] ppocr INFO: epoch: [492/500], global_step: 523700, lr: 0.000100, edit distance: 0.001160, exp_rate: 0.949182, exp_rate<=1 : 0.980536, exp_rate<=2 : 0.994814, exp_rate<=3 : 0.998122, loss: 0.001453, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18279 s, avg_samples: 56.0, ips: 306.36881 samples/s, eta: 1:16:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:33:43] ppocr INFO: epoch: [492/500], global_step: 523800, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.948852, exp_rate<=1 : 0.980603, exp_rate<=2 : 0.994935, exp_rate<=3 : 0.998260, loss: 0.001219, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19187 s, avg_samples: 56.0, ips: 291.86348 samples/s, eta: 1:16:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:34:02] ppocr INFO: epoch: [492/500], global_step: 523900, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.948691, exp_rate<=1 : 0.980870, exp_rate<=2 : 0.995249, exp_rate<=3 : 0.998346, loss: 0.001415, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18820 s, avg_samples: 56.0, ips: 297.55188 samples/s, eta: 1:16:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:34:21] ppocr INFO: epoch: [492/500], global_step: 524000, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.947989, exp_rate<=1 : 0.980601, exp_rate<=2 : 0.995126, exp_rate<=3 : 0.998293, loss: 0.001271, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19383 s, avg_samples: 56.0, ips: 288.90726 samples/s, eta: 1:16:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:34:41] ppocr INFO: epoch: [492/500], global_step: 524100, lr: 0.000100, edit distance: 0.001126, exp_rate: 0.947283, exp_rate<=1 : 0.980192, exp_rate<=2 : 0.995007, exp_rate<=3 : 0.998163, loss: 0.002045, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19601 s, avg_samples: 56.0, ips: 285.70523 samples/s, eta: 1:15:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:35:00] ppocr INFO: epoch: [492/500], global_step: 524200, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.947887, exp_rate<=1 : 0.980457, exp_rate<=2 : 0.995044, exp_rate<=3 : 0.998177, loss: 0.000915, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19260 s, avg_samples: 56.0, ips: 290.75816 samples/s, eta: 1:15:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:35:18] ppocr INFO: epoch: [492/500], global_step: 524300, lr: 0.000100, edit distance: 0.001127, exp_rate: 0.948123, exp_rate<=1 : 0.980569, exp_rate<=2 : 0.995119, exp_rate<=3 : 0.998233, loss: 0.001236, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18117 s, avg_samples: 56.0, ips: 309.09825 samples/s, eta: 1:15:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:35:38] ppocr INFO: epoch: [492/500], global_step: 524400, lr: 0.000100, edit distance: 0.001113, exp_rate: 0.948306, exp_rate<=1 : 0.980572, exp_rate<=2 : 0.995180, exp_rate<=3 : 0.998343, loss: 0.001218, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19642 s, avg_samples: 56.0, ips: 285.10878 samples/s, eta: 1:14:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:35:57] ppocr INFO: epoch: [492/500], global_step: 524500, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.948118, exp_rate<=1 : 0.980468, exp_rate<=2 : 0.995106, exp_rate<=3 : 0.998294, loss: 0.001192, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18624 s, avg_samples: 56.0, ips: 300.69262 samples/s, eta: 1:14:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:36:15] ppocr INFO: epoch: [492/500], global_step: 524600, lr: 0.000100, edit distance: 0.001117, exp_rate: 0.948220, exp_rate<=1 : 0.980486, exp_rate<=2 : 0.995202, exp_rate<=3 : 0.998349, loss: 0.001514, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18150 s, avg_samples: 56.0, ips: 308.53543 samples/s, eta: 1:14:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:36:33] ppocr INFO: epoch: [492/500], global_step: 524700, lr: 0.000100, edit distance: 0.001115, exp_rate: 0.948040, exp_rate<=1 : 0.980411, exp_rate<=2 : 0.995310, exp_rate<=3 : 0.998407, loss: 0.000955, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18354 s, avg_samples: 56.0, ips: 305.11504 samples/s, eta: 1:13:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:36:52] ppocr INFO: epoch: [492/500], global_step: 524800, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.948067, exp_rate<=1 : 0.980382, exp_rate<=2 : 0.995335, exp_rate<=3 : 0.998395, loss: 0.001421, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18623 s, avg_samples: 56.0, ips: 300.71076 samples/s, eta: 1:13:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:37:11] ppocr INFO: epoch: [492/500], global_step: 524900, lr: 0.000100, edit distance: 0.001125, exp_rate: 0.947810, exp_rate<=1 : 0.980233, exp_rate<=2 : 0.995340, exp_rate<=3 : 0.998445, loss: 0.001408, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19345 s, avg_samples: 56.0, ips: 289.48752 samples/s, eta: 1:13:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:37:31] ppocr INFO: epoch: [492/500], global_step: 525000, lr: 0.000100, edit distance: 0.001135, exp_rate: 0.947279, exp_rate<=1 : 0.979983, exp_rate<=2 : 0.995268, exp_rate<=3 : 0.998471, loss: 0.001460, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19777 s, avg_samples: 56.0, ips: 283.15772 samples/s, eta: 1:12:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:37:49] ppocr INFO: epoch: [492/500], global_step: 525100, lr: 0.000100, edit distance: 0.001135, exp_rate: 0.947214, exp_rate<=1 : 0.979880, exp_rate<=2 : 0.995250, exp_rate<=3 : 0.998478, loss: 0.001264, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18330 s, avg_samples: 56.0, ips: 305.50246 samples/s, eta: 1:12:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:38:09] ppocr INFO: epoch: [492/500], global_step: 525200, lr: 0.000100, edit distance: 0.001135, exp_rate: 0.946829, exp_rate<=1 : 0.979446, exp_rate<=2 : 0.995057, exp_rate<=3 : 0.998384, loss: 0.000948, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19954 s, avg_samples: 56.0, ips: 280.64360 samples/s, eta: 1:12:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:38:27] ppocr INFO: epoch: [492/500], global_step: 525300, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.946978, exp_rate<=1 : 0.979458, exp_rate<=2 : 0.995055, exp_rate<=3 : 0.998394, loss: 0.001645, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18086 s, avg_samples: 56.0, ips: 309.63891 samples/s, eta: 1:11:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:38:45] ppocr INFO: epoch: [492/500], global_step: 525400, lr: 0.000100, edit distance: 0.001128, exp_rate: 0.947026, exp_rate<=1 : 0.979460, exp_rate<=2 : 0.995017, exp_rate<=3 : 0.998363, loss: 0.001474, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17280 s, avg_samples: 56.0, ips: 324.08043 samples/s, eta: 1:11:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:39:04] ppocr INFO: epoch: [492/500], global_step: 525500, lr: 0.000100, edit distance: 0.001136, exp_rate: 0.946445, exp_rate<=1 : 0.979289, exp_rate<=2 : 0.994981, exp_rate<=3 : 0.998355, loss: 0.001515, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19491 s, avg_samples: 56.0, ips: 287.31447 samples/s, eta: 1:11:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:39:24] ppocr INFO: epoch: [492/500], global_step: 525600, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.945785, exp_rate<=1 : 0.979031, exp_rate<=2 : 0.994931, exp_rate<=3 : 0.998335, loss: 0.001572, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19842 s, avg_samples: 56.0, ips: 282.23630 samples/s, eta: 1:10:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:39:42] ppocr INFO: epoch: [492/500], global_step: 525700, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.945762, exp_rate<=1 : 0.979096, exp_rate<=2 : 0.995007, exp_rate<=3 : 0.998358, loss: 0.001588, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18185 s, avg_samples: 56.0, ips: 307.95045 samples/s, eta: 1:10:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:40:01] ppocr INFO: epoch: [492/500], global_step: 525800, lr: 0.000100, edit distance: 0.001175, exp_rate: 0.945337, exp_rate<=1 : 0.978976, exp_rate<=2 : 0.995033, exp_rate<=3 : 0.998345, loss: 0.001436, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18176 s, avg_samples: 56.0, ips: 308.09534 samples/s, eta: 1:10:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:40:20] ppocr INFO: epoch: [492/500], global_step: 525900, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.944185, exp_rate<=1 : 0.978363, exp_rate<=2 : 0.994767, exp_rate<=3 : 0.998197, loss: 0.001391, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18979 s, avg_samples: 56.0, ips: 295.06026 samples/s, eta: 1:10:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:40:38] ppocr INFO: epoch: [492/500], global_step: 526000, lr: 0.000100, edit distance: 0.001200, exp_rate: 0.943912, exp_rate<=1 : 0.978237, exp_rate<=2 : 0.994759, exp_rate<=3 : 0.998196, loss: 0.001422, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18651 s, avg_samples: 56.0, ips: 300.25032 samples/s, eta: 1:09:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:40:53] ppocr INFO: epoch: [492/500], global_step: 526080, lr: 0.000100, edit distance: 0.001196, exp_rate: 0.944059, exp_rate<=1 : 0.978331, exp_rate<=2 : 0.994822, exp_rate<=3 : 0.998223, loss: 0.001150, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.14540 s, avg_samples: 44.8, ips: 308.10579 samples/s, eta: 1:09:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:40:55] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 06:41:00] ppocr INFO: epoch: [493/500], global_step: 526100, lr: 0.000100, edit distance: 0.001059, exp_rate: 0.916189, exp_rate<=1 : 0.976190, exp_rate<=2 : 0.996230, exp_rate<=3 : 1.000000, loss: 0.001421, avg_reader_cost: 0.02588 s, avg_batch_cost: 0.07354 s, avg_samples: 11.2, ips: 152.30288 samples/s, eta: 1:09:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:41:20] ppocr INFO: epoch: [493/500], global_step: 526200, lr: 0.000100, edit distance: 0.001117, exp_rate: 0.941069, exp_rate<=1 : 0.978138, exp_rate<=2 : 0.995026, exp_rate<=3 : 0.997715, loss: 0.001133, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19437 s, avg_samples: 56.0, ips: 288.10608 samples/s, eta: 1:09:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:41:39] ppocr INFO: epoch: [493/500], global_step: 526300, lr: 0.000100, edit distance: 0.001005, exp_rate: 0.945174, exp_rate<=1 : 0.979379, exp_rate<=2 : 0.994940, exp_rate<=3 : 0.998342, loss: 0.000786, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18926 s, avg_samples: 56.0, ips: 295.88585 samples/s, eta: 1:08:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:41:57] ppocr INFO: epoch: [493/500], global_step: 526400, lr: 0.000100, edit distance: 0.000953, exp_rate: 0.949084, exp_rate<=1 : 0.981273, exp_rate<=2 : 0.995514, exp_rate<=3 : 0.998505, loss: 0.000921, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18386 s, avg_samples: 56.0, ips: 304.57201 samples/s, eta: 1:08:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:42:13] ppocr INFO: epoch: [493/500], global_step: 526500, lr: 0.000100, edit distance: 0.000910, exp_rate: 0.953281, exp_rate<=1 : 0.982996, exp_rate<=2 : 0.996049, exp_rate<=3 : 0.998814, loss: 0.000998, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16527 s, avg_samples: 56.0, ips: 338.84913 samples/s, eta: 1:08:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:42:31] ppocr INFO: epoch: [493/500], global_step: 526600, lr: 0.000100, edit distance: 0.000899, exp_rate: 0.954037, exp_rate<=1 : 0.983246, exp_rate<=2 : 0.996327, exp_rate<=3 : 0.998986, loss: 0.000666, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17152 s, avg_samples: 56.0, ips: 326.48583 samples/s, eta: 1:07:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:42:49] ppocr INFO: epoch: [493/500], global_step: 526700, lr: 0.000100, edit distance: 0.000912, exp_rate: 0.954434, exp_rate<=1 : 0.983499, exp_rate<=2 : 0.996494, exp_rate<=3 : 0.999010, loss: 0.001729, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18320 s, avg_samples: 56.0, ips: 305.66906 samples/s, eta: 1:07:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:43:08] ppocr INFO: epoch: [493/500], global_step: 526800, lr: 0.000100, edit distance: 0.000985, exp_rate: 0.951618, exp_rate<=1 : 0.982567, exp_rate<=2 : 0.996255, exp_rate<=3 : 0.998819, loss: 0.001180, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19290 s, avg_samples: 56.0, ips: 290.30508 samples/s, eta: 1:07:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:43:26] ppocr INFO: epoch: [493/500], global_step: 526900, lr: 0.000100, edit distance: 0.000997, exp_rate: 0.951672, exp_rate<=1 : 0.982562, exp_rate<=2 : 0.996360, exp_rate<=3 : 0.998788, loss: 0.001442, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17592 s, avg_samples: 56.0, ips: 318.32936 samples/s, eta: 1:06:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:43:43] ppocr INFO: epoch: [493/500], global_step: 527000, lr: 0.000100, edit distance: 0.001020, exp_rate: 0.951738, exp_rate<=1 : 0.982603, exp_rate<=2 : 0.996450, exp_rate<=3 : 0.998804, loss: 0.001470, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16969 s, avg_samples: 56.0, ips: 330.02024 samples/s, eta: 1:06:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:44:02] ppocr INFO: epoch: [493/500], global_step: 527100, lr: 0.000100, edit distance: 0.001004, exp_rate: 0.951034, exp_rate<=1 : 0.982134, exp_rate<=2 : 0.996253, exp_rate<=3 : 0.998734, loss: 0.001218, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19511 s, avg_samples: 56.0, ips: 287.01781 samples/s, eta: 1:06:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:44:21] ppocr INFO: epoch: [493/500], global_step: 527200, lr: 0.000100, edit distance: 0.001003, exp_rate: 0.951191, exp_rate<=1 : 0.981982, exp_rate<=2 : 0.996274, exp_rate<=3 : 0.998687, loss: 0.001131, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18984 s, avg_samples: 56.0, ips: 294.98788 samples/s, eta: 1:05:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:44:41] ppocr INFO: epoch: [493/500], global_step: 527300, lr: 0.000100, edit distance: 0.001032, exp_rate: 0.950255, exp_rate<=1 : 0.981620, exp_rate<=2 : 0.996227, exp_rate<=3 : 0.998737, loss: 0.001476, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19565 s, avg_samples: 56.0, ips: 286.22038 samples/s, eta: 1:05:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:45:00] ppocr INFO: epoch: [493/500], global_step: 527400, lr: 0.000100, edit distance: 0.001044, exp_rate: 0.950221, exp_rate<=1 : 0.981979, exp_rate<=2 : 0.996280, exp_rate<=3 : 0.998774, loss: 0.001344, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18615 s, avg_samples: 56.0, ips: 300.83078 samples/s, eta: 1:05:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:45:19] ppocr INFO: epoch: [493/500], global_step: 527500, lr: 0.000100, edit distance: 0.001043, exp_rate: 0.950029, exp_rate<=1 : 0.981858, exp_rate<=2 : 0.996267, exp_rate<=3 : 0.998793, loss: 0.001022, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19038 s, avg_samples: 56.0, ips: 294.15104 samples/s, eta: 1:04:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:45:36] ppocr INFO: epoch: [493/500], global_step: 527600, lr: 0.000100, edit distance: 0.001068, exp_rate: 0.949867, exp_rate<=1 : 0.981787, exp_rate<=2 : 0.996233, exp_rate<=3 : 0.998783, loss: 0.001795, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17411 s, avg_samples: 56.0, ips: 321.62850 samples/s, eta: 1:04:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:45:56] ppocr INFO: epoch: [493/500], global_step: 527700, lr: 0.000100, edit distance: 0.001103, exp_rate: 0.948613, exp_rate<=1 : 0.981054, exp_rate<=2 : 0.996024, exp_rate<=3 : 0.998692, loss: 0.001590, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19568 s, avg_samples: 56.0, ips: 286.18453 samples/s, eta: 1:04:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:46:14] ppocr INFO: epoch: [493/500], global_step: 527800, lr: 0.000100, edit distance: 0.001117, exp_rate: 0.947789, exp_rate<=1 : 0.980823, exp_rate<=2 : 0.995978, exp_rate<=3 : 0.998695, loss: 0.001560, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17930 s, avg_samples: 56.0, ips: 312.32543 samples/s, eta: 1:03:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:46:33] ppocr INFO: epoch: [493/500], global_step: 527900, lr: 0.000100, edit distance: 0.001117, exp_rate: 0.947579, exp_rate<=1 : 0.980608, exp_rate<=2 : 0.995901, exp_rate<=3 : 0.998693, loss: 0.001228, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19726 s, avg_samples: 56.0, ips: 283.88419 samples/s, eta: 1:03:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:46:52] ppocr INFO: epoch: [493/500], global_step: 528000, lr: 0.000100, edit distance: 0.001119, exp_rate: 0.947494, exp_rate<=1 : 0.980626, exp_rate<=2 : 0.995896, exp_rate<=3 : 0.998689, loss: 0.000905, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18962 s, avg_samples: 56.0, ips: 295.32885 samples/s, eta: 1:03:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:47:11] ppocr INFO: epoch: [493/500], global_step: 528100, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.947087, exp_rate<=1 : 0.980347, exp_rate<=2 : 0.995753, exp_rate<=3 : 0.998626, loss: 0.001008, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18841 s, avg_samples: 56.0, ips: 297.22541 samples/s, eta: 1:03:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:47:30] ppocr INFO: epoch: [493/500], global_step: 528200, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.946915, exp_rate<=1 : 0.980238, exp_rate<=2 : 0.995695, exp_rate<=3 : 0.998564, loss: 0.001246, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18552 s, avg_samples: 56.0, ips: 301.85186 samples/s, eta: 1:02:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:47:50] ppocr INFO: epoch: [493/500], global_step: 528300, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.946315, exp_rate<=1 : 0.979999, exp_rate<=2 : 0.995615, exp_rate<=3 : 0.998495, loss: 0.001283, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.20403 s, avg_samples: 56.0, ips: 274.46362 samples/s, eta: 1:02:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:48:09] ppocr INFO: epoch: [493/500], global_step: 528400, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.946073, exp_rate<=1 : 0.980026, exp_rate<=2 : 0.995569, exp_rate<=3 : 0.998431, loss: 0.001221, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19161 s, avg_samples: 56.0, ips: 292.26288 samples/s, eta: 1:02:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:48:28] ppocr INFO: epoch: [493/500], global_step: 528500, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.946229, exp_rate<=1 : 0.980098, exp_rate<=2 : 0.995548, exp_rate<=3 : 0.998437, loss: 0.000965, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18170 s, avg_samples: 56.0, ips: 308.19606 samples/s, eta: 1:01:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:48:47] ppocr INFO: epoch: [493/500], global_step: 528600, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.946091, exp_rate<=1 : 0.980113, exp_rate<=2 : 0.995560, exp_rate<=3 : 0.998476, loss: 0.000919, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19399 s, avg_samples: 56.0, ips: 288.67182 samples/s, eta: 1:01:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:49:06] ppocr INFO: epoch: [493/500], global_step: 528700, lr: 0.000100, edit distance: 0.001139, exp_rate: 0.946024, exp_rate<=1 : 0.980086, exp_rate<=2 : 0.995532, exp_rate<=3 : 0.998435, loss: 0.001761, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19301 s, avg_samples: 56.0, ips: 290.14662 samples/s, eta: 1:01:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:49:26] ppocr INFO: epoch: [493/500], global_step: 528800, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.945462, exp_rate<=1 : 0.979789, exp_rate<=2 : 0.995507, exp_rate<=3 : 0.998392, loss: 0.001679, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19234 s, avg_samples: 56.0, ips: 291.15484 samples/s, eta: 1:00:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:49:29] ppocr INFO: epoch: [493/500], global_step: 528820, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.945410, exp_rate<=1 : 0.979778, exp_rate<=2 : 0.995514, exp_rate<=3 : 0.998386, loss: 0.001611, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03693 s, avg_samples: 11.2, ips: 303.24739 samples/s, eta: 1:00:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:49:32] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 06:49:47] ppocr INFO: epoch: [494/500], global_step: 528900, lr: 0.000100, edit distance: 0.001016, exp_rate: 0.954476, exp_rate<=1 : 0.981750, exp_rate<=2 : 0.995156, exp_rate<=3 : 0.998942, loss: 0.001002, avg_reader_cost: 0.02822 s, avg_batch_cost: 0.17574 s, avg_samples: 44.8, ips: 254.92163 samples/s, eta: 1:00:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:50:07] ppocr INFO: epoch: [494/500], global_step: 529000, lr: 0.000100, edit distance: 0.001050, exp_rate: 0.948157, exp_rate<=1 : 0.979281, exp_rate<=2 : 0.995973, exp_rate<=3 : 0.998662, loss: 0.000992, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19891 s, avg_samples: 56.0, ips: 281.52952 samples/s, eta: 1:00:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:50:25] ppocr INFO: epoch: [494/500], global_step: 529100, lr: 0.000100, edit distance: 0.001023, exp_rate: 0.950013, exp_rate<=1 : 0.980082, exp_rate<=2 : 0.996369, exp_rate<=3 : 0.998944, loss: 0.000609, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17897 s, avg_samples: 56.0, ips: 312.90844 samples/s, eta: 0:59:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:50:43] ppocr INFO: epoch: [494/500], global_step: 529200, lr: 0.000100, edit distance: 0.001024, exp_rate: 0.949952, exp_rate<=1 : 0.979986, exp_rate<=2 : 0.996094, exp_rate<=3 : 0.998777, loss: 0.001122, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18542 s, avg_samples: 56.0, ips: 302.01017 samples/s, eta: 0:59:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:51:02] ppocr INFO: epoch: [494/500], global_step: 529300, lr: 0.000100, edit distance: 0.001011, exp_rate: 0.951052, exp_rate<=1 : 0.980795, exp_rate<=2 : 0.996360, exp_rate<=3 : 0.998884, loss: 0.000886, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18200 s, avg_samples: 56.0, ips: 307.69463 samples/s, eta: 0:59:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:51:21] ppocr INFO: epoch: [494/500], global_step: 529400, lr: 0.000100, edit distance: 0.001016, exp_rate: 0.950421, exp_rate<=1 : 0.980332, exp_rate<=2 : 0.996407, exp_rate<=3 : 0.998906, loss: 0.001040, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19209 s, avg_samples: 56.0, ips: 291.52382 samples/s, eta: 0:58:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:51:38] ppocr INFO: epoch: [494/500], global_step: 529500, lr: 0.000100, edit distance: 0.001036, exp_rate: 0.950133, exp_rate<=1 : 0.980330, exp_rate<=2 : 0.996548, exp_rate<=3 : 0.998905, loss: 0.001227, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17384 s, avg_samples: 56.0, ips: 322.14274 samples/s, eta: 0:58:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:51:58] ppocr INFO: epoch: [494/500], global_step: 529600, lr: 0.000100, edit distance: 0.001026, exp_rate: 0.949710, exp_rate<=1 : 0.979960, exp_rate<=2 : 0.996183, exp_rate<=3 : 0.998734, loss: 0.000630, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19385 s, avg_samples: 56.0, ips: 288.88248 samples/s, eta: 0:58:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:52:16] ppocr INFO: epoch: [494/500], global_step: 529700, lr: 0.000100, edit distance: 0.001032, exp_rate: 0.949614, exp_rate<=1 : 0.980230, exp_rate<=2 : 0.996099, exp_rate<=3 : 0.998667, loss: 0.001325, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18182 s, avg_samples: 56.0, ips: 307.99751 samples/s, eta: 0:57:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:52:34] ppocr INFO: epoch: [494/500], global_step: 529800, lr: 0.000100, edit distance: 0.001032, exp_rate: 0.949726, exp_rate<=1 : 0.980250, exp_rate<=2 : 0.996132, exp_rate<=3 : 0.998726, loss: 0.000917, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18490 s, avg_samples: 56.0, ips: 302.85893 samples/s, eta: 0:57:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:52:53] ppocr INFO: epoch: [494/500], global_step: 529900, lr: 0.000100, edit distance: 0.001032, exp_rate: 0.949867, exp_rate<=1 : 0.980335, exp_rate<=2 : 0.996248, exp_rate<=3 : 0.998753, loss: 0.001734, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18320 s, avg_samples: 56.0, ips: 305.68024 samples/s, eta: 0:57:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:53:13] ppocr INFO: epoch: [494/500], global_step: 530000, lr: 0.000100, edit distance: 0.001042, exp_rate: 0.948716, exp_rate<=1 : 0.979742, exp_rate<=2 : 0.995976, exp_rate<=3 : 0.998628, loss: 0.001408, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20117 s, avg_samples: 56.0, ips: 278.36878 samples/s, eta: 0:57:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:53:32] ppocr INFO: epoch: [494/500], global_step: 530100, lr: 0.000100, edit distance: 0.001071, exp_rate: 0.947651, exp_rate<=1 : 0.979469, exp_rate<=2 : 0.995865, exp_rate<=3 : 0.998575, loss: 0.001852, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18794 s, avg_samples: 56.0, ips: 297.96267 samples/s, eta: 0:56:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:53:50] ppocr INFO: epoch: [494/500], global_step: 530200, lr: 0.000100, edit distance: 0.001077, exp_rate: 0.946781, exp_rate<=1 : 0.979245, exp_rate<=2 : 0.995742, exp_rate<=3 : 0.998498, loss: 0.001544, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18766 s, avg_samples: 56.0, ips: 298.41117 samples/s, eta: 0:56:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:54:10] ppocr INFO: epoch: [494/500], global_step: 530300, lr: 0.000100, edit distance: 0.001100, exp_rate: 0.945992, exp_rate<=1 : 0.978965, exp_rate<=2 : 0.995619, exp_rate<=3 : 0.998452, loss: 0.001551, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19479 s, avg_samples: 56.0, ips: 287.48675 samples/s, eta: 0:56:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:54:29] ppocr INFO: epoch: [494/500], global_step: 530400, lr: 0.000100, edit distance: 0.001131, exp_rate: 0.945547, exp_rate<=1 : 0.978776, exp_rate<=2 : 0.995631, exp_rate<=3 : 0.998459, loss: 0.001784, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18974 s, avg_samples: 56.0, ips: 295.14139 samples/s, eta: 0:55:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:54:48] ppocr INFO: epoch: [494/500], global_step: 530500, lr: 0.000100, edit distance: 0.001141, exp_rate: 0.945066, exp_rate<=1 : 0.978625, exp_rate<=2 : 0.995610, exp_rate<=3 : 0.998456, loss: 0.001439, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19082 s, avg_samples: 56.0, ips: 293.46294 samples/s, eta: 0:55:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:55:06] ppocr INFO: epoch: [494/500], global_step: 530600, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.944991, exp_rate<=1 : 0.978723, exp_rate<=2 : 0.995625, exp_rate<=3 : 0.998486, loss: 0.001473, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18250 s, avg_samples: 56.0, ips: 306.84667 samples/s, eta: 0:55:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:55:24] ppocr INFO: epoch: [494/500], global_step: 530700, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.944937, exp_rate<=1 : 0.978725, exp_rate<=2 : 0.995666, exp_rate<=3 : 0.998511, loss: 0.001556, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18077 s, avg_samples: 56.0, ips: 309.78946 samples/s, eta: 0:54:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:55:43] ppocr INFO: epoch: [494/500], global_step: 530800, lr: 0.000100, edit distance: 0.001144, exp_rate: 0.945114, exp_rate<=1 : 0.978948, exp_rate<=2 : 0.995659, exp_rate<=3 : 0.998508, loss: 0.001048, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18591 s, avg_samples: 56.0, ips: 301.22824 samples/s, eta: 0:54:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:56:02] ppocr INFO: epoch: [494/500], global_step: 530900, lr: 0.000100, edit distance: 0.001149, exp_rate: 0.945061, exp_rate<=1 : 0.978893, exp_rate<=2 : 0.995662, exp_rate<=3 : 0.998512, loss: 0.001290, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18568 s, avg_samples: 56.0, ips: 301.59884 samples/s, eta: 0:54:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:56:21] ppocr INFO: epoch: [494/500], global_step: 531000, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.944482, exp_rate<=1 : 0.978566, exp_rate<=2 : 0.995522, exp_rate<=3 : 0.998462, loss: 0.001124, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19601 s, avg_samples: 56.0, ips: 285.70232 samples/s, eta: 0:53:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:56:38] ppocr INFO: epoch: [494/500], global_step: 531100, lr: 0.000100, edit distance: 0.001155, exp_rate: 0.944671, exp_rate<=1 : 0.978722, exp_rate<=2 : 0.995619, exp_rate<=3 : 0.998514, loss: 0.000862, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17240 s, avg_samples: 56.0, ips: 324.83504 samples/s, eta: 0:53:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:56:59] ppocr INFO: epoch: [494/500], global_step: 531200, lr: 0.000100, edit distance: 0.001151, exp_rate: 0.944566, exp_rate<=1 : 0.978712, exp_rate<=2 : 0.995609, exp_rate<=3 : 0.998467, loss: 0.000981, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20249 s, avg_samples: 56.0, ips: 276.55510 samples/s, eta: 0:53:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:57:18] ppocr INFO: epoch: [494/500], global_step: 531300, lr: 0.000100, edit distance: 0.001158, exp_rate: 0.944531, exp_rate<=1 : 0.978748, exp_rate<=2 : 0.995604, exp_rate<=3 : 0.998475, loss: 0.001540, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19392 s, avg_samples: 56.0, ips: 288.77462 samples/s, eta: 0:52:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:57:37] ppocr INFO: epoch: [494/500], global_step: 531400, lr: 0.000100, edit distance: 0.001161, exp_rate: 0.944390, exp_rate<=1 : 0.978724, exp_rate<=2 : 0.995546, exp_rate<=3 : 0.998444, loss: 0.001131, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18931 s, avg_samples: 56.0, ips: 295.81462 samples/s, eta: 0:52:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:57:55] ppocr INFO: epoch: [494/500], global_step: 531500, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.944530, exp_rate<=1 : 0.978842, exp_rate<=2 : 0.995558, exp_rate<=3 : 0.998450, loss: 0.001616, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18208 s, avg_samples: 56.0, ips: 307.55111 samples/s, eta: 0:52:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:58:07] ppocr INFO: epoch: [494/500], global_step: 531560, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.944112, exp_rate<=1 : 0.978701, exp_rate<=2 : 0.995527, exp_rate<=3 : 0.998406, loss: 0.001191, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.11775 s, avg_samples: 33.6, ips: 285.33919 samples/s, eta: 0:52:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:58:09] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 06:58:18] ppocr INFO: epoch: [495/500], global_step: 531600, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.948237, exp_rate<=1 : 0.979321, exp_rate<=2 : 0.995287, exp_rate<=3 : 0.997739, loss: 0.001393, avg_reader_cost: 0.02780 s, avg_batch_cost: 0.10600 s, avg_samples: 22.4, ips: 211.31350 samples/s, eta: 0:51:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:58:38] ppocr INFO: epoch: [495/500], global_step: 531700, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.943324, exp_rate<=1 : 0.976525, exp_rate<=2 : 0.995884, exp_rate<=3 : 0.998084, loss: 0.001446, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20508 s, avg_samples: 56.0, ips: 273.06423 samples/s, eta: 0:51:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:58:56] ppocr INFO: epoch: [495/500], global_step: 531800, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.945298, exp_rate<=1 : 0.979378, exp_rate<=2 : 0.996560, exp_rate<=3 : 0.998551, loss: 0.001372, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18049 s, avg_samples: 56.0, ips: 310.26636 samples/s, eta: 0:51:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:59:16] ppocr INFO: epoch: [495/500], global_step: 531900, lr: 0.000100, edit distance: 0.001260, exp_rate: 0.943916, exp_rate<=1 : 0.978717, exp_rate<=2 : 0.995664, exp_rate<=3 : 0.998325, loss: 0.001260, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19499 s, avg_samples: 56.0, ips: 287.19191 samples/s, eta: 0:50:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:59:34] ppocr INFO: epoch: [495/500], global_step: 532000, lr: 0.000100, edit distance: 0.001269, exp_rate: 0.944601, exp_rate<=1 : 0.979426, exp_rate<=2 : 0.995809, exp_rate<=3 : 0.998386, loss: 0.000916, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18176 s, avg_samples: 56.0, ips: 308.09136 samples/s, eta: 0:50:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 06:59:54] ppocr INFO: epoch: [495/500], global_step: 532100, lr: 0.000100, edit distance: 0.001220, exp_rate: 0.945317, exp_rate<=1 : 0.979642, exp_rate<=2 : 0.995691, exp_rate<=3 : 0.998418, loss: 0.000974, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20307 s, avg_samples: 56.0, ips: 275.76599 samples/s, eta: 0:50:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:00:12] ppocr INFO: epoch: [495/500], global_step: 532200, lr: 0.000100, edit distance: 0.001215, exp_rate: 0.946584, exp_rate<=1 : 0.980072, exp_rate<=2 : 0.995859, exp_rate<=3 : 0.998612, loss: 0.000831, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17883 s, avg_samples: 56.0, ips: 313.14778 samples/s, eta: 0:50:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:00:31] ppocr INFO: epoch: [495/500], global_step: 532300, lr: 0.000100, edit distance: 0.001218, exp_rate: 0.945644, exp_rate<=1 : 0.979431, exp_rate<=2 : 0.995731, exp_rate<=3 : 0.998513, loss: 0.001610, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18563 s, avg_samples: 56.0, ips: 301.67714 samples/s, eta: 0:49:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:00:49] ppocr INFO: epoch: [495/500], global_step: 532400, lr: 0.000100, edit distance: 0.001234, exp_rate: 0.945076, exp_rate<=1 : 0.978989, exp_rate<=2 : 0.995635, exp_rate<=3 : 0.998394, loss: 0.001669, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18593 s, avg_samples: 56.0, ips: 301.18567 samples/s, eta: 0:49:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:01:08] ppocr INFO: epoch: [495/500], global_step: 532500, lr: 0.000100, edit distance: 0.001242, exp_rate: 0.943826, exp_rate<=1 : 0.978501, exp_rate<=2 : 0.995569, exp_rate<=3 : 0.998452, loss: 0.002084, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18499 s, avg_samples: 56.0, ips: 302.71482 samples/s, eta: 0:49:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:01:26] ppocr INFO: epoch: [495/500], global_step: 532600, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.943571, exp_rate<=1 : 0.978408, exp_rate<=2 : 0.995585, exp_rate<=3 : 0.998458, loss: 0.001671, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17945 s, avg_samples: 56.0, ips: 312.06974 samples/s, eta: 0:48:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:01:44] ppocr INFO: epoch: [495/500], global_step: 532700, lr: 0.000100, edit distance: 0.001236, exp_rate: 0.944134, exp_rate<=1 : 0.978620, exp_rate<=2 : 0.995555, exp_rate<=3 : 0.998495, loss: 0.000989, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18079 s, avg_samples: 56.0, ips: 309.74638 samples/s, eta: 0:48:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:02:02] ppocr INFO: epoch: [495/500], global_step: 532800, lr: 0.000100, edit distance: 0.001243, exp_rate: 0.943978, exp_rate<=1 : 0.978453, exp_rate<=2 : 0.995530, exp_rate<=3 : 0.998452, loss: 0.001395, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18320 s, avg_samples: 56.0, ips: 305.66991 samples/s, eta: 0:48:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:02:22] ppocr INFO: epoch: [495/500], global_step: 532900, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.944342, exp_rate<=1 : 0.978567, exp_rate<=2 : 0.995547, exp_rate<=3 : 0.998446, loss: 0.001510, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19406 s, avg_samples: 56.0, ips: 288.57460 samples/s, eta: 0:47:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:02:40] ppocr INFO: epoch: [495/500], global_step: 533000, lr: 0.000100, edit distance: 0.001219, exp_rate: 0.944392, exp_rate<=1 : 0.978634, exp_rate<=2 : 0.995536, exp_rate<=3 : 0.998456, loss: 0.001720, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17774 s, avg_samples: 56.0, ips: 315.06561 samples/s, eta: 0:47:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:02:58] ppocr INFO: epoch: [495/500], global_step: 533100, lr: 0.000100, edit distance: 0.001217, exp_rate: 0.944155, exp_rate<=1 : 0.978598, exp_rate<=2 : 0.995518, exp_rate<=3 : 0.998439, loss: 0.001607, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18513 s, avg_samples: 56.0, ips: 302.48260 samples/s, eta: 0:47:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:03:15] ppocr INFO: epoch: [495/500], global_step: 533200, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.944338, exp_rate<=1 : 0.978748, exp_rate<=2 : 0.995529, exp_rate<=3 : 0.998430, loss: 0.001241, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17290 s, avg_samples: 56.0, ips: 323.87818 samples/s, eta: 0:46:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:03:34] ppocr INFO: epoch: [495/500], global_step: 533300, lr: 0.000100, edit distance: 0.001235, exp_rate: 0.944176, exp_rate<=1 : 0.978721, exp_rate<=2 : 0.995543, exp_rate<=3 : 0.998432, loss: 0.001514, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19063 s, avg_samples: 56.0, ips: 293.76635 samples/s, eta: 0:46:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:03:54] ppocr INFO: epoch: [495/500], global_step: 533400, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.943890, exp_rate<=1 : 0.978672, exp_rate<=2 : 0.995452, exp_rate<=3 : 0.998332, loss: 0.001208, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19509 s, avg_samples: 56.0, ips: 287.04008 samples/s, eta: 0:46:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:04:13] ppocr INFO: epoch: [495/500], global_step: 533500, lr: 0.000100, edit distance: 0.001233, exp_rate: 0.943678, exp_rate<=1 : 0.978678, exp_rate<=2 : 0.995488, exp_rate<=3 : 0.998377, loss: 0.001319, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19059 s, avg_samples: 56.0, ips: 293.82247 samples/s, eta: 0:45:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:04:33] ppocr INFO: epoch: [495/500], global_step: 533600, lr: 0.000100, edit distance: 0.001237, exp_rate: 0.943483, exp_rate<=1 : 0.978478, exp_rate<=2 : 0.995394, exp_rate<=3 : 0.998292, loss: 0.001369, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20045 s, avg_samples: 56.0, ips: 279.37130 samples/s, eta: 0:45:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:04:52] ppocr INFO: epoch: [495/500], global_step: 533700, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.943480, exp_rate<=1 : 0.978410, exp_rate<=2 : 0.995339, exp_rate<=3 : 0.998244, loss: 0.001048, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18592 s, avg_samples: 56.0, ips: 301.20887 samples/s, eta: 0:45:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:05:12] ppocr INFO: epoch: [495/500], global_step: 533800, lr: 0.000100, edit distance: 0.001232, exp_rate: 0.943339, exp_rate<=1 : 0.978390, exp_rate<=2 : 0.995388, exp_rate<=3 : 0.998255, loss: 0.001366, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19954 s, avg_samples: 56.0, ips: 280.64594 samples/s, eta: 0:44:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:05:30] ppocr INFO: epoch: [495/500], global_step: 533900, lr: 0.000100, edit distance: 0.001231, exp_rate: 0.943410, exp_rate<=1 : 0.978369, exp_rate<=2 : 0.995399, exp_rate<=3 : 0.998245, loss: 0.001410, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18121 s, avg_samples: 56.0, ips: 309.02537 samples/s, eta: 0:44:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:05:49] ppocr INFO: epoch: [495/500], global_step: 534000, lr: 0.000100, edit distance: 0.001241, exp_rate: 0.943345, exp_rate<=1 : 0.978300, exp_rate<=2 : 0.995413, exp_rate<=3 : 0.998254, loss: 0.001572, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19276 s, avg_samples: 56.0, ips: 290.51297 samples/s, eta: 0:44:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:06:07] ppocr INFO: epoch: [495/500], global_step: 534100, lr: 0.000100, edit distance: 0.001247, exp_rate: 0.943310, exp_rate<=1 : 0.978364, exp_rate<=2 : 0.995454, exp_rate<=3 : 0.998288, loss: 0.001225, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17649 s, avg_samples: 56.0, ips: 317.29776 samples/s, eta: 0:44:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:06:26] ppocr INFO: epoch: [495/500], global_step: 534200, lr: 0.000100, edit distance: 0.001244, exp_rate: 0.943236, exp_rate<=1 : 0.978400, exp_rate<=2 : 0.995464, exp_rate<=3 : 0.998305, loss: 0.001670, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19409 s, avg_samples: 56.0, ips: 288.52841 samples/s, eta: 0:43:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:06:45] ppocr INFO: epoch: [495/500], global_step: 534300, lr: 0.000100, edit distance: 0.001240, exp_rate: 0.943305, exp_rate<=1 : 0.978424, exp_rate<=2 : 0.995444, exp_rate<=3 : 0.998304, loss: 0.001024, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18441 s, avg_samples: 56.0, ips: 303.66506 samples/s, eta: 0:43:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:06:47] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 07:06:48] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_495 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 07:07:08] ppocr INFO: epoch: [496/500], global_step: 534400, lr: 0.000100, edit distance: 0.000957, exp_rate: 0.955307, exp_rate<=1 : 0.983682, exp_rate<=2 : 0.997006, exp_rate<=3 : 0.998931, loss: 0.000860, avg_reader_cost: 0.03232 s, avg_batch_cost: 0.23029 s, avg_samples: 56.0, ips: 243.17182 samples/s, eta: 0:43:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:07:25] ppocr INFO: epoch: [496/500], global_step: 534500, lr: 0.000100, edit distance: 0.000873, exp_rate: 0.957879, exp_rate<=1 : 0.985104, exp_rate<=2 : 0.997238, exp_rate<=3 : 0.999063, loss: 0.000910, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17418 s, avg_samples: 56.0, ips: 321.51302 samples/s, eta: 0:42:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:07:45] ppocr INFO: epoch: [496/500], global_step: 534600, lr: 0.000100, edit distance: 0.000869, exp_rate: 0.954262, exp_rate<=1 : 0.983780, exp_rate<=2 : 0.996723, exp_rate<=3 : 0.998795, loss: 0.001256, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19602 s, avg_samples: 56.0, ips: 285.68512 samples/s, eta: 0:42:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:08:02] ppocr INFO: epoch: [496/500], global_step: 534700, lr: 0.000100, edit distance: 0.000966, exp_rate: 0.954930, exp_rate<=1 : 0.984036, exp_rate<=2 : 0.996929, exp_rate<=3 : 0.998885, loss: 0.000844, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17348 s, avg_samples: 56.0, ips: 322.80939 samples/s, eta: 0:42:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:08:21] ppocr INFO: epoch: [496/500], global_step: 534800, lr: 0.000100, edit distance: 0.000994, exp_rate: 0.954634, exp_rate<=1 : 0.983526, exp_rate<=2 : 0.996933, exp_rate<=3 : 0.999000, loss: 0.000806, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18686 s, avg_samples: 56.0, ips: 299.68583 samples/s, eta: 0:41:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:08:41] ppocr INFO: epoch: [496/500], global_step: 534900, lr: 0.000100, edit distance: 0.001024, exp_rate: 0.952985, exp_rate<=1 : 0.982524, exp_rate<=2 : 0.996712, exp_rate<=3 : 0.998927, loss: 0.001546, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19729 s, avg_samples: 56.0, ips: 283.84788 samples/s, eta: 0:41:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:08:59] ppocr INFO: epoch: [496/500], global_step: 535000, lr: 0.000100, edit distance: 0.001011, exp_rate: 0.953062, exp_rate<=1 : 0.982739, exp_rate<=2 : 0.996763, exp_rate<=3 : 0.998938, loss: 0.001145, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18011 s, avg_samples: 56.0, ips: 310.91600 samples/s, eta: 0:41:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:09:17] ppocr INFO: epoch: [496/500], global_step: 535100, lr: 0.000100, edit distance: 0.001007, exp_rate: 0.952606, exp_rate<=1 : 0.982802, exp_rate<=2 : 0.996812, exp_rate<=3 : 0.998998, loss: 0.001183, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17833 s, avg_samples: 56.0, ips: 314.01606 samples/s, eta: 0:40:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:09:36] ppocr INFO: epoch: [496/500], global_step: 535200, lr: 0.000100, edit distance: 0.001024, exp_rate: 0.950820, exp_rate<=1 : 0.982022, exp_rate<=2 : 0.996487, exp_rate<=3 : 0.998795, loss: 0.001415, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19496 s, avg_samples: 56.0, ips: 287.24009 samples/s, eta: 0:40:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:09:56] ppocr INFO: epoch: [496/500], global_step: 535300, lr: 0.000100, edit distance: 0.001019, exp_rate: 0.950356, exp_rate<=1 : 0.981755, exp_rate<=2 : 0.996304, exp_rate<=3 : 0.998702, loss: 0.000823, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19817 s, avg_samples: 56.0, ips: 282.58939 samples/s, eta: 0:40:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:10:14] ppocr INFO: epoch: [496/500], global_step: 535400, lr: 0.000100, edit distance: 0.001001, exp_rate: 0.950991, exp_rate<=1 : 0.982077, exp_rate<=2 : 0.996496, exp_rate<=3 : 0.998788, loss: 0.000863, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17866 s, avg_samples: 56.0, ips: 313.44379 samples/s, eta: 0:39:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:10:32] ppocr INFO: epoch: [496/500], global_step: 535500, lr: 0.000100, edit distance: 0.001008, exp_rate: 0.951024, exp_rate<=1 : 0.982045, exp_rate<=2 : 0.996565, exp_rate<=3 : 0.998828, loss: 0.000873, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18226 s, avg_samples: 56.0, ips: 307.26114 samples/s, eta: 0:39:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:10:51] ppocr INFO: epoch: [496/500], global_step: 535600, lr: 0.000100, edit distance: 0.001021, exp_rate: 0.950447, exp_rate<=1 : 0.981768, exp_rate<=2 : 0.996455, exp_rate<=3 : 0.998811, loss: 0.001238, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19160 s, avg_samples: 56.0, ips: 292.28287 samples/s, eta: 0:39:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:11:11] ppocr INFO: epoch: [496/500], global_step: 535700, lr: 0.000100, edit distance: 0.001034, exp_rate: 0.950065, exp_rate<=1 : 0.981514, exp_rate<=2 : 0.996381, exp_rate<=3 : 0.998813, loss: 0.002088, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19364 s, avg_samples: 56.0, ips: 289.19745 samples/s, eta: 0:38:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:11:29] ppocr INFO: epoch: [496/500], global_step: 535800, lr: 0.000100, edit distance: 0.001057, exp_rate: 0.949021, exp_rate<=1 : 0.981137, exp_rate<=2 : 0.996299, exp_rate<=3 : 0.998778, loss: 0.001498, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18726 s, avg_samples: 56.0, ips: 299.04509 samples/s, eta: 0:38:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:11:47] ppocr INFO: epoch: [496/500], global_step: 535900, lr: 0.000100, edit distance: 0.001100, exp_rate: 0.947748, exp_rate<=1 : 0.980537, exp_rate<=2 : 0.995974, exp_rate<=3 : 0.998595, loss: 0.001178, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18057 s, avg_samples: 56.0, ips: 310.13531 samples/s, eta: 0:38:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:12:07] ppocr INFO: epoch: [496/500], global_step: 536000, lr: 0.000100, edit distance: 0.001123, exp_rate: 0.946645, exp_rate<=1 : 0.980129, exp_rate<=2 : 0.995799, exp_rate<=3 : 0.998485, loss: 0.001743, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19472 s, avg_samples: 56.0, ips: 287.59959 samples/s, eta: 0:38:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:12:25] ppocr INFO: epoch: [496/500], global_step: 536100, lr: 0.000100, edit distance: 0.001131, exp_rate: 0.946544, exp_rate<=1 : 0.980043, exp_rate<=2 : 0.995781, exp_rate<=3 : 0.998464, loss: 0.001620, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18061 s, avg_samples: 56.0, ips: 310.06151 samples/s, eta: 0:37:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:12:42] ppocr INFO: epoch: [496/500], global_step: 536200, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.946607, exp_rate<=1 : 0.980177, exp_rate<=2 : 0.995789, exp_rate<=3 : 0.998487, loss: 0.001678, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16859 s, avg_samples: 56.0, ips: 332.17008 samples/s, eta: 0:37:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:13:01] ppocr INFO: epoch: [496/500], global_step: 536300, lr: 0.000100, edit distance: 0.001144, exp_rate: 0.946415, exp_rate<=1 : 0.980125, exp_rate<=2 : 0.995743, exp_rate<=3 : 0.998475, loss: 0.000902, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18732 s, avg_samples: 56.0, ips: 298.96084 samples/s, eta: 0:37:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:13:20] ppocr INFO: epoch: [496/500], global_step: 536400, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.946318, exp_rate<=1 : 0.980085, exp_rate<=2 : 0.995720, exp_rate<=3 : 0.998522, loss: 0.000943, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19228 s, avg_samples: 56.0, ips: 291.24149 samples/s, eta: 0:36:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:13:40] ppocr INFO: epoch: [496/500], global_step: 536500, lr: 0.000100, edit distance: 0.001150, exp_rate: 0.945918, exp_rate<=1 : 0.980033, exp_rate<=2 : 0.995685, exp_rate<=3 : 0.998527, loss: 0.000947, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19917 s, avg_samples: 56.0, ips: 281.16782 samples/s, eta: 0:36:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:13:58] ppocr INFO: epoch: [496/500], global_step: 536600, lr: 0.000100, edit distance: 0.001157, exp_rate: 0.945631, exp_rate<=1 : 0.979959, exp_rate<=2 : 0.995695, exp_rate<=3 : 0.998511, loss: 0.001045, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17742 s, avg_samples: 56.0, ips: 315.64407 samples/s, eta: 0:36:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:14:16] ppocr INFO: epoch: [496/500], global_step: 536700, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.945943, exp_rate<=1 : 0.980083, exp_rate<=2 : 0.995757, exp_rate<=3 : 0.998551, loss: 0.001573, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17912 s, avg_samples: 56.0, ips: 312.64172 samples/s, eta: 0:35:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:14:36] ppocr INFO: epoch: [496/500], global_step: 536800, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.945350, exp_rate<=1 : 0.979693, exp_rate<=2 : 0.995722, exp_rate<=3 : 0.998536, loss: 0.001634, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20617 s, avg_samples: 56.0, ips: 271.62479 samples/s, eta: 0:35:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:14:56] ppocr INFO: epoch: [496/500], global_step: 536900, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.944979, exp_rate<=1 : 0.979469, exp_rate<=2 : 0.995671, exp_rate<=3 : 0.998494, loss: 0.001422, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19905 s, avg_samples: 56.0, ips: 281.33126 samples/s, eta: 0:35:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:15:17] ppocr INFO: epoch: [496/500], global_step: 537000, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.944687, exp_rate<=1 : 0.979418, exp_rate<=2 : 0.995658, exp_rate<=3 : 0.998484, loss: 0.001627, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20595 s, avg_samples: 56.0, ips: 271.90526 samples/s, eta: 0:34:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:15:23] ppocr INFO: epoch: [496/500], global_step: 537040, lr: 0.000100, edit distance: 0.001169, exp_rate: 0.944757, exp_rate<=1 : 0.979471, exp_rate<=2 : 0.995684, exp_rate<=3 : 0.998502, loss: 0.000948, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.06443 s, avg_samples: 22.4, ips: 347.64266 samples/s, eta: 0:34:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:15:25] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 07:15:35] ppocr INFO: epoch: [497/500], global_step: 537100, lr: 0.000100, edit distance: 0.001011, exp_rate: 0.959159, exp_rate<=1 : 0.984146, exp_rate<=2 : 0.995227, exp_rate<=3 : 0.998665, loss: 0.001609, avg_reader_cost: 0.02448 s, avg_batch_cost: 0.12271 s, avg_samples: 33.6, ips: 273.82256 samples/s, eta: 0:34:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:15:55] ppocr INFO: epoch: [497/500], global_step: 537200, lr: 0.000100, edit distance: 0.001082, exp_rate: 0.950761, exp_rate<=1 : 0.981292, exp_rate<=2 : 0.994419, exp_rate<=3 : 0.997855, loss: 0.000986, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19368 s, avg_samples: 56.0, ips: 289.13592 samples/s, eta: 0:34:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:16:12] ppocr INFO: epoch: [497/500], global_step: 537300, lr: 0.000100, edit distance: 0.001019, exp_rate: 0.953328, exp_rate<=1 : 0.982902, exp_rate<=2 : 0.994878, exp_rate<=3 : 0.998396, loss: 0.001376, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.17351 s, avg_samples: 56.0, ips: 322.74811 samples/s, eta: 0:33:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:16:32] ppocr INFO: epoch: [497/500], global_step: 537400, lr: 0.000100, edit distance: 0.001033, exp_rate: 0.953750, exp_rate<=1 : 0.983009, exp_rate<=2 : 0.995212, exp_rate<=3 : 0.998467, loss: 0.000875, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19378 s, avg_samples: 56.0, ips: 288.98528 samples/s, eta: 0:33:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:16:49] ppocr INFO: epoch: [497/500], global_step: 537500, lr: 0.000100, edit distance: 0.001021, exp_rate: 0.955308, exp_rate<=1 : 0.984024, exp_rate<=2 : 0.995844, exp_rate<=3 : 0.998732, loss: 0.000780, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17422 s, avg_samples: 56.0, ips: 321.42850 samples/s, eta: 0:33:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:17:08] ppocr INFO: epoch: [497/500], global_step: 537600, lr: 0.000100, edit distance: 0.001016, exp_rate: 0.954623, exp_rate<=1 : 0.983838, exp_rate<=2 : 0.996126, exp_rate<=3 : 0.998858, loss: 0.001357, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18836 s, avg_samples: 56.0, ips: 297.30727 samples/s, eta: 0:32:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:17:26] ppocr INFO: epoch: [497/500], global_step: 537700, lr: 0.000100, edit distance: 0.001021, exp_rate: 0.954689, exp_rate<=1 : 0.984040, exp_rate<=2 : 0.996171, exp_rate<=3 : 0.998878, loss: 0.001069, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18176 s, avg_samples: 56.0, ips: 308.09918 samples/s, eta: 0:32:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:17:44] ppocr INFO: epoch: [497/500], global_step: 537800, lr: 0.000100, edit distance: 0.001018, exp_rate: 0.954713, exp_rate<=1 : 0.983982, exp_rate<=2 : 0.996292, exp_rate<=3 : 0.998864, loss: 0.001383, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18376 s, avg_samples: 56.0, ips: 304.75211 samples/s, eta: 0:32:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:18:04] ppocr INFO: epoch: [497/500], global_step: 537900, lr: 0.000100, edit distance: 0.001044, exp_rate: 0.952430, exp_rate<=1 : 0.983036, exp_rate<=2 : 0.996241, exp_rate<=3 : 0.998791, loss: 0.001142, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19794 s, avg_samples: 56.0, ips: 282.90947 samples/s, eta: 0:31:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:18:24] ppocr INFO: epoch: [497/500], global_step: 538000, lr: 0.000100, edit distance: 0.001069, exp_rate: 0.950947, exp_rate<=1 : 0.982425, exp_rate<=2 : 0.996106, exp_rate<=3 : 0.998713, loss: 0.001932, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19519 s, avg_samples: 56.0, ips: 286.90726 samples/s, eta: 0:31:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:18:43] ppocr INFO: epoch: [497/500], global_step: 538100, lr: 0.000100, edit distance: 0.001088, exp_rate: 0.950225, exp_rate<=1 : 0.982024, exp_rate<=2 : 0.995943, exp_rate<=3 : 0.998546, loss: 0.000830, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18912 s, avg_samples: 56.0, ips: 296.11460 samples/s, eta: 0:31:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:19:02] ppocr INFO: epoch: [497/500], global_step: 538200, lr: 0.000100, edit distance: 0.001105, exp_rate: 0.949074, exp_rate<=1 : 0.981175, exp_rate<=2 : 0.995629, exp_rate<=3 : 0.998249, loss: 0.001855, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18714 s, avg_samples: 56.0, ips: 299.23597 samples/s, eta: 0:31:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:19:22] ppocr INFO: epoch: [497/500], global_step: 538300, lr: 0.000100, edit distance: 0.001115, exp_rate: 0.948032, exp_rate<=1 : 0.980627, exp_rate<=2 : 0.995329, exp_rate<=3 : 0.998139, loss: 0.000822, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20129 s, avg_samples: 56.0, ips: 278.21000 samples/s, eta: 0:30:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:19:40] ppocr INFO: epoch: [497/500], global_step: 538400, lr: 0.000100, edit distance: 0.001119, exp_rate: 0.948003, exp_rate<=1 : 0.980536, exp_rate<=2 : 0.995276, exp_rate<=3 : 0.998109, loss: 0.001117, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18165 s, avg_samples: 56.0, ips: 308.28177 samples/s, eta: 0:30:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:19:57] ppocr INFO: epoch: [497/500], global_step: 538500, lr: 0.000100, edit distance: 0.001103, exp_rate: 0.948410, exp_rate<=1 : 0.980782, exp_rate<=2 : 0.995335, exp_rate<=3 : 0.998190, loss: 0.001152, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17352 s, avg_samples: 56.0, ips: 322.73740 samples/s, eta: 0:30:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:20:15] ppocr INFO: epoch: [497/500], global_step: 538600, lr: 0.000100, edit distance: 0.001115, exp_rate: 0.948514, exp_rate<=1 : 0.980867, exp_rate<=2 : 0.995409, exp_rate<=3 : 0.998241, loss: 0.001233, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18052 s, avg_samples: 56.0, ips: 310.21964 samples/s, eta: 0:29:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:20:36] ppocr INFO: epoch: [497/500], global_step: 538700, lr: 0.000100, edit distance: 0.001127, exp_rate: 0.947727, exp_rate<=1 : 0.980545, exp_rate<=2 : 0.995272, exp_rate<=3 : 0.998189, loss: 0.001030, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20492 s, avg_samples: 56.0, ips: 273.27332 samples/s, eta: 0:29:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:20:54] ppocr INFO: epoch: [497/500], global_step: 538800, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.947833, exp_rate<=1 : 0.980526, exp_rate<=2 : 0.995242, exp_rate<=3 : 0.998202, loss: 0.001144, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18549 s, avg_samples: 56.0, ips: 301.89532 samples/s, eta: 0:29:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:21:15] ppocr INFO: epoch: [497/500], global_step: 538900, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.946890, exp_rate<=1 : 0.980133, exp_rate<=2 : 0.995087, exp_rate<=3 : 0.998191, loss: 0.001676, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20332 s, avg_samples: 56.0, ips: 275.43197 samples/s, eta: 0:28:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:21:34] ppocr INFO: epoch: [497/500], global_step: 539000, lr: 0.000100, edit distance: 0.001154, exp_rate: 0.946442, exp_rate<=1 : 0.979947, exp_rate<=2 : 0.995014, exp_rate<=3 : 0.998142, loss: 0.001275, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19481 s, avg_samples: 56.0, ips: 287.45955 samples/s, eta: 0:28:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:21:54] ppocr INFO: epoch: [497/500], global_step: 539100, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.946446, exp_rate<=1 : 0.979955, exp_rate<=2 : 0.995059, exp_rate<=3 : 0.998165, loss: 0.001501, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20017 s, avg_samples: 56.0, ips: 279.75656 samples/s, eta: 0:28:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:22:14] ppocr INFO: epoch: [497/500], global_step: 539200, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.946124, exp_rate<=1 : 0.979839, exp_rate<=2 : 0.994961, exp_rate<=3 : 0.998149, loss: 0.001304, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19384 s, avg_samples: 56.0, ips: 288.89752 samples/s, eta: 0:27:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:22:32] ppocr INFO: epoch: [497/500], global_step: 539300, lr: 0.000100, edit distance: 0.001163, exp_rate: 0.946321, exp_rate<=1 : 0.979914, exp_rate<=2 : 0.995031, exp_rate<=3 : 0.998170, loss: 0.001334, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17967 s, avg_samples: 56.0, ips: 311.67498 samples/s, eta: 0:27:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:22:50] ppocr INFO: epoch: [497/500], global_step: 539400, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.945867, exp_rate<=1 : 0.979731, exp_rate<=2 : 0.995012, exp_rate<=3 : 0.998199, loss: 0.001532, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18569 s, avg_samples: 56.0, ips: 301.57450 samples/s, eta: 0:27:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:23:09] ppocr INFO: epoch: [497/500], global_step: 539500, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.945516, exp_rate<=1 : 0.979628, exp_rate<=2 : 0.995006, exp_rate<=3 : 0.998229, loss: 0.001273, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18450 s, avg_samples: 56.0, ips: 303.51776 samples/s, eta: 0:26:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:23:29] ppocr INFO: epoch: [497/500], global_step: 539600, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.944994, exp_rate<=1 : 0.979563, exp_rate<=2 : 0.995001, exp_rate<=3 : 0.998216, loss: 0.001223, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20385 s, avg_samples: 56.0, ips: 274.71738 samples/s, eta: 0:26:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:23:47] ppocr INFO: epoch: [497/500], global_step: 539700, lr: 0.000100, edit distance: 0.001205, exp_rate: 0.944931, exp_rate<=1 : 0.979578, exp_rate<=2 : 0.995016, exp_rate<=3 : 0.998255, loss: 0.001750, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18285 s, avg_samples: 56.0, ips: 306.25751 samples/s, eta: 0:26:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:24:03] ppocr INFO: epoch: [497/500], global_step: 539780, lr: 0.000100, edit distance: 0.001210, exp_rate: 0.944733, exp_rate<=1 : 0.979402, exp_rate<=2 : 0.994902, exp_rate<=3 : 0.998238, loss: 0.000975, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.15395 s, avg_samples: 44.8, ips: 290.99552 samples/s, eta: 0:26:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:24:05] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 07:24:09] ppocr INFO: epoch: [498/500], global_step: 539800, lr: 0.000100, edit distance: 0.001301, exp_rate: 0.959673, exp_rate<=1 : 0.984828, exp_rate<=2 : 1.000000, exp_rate<=3 : 1.000000, loss: 0.001319, avg_reader_cost: 0.02521 s, avg_batch_cost: 0.06091 s, avg_samples: 11.2, ips: 183.88310 samples/s, eta: 0:25:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:24:28] ppocr INFO: epoch: [498/500], global_step: 539900, lr: 0.000100, edit distance: 0.001063, exp_rate: 0.952271, exp_rate<=1 : 0.981327, exp_rate<=2 : 0.994322, exp_rate<=3 : 0.997717, loss: 0.001150, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18916 s, avg_samples: 56.0, ips: 296.05140 samples/s, eta: 0:25:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:24:48] ppocr INFO: epoch: [498/500], global_step: 540000, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.949812, exp_rate<=1 : 0.980572, exp_rate<=2 : 0.994359, exp_rate<=3 : 0.997837, loss: 0.001105, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19915 s, avg_samples: 56.0, ips: 281.18945 samples/s, eta: 0:25:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:25:06] ppocr INFO: epoch: [498/500], global_step: 540100, lr: 0.000100, edit distance: 0.001137, exp_rate: 0.950464, exp_rate<=1 : 0.981287, exp_rate<=2 : 0.995054, exp_rate<=3 : 0.998160, loss: 0.001104, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18476 s, avg_samples: 56.0, ips: 303.08965 samples/s, eta: 0:25:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:25:25] ppocr INFO: epoch: [498/500], global_step: 540200, lr: 0.000100, edit distance: 0.001093, exp_rate: 0.951496, exp_rate<=1 : 0.981574, exp_rate<=2 : 0.995045, exp_rate<=3 : 0.998236, loss: 0.000857, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18266 s, avg_samples: 56.0, ips: 306.57695 samples/s, eta: 0:24:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:25:44] ppocr INFO: epoch: [498/500], global_step: 540300, lr: 0.000100, edit distance: 0.001092, exp_rate: 0.950510, exp_rate<=1 : 0.981632, exp_rate<=2 : 0.995281, exp_rate<=3 : 0.998308, loss: 0.000866, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18889 s, avg_samples: 56.0, ips: 296.46850 samples/s, eta: 0:24:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:26:02] ppocr INFO: epoch: [498/500], global_step: 540400, lr: 0.000100, edit distance: 0.001065, exp_rate: 0.951093, exp_rate<=1 : 0.981852, exp_rate<=2 : 0.995356, exp_rate<=3 : 0.998403, loss: 0.000704, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18108 s, avg_samples: 56.0, ips: 309.25114 samples/s, eta: 0:24:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:26:20] ppocr INFO: epoch: [498/500], global_step: 540500, lr: 0.000100, edit distance: 0.001054, exp_rate: 0.951753, exp_rate<=1 : 0.982043, exp_rate<=2 : 0.995567, exp_rate<=3 : 0.998536, loss: 0.001173, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17982 s, avg_samples: 56.0, ips: 311.42450 samples/s, eta: 0:23:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:26:37] ppocr INFO: epoch: [498/500], global_step: 540600, lr: 0.000100, edit distance: 0.001051, exp_rate: 0.952953, exp_rate<=1 : 0.982561, exp_rate<=2 : 0.995765, exp_rate<=3 : 0.998595, loss: 0.001301, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17318 s, avg_samples: 56.0, ips: 323.35593 samples/s, eta: 0:23:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:26:56] ppocr INFO: epoch: [498/500], global_step: 540700, lr: 0.000100, edit distance: 0.001052, exp_rate: 0.952262, exp_rate<=1 : 0.982182, exp_rate<=2 : 0.995671, exp_rate<=3 : 0.998562, loss: 0.001081, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18933 s, avg_samples: 56.0, ips: 295.78679 samples/s, eta: 0:23:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:27:17] ppocr INFO: epoch: [498/500], global_step: 540800, lr: 0.000100, edit distance: 0.001071, exp_rate: 0.950614, exp_rate<=1 : 0.981459, exp_rate<=2 : 0.995339, exp_rate<=3 : 0.998456, loss: 0.001212, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20892 s, avg_samples: 56.0, ips: 268.04617 samples/s, eta: 0:22:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:27:36] ppocr INFO: epoch: [498/500], global_step: 540900, lr: 0.000100, edit distance: 0.001074, exp_rate: 0.949853, exp_rate<=1 : 0.981239, exp_rate<=2 : 0.995325, exp_rate<=3 : 0.998440, loss: 0.001458, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19148 s, avg_samples: 56.0, ips: 292.46557 samples/s, eta: 0:22:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:27:56] ppocr INFO: epoch: [498/500], global_step: 541000, lr: 0.000100, edit distance: 0.001079, exp_rate: 0.948803, exp_rate<=1 : 0.981162, exp_rate<=2 : 0.995357, exp_rate<=3 : 0.998487, loss: 0.001643, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19445 s, avg_samples: 56.0, ips: 287.99480 samples/s, eta: 0:22:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:28:14] ppocr INFO: epoch: [498/500], global_step: 541100, lr: 0.000100, edit distance: 0.001088, exp_rate: 0.948297, exp_rate<=1 : 0.981057, exp_rate<=2 : 0.995439, exp_rate<=3 : 0.998569, loss: 0.001356, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18962 s, avg_samples: 56.0, ips: 295.32085 samples/s, eta: 0:21:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:28:34] ppocr INFO: epoch: [498/500], global_step: 541200, lr: 0.000100, edit distance: 0.001092, exp_rate: 0.948040, exp_rate<=1 : 0.981109, exp_rate<=2 : 0.995555, exp_rate<=3 : 0.998629, loss: 0.001355, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19639 s, avg_samples: 56.0, ips: 285.14570 samples/s, eta: 0:21:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:28:54] ppocr INFO: epoch: [498/500], global_step: 541300, lr: 0.000100, edit distance: 0.001092, exp_rate: 0.947860, exp_rate<=1 : 0.981084, exp_rate<=2 : 0.995556, exp_rate<=3 : 0.998610, loss: 0.001245, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19468 s, avg_samples: 56.0, ips: 287.65699 samples/s, eta: 0:21:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:29:11] ppocr INFO: epoch: [498/500], global_step: 541400, lr: 0.000100, edit distance: 0.001115, exp_rate: 0.947918, exp_rate<=1 : 0.980937, exp_rate<=2 : 0.995543, exp_rate<=3 : 0.998588, loss: 0.001065, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17434 s, avg_samples: 56.0, ips: 321.20847 samples/s, eta: 0:20:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:29:29] ppocr INFO: epoch: [498/500], global_step: 541500, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.947528, exp_rate<=1 : 0.980768, exp_rate<=2 : 0.995452, exp_rate<=3 : 0.998531, loss: 0.001305, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17872 s, avg_samples: 56.0, ips: 313.34747 samples/s, eta: 0:20:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:29:49] ppocr INFO: epoch: [498/500], global_step: 541600, lr: 0.000100, edit distance: 0.001127, exp_rate: 0.947001, exp_rate<=1 : 0.980481, exp_rate<=2 : 0.995473, exp_rate<=3 : 0.998501, loss: 0.001022, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19610 s, avg_samples: 56.0, ips: 285.56986 samples/s, eta: 0:20:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:30:08] ppocr INFO: epoch: [498/500], global_step: 541700, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.946905, exp_rate<=1 : 0.980403, exp_rate<=2 : 0.995458, exp_rate<=3 : 0.998482, loss: 0.001624, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19031 s, avg_samples: 56.0, ips: 294.26027 samples/s, eta: 0:19:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:30:25] ppocr INFO: epoch: [498/500], global_step: 541800, lr: 0.000100, edit distance: 0.001130, exp_rate: 0.946909, exp_rate<=1 : 0.980442, exp_rate<=2 : 0.995470, exp_rate<=3 : 0.998490, loss: 0.001367, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16968 s, avg_samples: 56.0, ips: 330.03664 samples/s, eta: 0:19:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:30:43] ppocr INFO: epoch: [498/500], global_step: 541900, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.946150, exp_rate<=1 : 0.980085, exp_rate<=2 : 0.995423, exp_rate<=3 : 0.998491, loss: 0.002110, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18258 s, avg_samples: 56.0, ips: 306.71961 samples/s, eta: 0:19:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:31:02] ppocr INFO: epoch: [498/500], global_step: 542000, lr: 0.000100, edit distance: 0.001159, exp_rate: 0.945613, exp_rate<=1 : 0.979917, exp_rate<=2 : 0.995442, exp_rate<=3 : 0.998485, loss: 0.001443, avg_reader_cost: 0.00017 s, avg_batch_cost: 0.18722 s, avg_samples: 56.0, ips: 299.10848 samples/s, eta: 0:19:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:31:21] ppocr INFO: epoch: [498/500], global_step: 542100, lr: 0.000100, edit distance: 0.001171, exp_rate: 0.945208, exp_rate<=1 : 0.979736, exp_rate<=2 : 0.995433, exp_rate<=3 : 0.998480, loss: 0.001420, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19720 s, avg_samples: 56.0, ips: 283.97362 samples/s, eta: 0:18:41, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:31:42] ppocr INFO: epoch: [498/500], global_step: 542200, lr: 0.000100, edit distance: 0.001176, exp_rate: 0.944833, exp_rate<=1 : 0.979499, exp_rate<=2 : 0.995376, exp_rate<=3 : 0.998456, loss: 0.001277, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20739 s, avg_samples: 56.0, ips: 270.01988 samples/s, eta: 0:18:22, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:32:02] ppocr INFO: epoch: [498/500], global_step: 542300, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.944463, exp_rate<=1 : 0.979385, exp_rate<=2 : 0.995304, exp_rate<=3 : 0.998433, loss: 0.001472, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20141 s, avg_samples: 56.0, ips: 278.04657 samples/s, eta: 0:18:03, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:32:21] ppocr INFO: epoch: [498/500], global_step: 542400, lr: 0.000100, edit distance: 0.001184, exp_rate: 0.944460, exp_rate<=1 : 0.979432, exp_rate<=2 : 0.995339, exp_rate<=3 : 0.998465, loss: 0.001479, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18799 s, avg_samples: 56.0, ips: 297.89201 samples/s, eta: 0:17:44, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:32:40] ppocr INFO: epoch: [498/500], global_step: 542500, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.944297, exp_rate<=1 : 0.979379, exp_rate<=2 : 0.995320, exp_rate<=3 : 0.998469, loss: 0.001571, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18844 s, avg_samples: 56.0, ips: 297.17715 samples/s, eta: 0:17:25, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:32:44] ppocr INFO: epoch: [498/500], global_step: 542520, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.944154, exp_rate<=1 : 0.979311, exp_rate<=2 : 0.995297, exp_rate<=3 : 0.998464, loss: 0.001853, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03675 s, avg_samples: 11.2, ips: 304.78532 samples/s, eta: 0:17:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:32:46] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 07:33:02] ppocr INFO: epoch: [499/500], global_step: 542600, lr: 0.000100, edit distance: 0.000873, exp_rate: 0.953107, exp_rate<=1 : 0.984157, exp_rate<=2 : 0.997085, exp_rate<=3 : 0.999015, loss: 0.001128, avg_reader_cost: 0.02511 s, avg_batch_cost: 0.17852 s, avg_samples: 44.8, ips: 250.94954 samples/s, eta: 0:17:06, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:33:20] ppocr INFO: epoch: [499/500], global_step: 542700, lr: 0.000100, edit distance: 0.000911, exp_rate: 0.953412, exp_rate<=1 : 0.982892, exp_rate<=2 : 0.996211, exp_rate<=3 : 0.998702, loss: 0.001023, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18834 s, avg_samples: 56.0, ips: 297.33094 samples/s, eta: 0:16:47, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:33:40] ppocr INFO: epoch: [499/500], global_step: 542800, lr: 0.000100, edit distance: 0.001029, exp_rate: 0.951573, exp_rate<=1 : 0.982538, exp_rate<=2 : 0.996369, exp_rate<=3 : 0.998825, loss: 0.001181, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19519 s, avg_samples: 56.0, ips: 286.90110 samples/s, eta: 0:16:28, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:33:58] ppocr INFO: epoch: [499/500], global_step: 542900, lr: 0.000100, edit distance: 0.001104, exp_rate: 0.950229, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.996462, exp_rate<=3 : 0.998891, loss: 0.001061, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18432 s, avg_samples: 56.0, ips: 303.81614 samples/s, eta: 0:16:09, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:34:17] ppocr INFO: epoch: [499/500], global_step: 543000, lr: 0.000100, edit distance: 0.001060, exp_rate: 0.950841, exp_rate<=1 : 0.982581, exp_rate<=2 : 0.996621, exp_rate<=3 : 0.999003, loss: 0.001115, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18413 s, avg_samples: 56.0, ips: 304.13065 samples/s, eta: 0:15:50, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:34:35] ppocr INFO: epoch: [499/500], global_step: 543100, lr: 0.000100, edit distance: 0.001017, exp_rate: 0.951230, exp_rate<=1 : 0.982710, exp_rate<=2 : 0.996721, exp_rate<=3 : 0.999033, loss: 0.001172, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18543 s, avg_samples: 56.0, ips: 302.00836 samples/s, eta: 0:15:31, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:34:55] ppocr INFO: epoch: [499/500], global_step: 543200, lr: 0.000100, edit distance: 0.001034, exp_rate: 0.950008, exp_rate<=1 : 0.981554, exp_rate<=2 : 0.996445, exp_rate<=3 : 0.998860, loss: 0.001786, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19861 s, avg_samples: 56.0, ips: 281.95561 samples/s, eta: 0:15:12, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:35:15] ppocr INFO: epoch: [499/500], global_step: 543300, lr: 0.000100, edit distance: 0.001046, exp_rate: 0.948927, exp_rate<=1 : 0.981381, exp_rate<=2 : 0.996373, exp_rate<=3 : 0.998758, loss: 0.001323, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19159 s, avg_samples: 56.0, ips: 292.29813 samples/s, eta: 0:14:53, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:35:33] ppocr INFO: epoch: [499/500], global_step: 543400, lr: 0.000100, edit distance: 0.001036, exp_rate: 0.949067, exp_rate<=1 : 0.981629, exp_rate<=2 : 0.996439, exp_rate<=3 : 0.998851, loss: 0.001049, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18876 s, avg_samples: 56.0, ips: 296.67433 samples/s, eta: 0:14:34, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:35:53] ppocr INFO: epoch: [499/500], global_step: 543500, lr: 0.000100, edit distance: 0.001039, exp_rate: 0.948808, exp_rate<=1 : 0.981406, exp_rate<=2 : 0.996353, exp_rate<=3 : 0.998832, loss: 0.001160, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19590 s, avg_samples: 56.0, ips: 285.86293 samples/s, eta: 0:14:15, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:36:12] ppocr INFO: epoch: [499/500], global_step: 543600, lr: 0.000100, edit distance: 0.001071, exp_rate: 0.948618, exp_rate<=1 : 0.981558, exp_rate<=2 : 0.996237, exp_rate<=3 : 0.998751, loss: 0.001272, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19040 s, avg_samples: 56.0, ips: 294.11856 samples/s, eta: 0:13:56, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:36:31] ppocr INFO: epoch: [499/500], global_step: 543700, lr: 0.000100, edit distance: 0.001120, exp_rate: 0.947307, exp_rate<=1 : 0.981332, exp_rate<=2 : 0.996218, exp_rate<=3 : 0.998764, loss: 0.000960, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18506 s, avg_samples: 56.0, ips: 302.59783 samples/s, eta: 0:13:37, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:36:50] ppocr INFO: epoch: [499/500], global_step: 543800, lr: 0.000100, edit distance: 0.001114, exp_rate: 0.947235, exp_rate<=1 : 0.981315, exp_rate<=2 : 0.996248, exp_rate<=3 : 0.998778, loss: 0.001467, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19318 s, avg_samples: 56.0, ips: 289.87878 samples/s, eta: 0:13:18, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:37:08] ppocr INFO: epoch: [499/500], global_step: 543900, lr: 0.000100, edit distance: 0.001123, exp_rate: 0.946768, exp_rate<=1 : 0.981014, exp_rate<=2 : 0.996091, exp_rate<=3 : 0.998606, loss: 0.001368, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18134 s, avg_samples: 56.0, ips: 308.81363 samples/s, eta: 0:12:59, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:37:26] ppocr INFO: epoch: [499/500], global_step: 544000, lr: 0.000100, edit distance: 0.001143, exp_rate: 0.946763, exp_rate<=1 : 0.980883, exp_rate<=2 : 0.996112, exp_rate<=3 : 0.998595, loss: 0.001515, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17816 s, avg_samples: 56.0, ips: 314.31668 samples/s, eta: 0:12:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:37:44] ppocr INFO: epoch: [499/500], global_step: 544100, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.947271, exp_rate<=1 : 0.981147, exp_rate<=2 : 0.996154, exp_rate<=3 : 0.998630, loss: 0.001254, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18544 s, avg_samples: 56.0, ips: 301.98094 samples/s, eta: 0:12:21, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:38:04] ppocr INFO: epoch: [499/500], global_step: 544200, lr: 0.000100, edit distance: 0.001148, exp_rate: 0.946909, exp_rate<=1 : 0.981003, exp_rate<=2 : 0.996136, exp_rate<=3 : 0.998638, loss: 0.001562, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19618 s, avg_samples: 56.0, ips: 285.45565 samples/s, eta: 0:12:02, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:38:23] ppocr INFO: epoch: [499/500], global_step: 544300, lr: 0.000100, edit distance: 0.001142, exp_rate: 0.947261, exp_rate<=1 : 0.981065, exp_rate<=2 : 0.996231, exp_rate<=3 : 0.998685, loss: 0.001811, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18513 s, avg_samples: 56.0, ips: 302.49104 samples/s, eta: 0:11:43, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:38:41] ppocr INFO: epoch: [499/500], global_step: 544400, lr: 0.000100, edit distance: 0.001153, exp_rate: 0.947353, exp_rate<=1 : 0.981068, exp_rate<=2 : 0.996244, exp_rate<=3 : 0.998688, loss: 0.001211, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18378 s, avg_samples: 56.0, ips: 304.71579 samples/s, eta: 0:11:24, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:39:00] ppocr INFO: epoch: [499/500], global_step: 544500, lr: 0.000100, edit distance: 0.001164, exp_rate: 0.946841, exp_rate<=1 : 0.980819, exp_rate<=2 : 0.996206, exp_rate<=3 : 0.998665, loss: 0.001516, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18880 s, avg_samples: 56.0, ips: 296.61591 samples/s, eta: 0:11:05, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:39:20] ppocr INFO: epoch: [499/500], global_step: 544600, lr: 0.000100, edit distance: 0.001177, exp_rate: 0.946446, exp_rate<=1 : 0.980662, exp_rate<=2 : 0.996163, exp_rate<=3 : 0.998663, loss: 0.001348, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19972 s, avg_samples: 56.0, ips: 280.39645 samples/s, eta: 0:10:46, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:39:39] ppocr INFO: epoch: [499/500], global_step: 544700, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.945937, exp_rate<=1 : 0.980463, exp_rate<=2 : 0.996068, exp_rate<=3 : 0.998618, loss: 0.001355, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18551 s, avg_samples: 56.0, ips: 301.87578 samples/s, eta: 0:10:27, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:39:57] ppocr INFO: epoch: [499/500], global_step: 544800, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.945894, exp_rate<=1 : 0.980357, exp_rate<=2 : 0.996058, exp_rate<=3 : 0.998600, loss: 0.001469, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18603 s, avg_samples: 56.0, ips: 301.02693 samples/s, eta: 0:10:08, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:40:16] ppocr INFO: epoch: [499/500], global_step: 544900, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.945671, exp_rate<=1 : 0.980305, exp_rate<=2 : 0.996033, exp_rate<=3 : 0.998614, loss: 0.001445, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19308 s, avg_samples: 56.0, ips: 290.03578 samples/s, eta: 0:09:49, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:40:35] ppocr INFO: epoch: [499/500], global_step: 545000, lr: 0.000100, edit distance: 0.001187, exp_rate: 0.945601, exp_rate<=1 : 0.980316, exp_rate<=2 : 0.996023, exp_rate<=3 : 0.998612, loss: 0.000819, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18187 s, avg_samples: 56.0, ips: 307.91976 samples/s, eta: 0:09:30, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:40:54] ppocr INFO: epoch: [499/500], global_step: 545100, lr: 0.000100, edit distance: 0.001197, exp_rate: 0.945269, exp_rate<=1 : 0.980143, exp_rate<=2 : 0.996004, exp_rate<=3 : 0.998622, loss: 0.001516, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18894 s, avg_samples: 56.0, ips: 296.39204 samples/s, eta: 0:09:11, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:41:12] ppocr INFO: epoch: [499/500], global_step: 545200, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.945403, exp_rate<=1 : 0.980039, exp_rate<=2 : 0.996036, exp_rate<=3 : 0.998653, loss: 0.001047, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18632 s, avg_samples: 56.0, ips: 300.55770 samples/s, eta: 0:08:52, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:41:23] ppocr INFO: epoch: [499/500], global_step: 545260, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.945280, exp_rate<=1 : 0.979940, exp_rate<=2 : 0.996039, exp_rate<=3 : 0.998657, loss: 0.001753, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10858 s, avg_samples: 33.6, ips: 309.45269 samples/s, eta: 0:08:40, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:41:25] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/11 07:41:34] ppocr INFO: epoch: [500/500], global_step: 545300, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.946717, exp_rate<=1 : 0.982143, exp_rate<=2 : 0.996917, exp_rate<=3 : 0.999047, loss: 0.000670, avg_reader_cost: 0.02653 s, avg_batch_cost: 0.10723 s, avg_samples: 22.4, ips: 208.89860 samples/s, eta: 0:08:33, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:41:51] ppocr INFO: epoch: [500/500], global_step: 545400, lr: 0.000100, edit distance: 0.000986, exp_rate: 0.955190, exp_rate<=1 : 0.984906, exp_rate<=2 : 0.997198, exp_rate<=3 : 0.999311, loss: 0.001295, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17605 s, avg_samples: 56.0, ips: 318.09497 samples/s, eta: 0:08:14, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:42:10] ppocr INFO: epoch: [500/500], global_step: 545500, lr: 0.000100, edit distance: 0.001103, exp_rate: 0.953532, exp_rate<=1 : 0.983413, exp_rate<=2 : 0.996819, exp_rate<=3 : 0.998891, loss: 0.001026, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18758 s, avg_samples: 56.0, ips: 298.54623 samples/s, eta: 0:07:55, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:42:28] ppocr INFO: epoch: [500/500], global_step: 545600, lr: 0.000100, edit distance: 0.001157, exp_rate: 0.951994, exp_rate<=1 : 0.983683, exp_rate<=2 : 0.996731, exp_rate<=3 : 0.998893, loss: 0.002232, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18152 s, avg_samples: 56.0, ips: 308.50683 samples/s, eta: 0:07:36, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:42:45] ppocr INFO: epoch: [500/500], global_step: 545700, lr: 0.000100, edit distance: 0.001170, exp_rate: 0.952098, exp_rate<=1 : 0.983762, exp_rate<=2 : 0.996596, exp_rate<=3 : 0.999004, loss: 0.001466, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16830 s, avg_samples: 56.0, ips: 332.73948 samples/s, eta: 0:07:17, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:43:06] ppocr INFO: epoch: [500/500], global_step: 545800, lr: 0.000100, edit distance: 0.001203, exp_rate: 0.949509, exp_rate<=1 : 0.982126, exp_rate<=2 : 0.995906, exp_rate<=3 : 0.998672, loss: 0.001899, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20216 s, avg_samples: 56.0, ips: 277.01196 samples/s, eta: 0:06:58, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:43:23] ppocr INFO: epoch: [500/500], global_step: 545900, lr: 0.000100, edit distance: 0.001162, exp_rate: 0.950097, exp_rate<=1 : 0.982171, exp_rate<=2 : 0.995941, exp_rate<=3 : 0.998713, loss: 0.001051, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17489 s, avg_samples: 56.0, ips: 320.20703 samples/s, eta: 0:06:39, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:43:42] ppocr INFO: epoch: [500/500], global_step: 546000, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.949474, exp_rate<=1 : 0.982019, exp_rate<=2 : 0.996086, exp_rate<=3 : 0.998807, loss: 0.001305, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18638 s, avg_samples: 56.0, ips: 300.46072 samples/s, eta: 0:06:20, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:44:00] ppocr INFO: epoch: [500/500], global_step: 546100, lr: 0.000100, edit distance: 0.001147, exp_rate: 0.948768, exp_rate<=1 : 0.981884, exp_rate<=2 : 0.996136, exp_rate<=3 : 0.998882, loss: 0.001188, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18230 s, avg_samples: 56.0, ips: 307.18516 samples/s, eta: 0:06:01, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:44:19] ppocr INFO: epoch: [500/500], global_step: 546200, lr: 0.000100, edit distance: 0.001152, exp_rate: 0.947877, exp_rate<=1 : 0.981495, exp_rate<=2 : 0.996032, exp_rate<=3 : 0.998898, loss: 0.001354, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19436 s, avg_samples: 56.0, ips: 288.12321 samples/s, eta: 0:05:42, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:44:38] ppocr INFO: epoch: [500/500], global_step: 546300, lr: 0.000100, edit distance: 0.001138, exp_rate: 0.947631, exp_rate<=1 : 0.981106, exp_rate<=2 : 0.995874, exp_rate<=3 : 0.998770, loss: 0.001608, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19046 s, avg_samples: 56.0, ips: 294.02571 samples/s, eta: 0:05:23, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:44:58] ppocr INFO: epoch: [500/500], global_step: 546400, lr: 0.000100, edit distance: 0.001140, exp_rate: 0.947654, exp_rate<=1 : 0.981092, exp_rate<=2 : 0.995893, exp_rate<=3 : 0.998769, loss: 0.001066, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19041 s, avg_samples: 56.0, ips: 294.10483 samples/s, eta: 0:05:04, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:45:16] ppocr INFO: epoch: [500/500], global_step: 546500, lr: 0.000100, edit distance: 0.001151, exp_rate: 0.947407, exp_rate<=1 : 0.980948, exp_rate<=2 : 0.995809, exp_rate<=3 : 0.998714, loss: 0.001773, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18455 s, avg_samples: 56.0, ips: 303.43737 samples/s, eta: 0:04:45, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:45:35] ppocr INFO: epoch: [500/500], global_step: 546600, lr: 0.000100, edit distance: 0.001166, exp_rate: 0.946012, exp_rate<=1 : 0.980329, exp_rate<=2 : 0.995672, exp_rate<=3 : 0.998689, loss: 0.001660, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19314 s, avg_samples: 56.0, ips: 289.93878 samples/s, eta: 0:04:26, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:45:54] ppocr INFO: epoch: [500/500], global_step: 546700, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.945036, exp_rate<=1 : 0.979783, exp_rate<=2 : 0.995460, exp_rate<=3 : 0.998598, loss: 0.001594, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18694 s, avg_samples: 56.0, ips: 299.55955 samples/s, eta: 0:04:07, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:46:13] ppocr INFO: epoch: [500/500], global_step: 546800, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.944590, exp_rate<=1 : 0.979552, exp_rate<=2 : 0.995407, exp_rate<=3 : 0.998592, loss: 0.001325, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18721 s, avg_samples: 56.0, ips: 299.13280 samples/s, eta: 0:03:48, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:46:30] ppocr INFO: epoch: [500/500], global_step: 546900, lr: 0.000100, edit distance: 0.001173, exp_rate: 0.945460, exp_rate<=1 : 0.979981, exp_rate<=2 : 0.995518, exp_rate<=3 : 0.998633, loss: 0.001428, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17378 s, avg_samples: 56.0, ips: 322.25488 samples/s, eta: 0:03:29, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:46:49] ppocr INFO: epoch: [500/500], global_step: 547000, lr: 0.000100, edit distance: 0.001177, exp_rate: 0.945055, exp_rate<=1 : 0.979775, exp_rate<=2 : 0.995470, exp_rate<=3 : 0.998607, loss: 0.001538, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19169 s, avg_samples: 56.0, ips: 292.13764 samples/s, eta: 0:03:10, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:47:09] ppocr INFO: epoch: [500/500], global_step: 547100, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.944564, exp_rate<=1 : 0.979582, exp_rate<=2 : 0.995483, exp_rate<=3 : 0.998559, loss: 0.001062, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19424 s, avg_samples: 56.0, ips: 288.29678 samples/s, eta: 0:02:51, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:47:28] ppocr INFO: epoch: [500/500], global_step: 547200, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.944307, exp_rate<=1 : 0.979473, exp_rate<=2 : 0.995489, exp_rate<=3 : 0.998594, loss: 0.001688, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18842 s, avg_samples: 56.0, ips: 297.21065 samples/s, eta: 0:02:32, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:47:47] ppocr INFO: epoch: [500/500], global_step: 547300, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.944162, exp_rate<=1 : 0.979422, exp_rate<=2 : 0.995499, exp_rate<=3 : 0.998578, loss: 0.001292, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19223 s, avg_samples: 56.0, ips: 291.31293 samples/s, eta: 0:02:13, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:48:06] ppocr INFO: epoch: [500/500], global_step: 547400, lr: 0.000100, edit distance: 0.001182, exp_rate: 0.944142, exp_rate<=1 : 0.979316, exp_rate<=2 : 0.995456, exp_rate<=3 : 0.998537, loss: 0.001031, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19309 s, avg_samples: 56.0, ips: 290.02163 samples/s, eta: 0:01:54, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:48:25] ppocr INFO: epoch: [500/500], global_step: 547500, lr: 0.000100, edit distance: 0.001186, exp_rate: 0.944289, exp_rate<=1 : 0.979395, exp_rate<=2 : 0.995437, exp_rate<=3 : 0.998515, loss: 0.001357, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18507 s, avg_samples: 56.0, ips: 302.58699 samples/s, eta: 0:01:35, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:48:45] ppocr INFO: epoch: [500/500], global_step: 547600, lr: 0.000100, edit distance: 0.001190, exp_rate: 0.944111, exp_rate<=1 : 0.979325, exp_rate<=2 : 0.995474, exp_rate<=3 : 0.998539, loss: 0.001545, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20124 s, avg_samples: 56.0, ips: 278.27629 samples/s, eta: 0:01:16, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:49:05] ppocr INFO: epoch: [500/500], global_step: 547700, lr: 0.000100, edit distance: 0.001191, exp_rate: 0.943875, exp_rate<=1 : 0.979108, exp_rate<=2 : 0.995415, exp_rate<=3 : 0.998517, loss: 0.000965, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19598 s, avg_samples: 56.0, ips: 285.74539 samples/s, eta: 0:00:57, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:49:24] ppocr INFO: epoch: [500/500], global_step: 547800, lr: 0.000100, edit distance: 0.001192, exp_rate: 0.943786, exp_rate<=1 : 0.979137, exp_rate<=2 : 0.995471, exp_rate<=3 : 0.998539, loss: 0.001463, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19170 s, avg_samples: 56.0, ips: 292.12228 samples/s, eta: 0:00:38, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:49:44] ppocr INFO: epoch: [500/500], global_step: 547900, lr: 0.000100, edit distance: 0.001188, exp_rate: 0.943784, exp_rate<=1 : 0.979215, exp_rate<=2 : 0.995479, exp_rate<=3 : 0.998557, loss: 0.001712, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20184 s, avg_samples: 56.0, ips: 277.44248 samples/s, eta: 0:00:19, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:50:03] ppocr INFO: epoch: [500/500], global_step: 548000, lr: 0.000100, edit distance: 0.001194, exp_rate: 0.943579, exp_rate<=1 : 0.979110, exp_rate<=2 : 0.995476, exp_rate<=3 : 0.998571, loss: 0.001448, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19362 s, avg_samples: 56.0, ips: 289.22115 samples/s, eta: 0:00:00, max_mem_reserved: 30359 MB, max_mem_allocated: 25023 MB [2024/07/11 07:50:05] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/11 07:50:06] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_500 [2024/07/11 07:50:06] ppocr INFO: best metric, exp_rate: 0.41323024054982815, is_float16: False, edit distance: 0.0982085949333527, exp_rate<=1 : 0.7014604810996563, exp_rate<=2 : 0.8552405498281787, exp_rate<=3 : 0.9248281786941581, fps: 7.411227677740838, best_epoch: 99, start_epoch: 301