报错BrokenPipeError: [Errno 32] Broken pipe，完整报错如下，请问这是哪里的问题

[pandazzh2020]

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

/opt/conda/envs/huatuo/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /opt/conda/envs/huatuo did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/opt/conda/envs/huatuo/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/extras/CUPTI/lib64'), PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/cuda/compat/lib'), PosixPath('/usr/local/nvidia/lib64')}
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 114
CUDA SETUP: Loading binary /opt/conda/envs/huatuo/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda114.so...
Training Alpaca-LoRA model with params:
base_model: decapoda-research/llama-7b-hf
data_path: ./data/Format_data_sheet_mini.json
output_dir: ./lora-llama-med-e1
batch_size: 1
micro_batch_size: 1
num_epochs: 5
learning_rate: 0.0003
cutoff_len: 256
val_set_size: 500
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: ['q_proj', 'v_proj']
train_on_inputs: False
group_by_length: False
wandb_project: llama_med
wandb_run_name: e1
wandb_watch:
wandb_log_model:
resume_from_checkpoint: False
prompt template: med_template

The model weights are not tied. Please use the tie_weights method before using the infer_auto_device function.
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [00:22<00:00, 1.44it/s]
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'LLaMATokenizer'.
The class this function is called from is 'LlamaTokenizer'.
Found cached dataset json (/root/.cache/huggingface/datasets/json/default-51d7aed489ad1911/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 376.91it/s]
trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199
Loading cached split indices for dataset at /root/.cache/huggingface/datasets/json/default-51d7aed489ad1911/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e/cache-9f4de28c7bc88c4b.arrow and /root/.cache/huggingface/datasets/json/default-51d7aed489ad1911/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e/cache-b165ac8522c98057.arrow
wandb: W&B API key is configured. Use wandb login --relogin to force relogin
wandb: Tracking run with wandb version 0.15.5
wandb: Run data is saved locally in /zheng_zhong_hua/Huatuo-llama-med-chinese/wandb/run-20230722_105219-wnnb9l4i
wandb: Run wandb offline to turn off syncing.
wandb: Syncing run e1
wandb: ⭐️ View project at https://wandb.ai/chat2023/llama_med
wandb: 🚀 View run at https://wandb.ai/chat2023/llama_med/runs/wnnb9l4i
0%| | 0/15535 [00:00<?, ?it/s]/opt/conda/envs/huatuo/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py:298: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
0%| | 2/15535 [00:03<7:23:53, 1.71s/it]wandb: Network error (TransientError), entering retry loop.
{'loss': 2.2941, 'learning_rate': 9.65250965250965e-07, 'epoch': 0.0}
{'loss': 2.1901, 'learning_rate': 2.5096525096525096e-06, 'epoch': 0.01}
{'loss': 2.6166, 'learning_rate': 4.054054054054054e-06, 'epoch': 0.01}
0%|▏ | 29/15535 [00:29<4:12:00, 1.03it/s]Exception in thread ChkStopThr:
Traceback (most recent call last):
File "/opt/conda/envs/huatuo/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/opt/conda/envs/huatuo/lib/python3.9/threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 273, in check_stop_status
Exception in thread NetStatThr:
Traceback (most recent call last):
self._loop_check_status(
File "/opt/conda/envs/huatuo/lib/python3.9/threading.py", line 980, in _bootstrap_inner
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 211, in _loop_check_status
self.run()
local_handle = request()
File "/opt/conda/envs/huatuo/lib/python3.9/threading.py", line 917, in run
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface.py", line 787, in deliver_stop_status
return self._deliver_stop_status(status)
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface_shared.py", line 585, in _deliver_stop_status
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 255, in check_network_status
return self._deliver_record(record)
self._loop_check_status(
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface_shared.py", line 560, in _deliver_record
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 211, in _loop_check_status
handle = mailbox._deliver_record(record, interface=self)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
local_handle = request()
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface.py", line 795, in deliver_network_status
interface._publish(record)
return self._deliver_network_status(status)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface_shared.py", line 601, in _deliver_network_status
return self._deliver_record(record)
self._sock_client.send_record_publish(record)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface_shared.py", line 560, in _deliver_record
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
self.send_server_request(server_req)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
handle = mailbox._deliver_record(record, interface=self)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
self._send_message(msg)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
interface._publish(record)
self._sendall_with_error_handle(header + data)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
self._sock_client.send_record_publish(record)
sent = self._sock.send(data)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
BrokenPipeError: [Errno 32] Broken pipe
self.send_server_request(server_req)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
self._send_message(msg)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
self._sendall_with_error_handle(header + data)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
0%|▎ | 32/15535 [00:32<4:04:48, 1.06it/s]Traceback (most recent call last):
File "/zheng_zhong_hua/Huatuo-llama-med-chinese/finetune.py", line 289, in
fire.Fire(train)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/zheng_zhong_hua/Huatuo-llama-med-chinese/finetune.py", line 279, in train
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/transformers/trainer.py", line 2307, in _maybe_log_save_evaluate
self.log(logs)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/transformers/trainer.py", line 2672, in log
self.control = self.callback_handler.on_log(self.args, self.state, self.control, logs)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/transformers/trainer_callback.py", line 390, in on_log
return self.call_event("on_log", args, state, control, logs=logs)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/transformers/trainer_callback.py", line 397, in call_event
result = getattr(callback, event)(
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/transformers/integrations.py", line 814, in on_log
self._wandb.log({**logs, "train/global_step": state.global_step})
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 389, in wrapper
return func(self, *args, **kwargs)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 340, in wrapper_fn
return func(self, *args, **kwargs)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 330, in wrapper
return func(self, *args, **kwargs)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 1745, in log
self._log(data=data, step=step, commit=commit)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 1526, in _log
self._partial_history_callback(data, step, commit)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 1396, in _partial_history_callback
self._backend.interface.publish_partial_history(
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface.py", line 584, in publish_partial_history
self._publish_partial_history(partial_history)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface_shared.py", line 89, in _publish_partial_history
self._publish(rec)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
self._sock_client.send_record_publish(record)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
self.send_server_request(server_req)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
self._send_message(msg)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
self._sendall_with_error_handle(header + data)
File "/opt/conda/envs/huatuo/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
wandb: While tearing down the service manager. The following error has occurred: [Errno 32] Broken pipe

[pandazzh2020]

batch_size: 1 调小后未解决
micro_batch_size: 1
num_epochs: 5