MUGE微调报错：ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7)

[Guan-Y]

在MUGE数据集上做微调时报错，在Flickr上正常运行，参数设置均一样，只有数据集不同，请问是什么原因？

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 1173017) of binary: /home/ma-user/anaconda3/envs/PyTorch-1.11.0/bin/python3
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/PyTorch-1.11.0/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/ma-user/anaconda3/envs/PyTorch-1.11.0/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/anaconda3/envs/PyTorch-1.11.0/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/ma-user/anaconda3/envs/PyTorch-1.11.0/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/ma-user/anaconda3/envs/PyTorch-1.11.0/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/ma-user/anaconda3/envs/PyTorch-1.11.0/lib/python3.7/site-packages/torch/distributed/run.py", line 718, in run
)(*cmd_args)
File "/home/ma-user/anaconda3/envs/PyTorch-1.11.0/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/ma-user/anaconda3/envs/PyTorch-1.11.0/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

cn_clip/training/main.py FAILED

Failures:
[1]:
time : 2024-09-12_10:09:54
host : notebook-be63e7be-de96-44af-a249-23010f6c1f29.notebook-be63e7be-de96-44af-a249-23010f6c1f29-distributed.default.svc.cluster.local
rank : 1 (local_rank: 1)
exitcode : -7 (pid: 1173029)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 1173029

Root Cause (first observed failure):
[0]:
time : 2024-09-12_10:09:54
host : notebook-be63e7be-de96-44af-a249-23010f6c1f29.notebook-be63e7be-de96-44af-a249-23010f6c1f29-distributed.default.svc.cluster.local
rank : 0 (local_rank: 0)
exitcode : -7 (pid: 1173017)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 1173017