UnicodeDecodeError fails the job

[phtu-cs]

I often got the following error message. But the error does not always occur. If I resubmit the exact same job, it can probably run without any errors. This is very weird. Normally, the error should always occur as long as I am submitting the same job. Could you help me with this? Thank you!

submitit ERROR (2022-09-26 00:00:00) - Submitted job triggered an exception
Traceback (most recent call last):
File "/fs/nexus-scratch/phtu/env/lib/python3.9/site-packages/submitit/core/utils.py", line 334, in call
copy_process_streams(process, stdout_buffer, stderr_buffer, self.verbose)
File "/fs/nexus-scratch/phtu/env/lib/python3.9/site-packages/submitit/core/utils.py", line 266, in copy_process_streams
buf = raw_buf.decode()
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1400-1401: unexpected end of data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/local/stow/Python3-3.9.5/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/local/stow/Python3-3.9.5/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/fs/nexus-scratch/phtu/env/lib/python3.9/site-packages/submitit/core/_submit.py", line 11, in
submitit_main()
File "/fs/nexus-scratch/phtu/env/lib/python3.9/site-packages/submitit/core/submission.py", line 72, in submitit_main
process_job(args.folder)
File "/fs/nexus-scratch/phtu/env/lib/python3.9/site-packages/submitit/core/submission.py", line 65, in process_job
raise error
File "/fs/nexus-scratch/phtu/env/lib/python3.9/site-packages/submitit/core/submission.py", line 54, in process_job
result = delayed.result()
File "/fs/nexus-scratch/phtu/env/lib/python3.9/site-packages/submitit/core/utils.py", line 133, in result
self._result = self.function(*self.args, **self.kwargs)
File "/fs/nexus-scratch/phtu/env/lib/python3.9/site-packages/submitit/core/utils.py", line 338, in call
raise FailedJobError("Job got killed for an unknown reason.") from e
submitit.core.utils.FailedJobError: Job got killed for an unknown reason.

[gwenzek]

are you using submitit.core.utils.CommandFunction ? if yes, which command are you submitting ?

[chongyi-zheng]

I got the same error when I executing my own python program using submitit.core.utils.CommandFunction and this error only happened occasionally. Any idea?