OSError: [Errno 30] Read-only file system: 'outputs'

[starlitsky2010]

Hi NVIDIA,

enroot 3.4.1
pyxis 0.7.0
slurm slurm-wlm 19.05.5
Ubuntu 20.04

When I execute this command
~/container_tools/NeMo-Megatron-Launcher/launcher_scripts# python3 main.py

The following occurs:

slurmstepd-user: pyxis: creating container filesystem ...
slurmstepd-user: pyxis: starting container ...
/usr/local/lib/python3.8/dist-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (5.1.0)/charset_normalizer (3.1.0) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
/root/container_tools/NeMo-Megatron-Launcher/launcher_scripts/nemo_launcher/collections/dataprep_scripts/pile_dataprep/download.py:22: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path="conf", config_name="config")
Traceback (most recent call last):
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'outputs/2023-06-09/07-06-52'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'outputs/2023-06-09'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/container_tools/NeMo-Megatron-Launcher/launcher_scripts/nemo_launcher/collections/dataprep_scripts/pile_dataprep/download.py", line 63, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 453, in <lambda>
    lambda: hydra.run(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 119, in run
    ret = run_job(
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 146, in run_job
    Path(str(output_dir)).mkdir(parents=True, exist_ok=True)
  File "/usr/lib/python3.8/pathlib.py", line 1292, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/lib/python3.8/pathlib.py", line 1292, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
OSError: [Errno 30] Read-only file system: 'outputs'
srun: error: user: task 0: Exited with exit code 1
~                                                       

I've print out the output_dir path:

output_dir is: ----------------- .
It means the current dir is output_dir.

I also print out some other path:

print(f"=========== {output_dir} {job_dir_key} {old_cwd}")
=========== . hydra.run.dir /root/container_tools/NeMo-Megatron-Launcher/launcher_scripts

I've also created outputs directory under launcher_scripts.

Thanks
Aaron