How to add the -- shm-size parameter at runtime

[kongxiangya]

Shm errors in running deep learning models within a docker container

Error: Unexpected bus error accounted in worker This might be caused by insufficient shared memory (shm).

Reason: The main reason is that the shm size is too small

When creating an analysis container using cwltool, it is expected to add the parameter – shm-size 4g through configuration

docker run --shm-size 4G

Can you add this configuration? Thank you

[tetron]

This is interesting, I was not familiar with this option.

I'd support adding something like a ShmSize requirement to specify that /dev/shm is needed and that should be a certain size, with the restriction that it can only be used by the CommandLineTool or sub processes launched by it within the container (no communication with processes outside the container).

I would accept a pull request to do something like this. Adding a new extension is slightly involved but if you are interested in doing the work to implement it we can walk you through it.

[kongxiangya]

Thank you for your reply

Should this configuration addition method be similar to --strict-memory-limit and used in conjunction with ramMin in the cwl configuration file. For example, add --set-shm-size, and add the shmSize setting value in the requirements section of the cwl configuration file

Or similar to --no-read-only, but increase the parameter setting value. For example, add --set-shm-size 5g to add this configuration when running the tool

[tetron]

It should be part of the requirements section of the CWL file. example:

class: CommandLineTool
requirements:
  cwltool:ShmSize:
    shmSize: 4096

The gist of what you need to do is:

1. Add the schema to cwltool/extensions-v1.2.yml
2. Add it to the list of supportedProcessRequirements in cwltool/process.py
3. In DockerCommandLineJob.create_runtime in cwltool/docker.py call self.builder.get_requirement("http://commonwl.org/cwltool#ShmSize") and use that to add --shm-size to the command line
4. Add a test in tests/test_docker.py

[mr-c]

Is there a Singularity/Apptainer equivalent to this? Likewise podman, udocker, ...

[kongxiangya]

Hi @tetron

Thanks for your answer

Due to limited capabilities and time, I use the method of adding from the outermost layer to quickly meet my current needs. If other researchers have this requirement, you can consider whether to develop this configuration.

My changes are:

1. argparser.py, Add under arg_parser

   parser.add_argument(
            "--set-shm-size",
            type=str,
            help="Set ShmSize, default 64m",
            default="64m",
            dest="set_shm_size"
    )

2. context.py, Add under RuntimeContext

self.set_shm_size = "64m"  # type: str

3. docker.py, Add under create_runtime

runtime.extend(["--shm-size",runtimeContext.set_shm_size])

Finally, add configuration at runtime:

cwltool_run --set-shm-size 5g

[kongxiangya]

Is there a Singularity/Apptainer equivalent to this? Likewise podman, udocker, ...

Hi @mr-c

Do you mean whether podman and udocker also have a --shm-size configuration ?

Due to restrictions on the use of udocker, I only tested podman, and the results also support this configuration.