MONAI Seg Inference Operator Shared Memory

[slbryson]

monai-deploy-app-sdk/monai/deploy/operators/monai_seg_inference_operator.py

Line 211 in 8ec336f

dataloader = DataLoader(dataset, batch_size=1, shuffle=False, num_workers=1)

Applications that use the Dataloader in MONAI Seg Inference Operator throw errors when trying to allocate shared memory in Argo pipelines.

Need to modify the num of workers argument to not use shared memory by default.

[MMelQin]

I changed this from bug to the closest label that I can find, because using shared_mem in dataloader is NOT a bug.

It had become known that when the application is packaged in a MAP (docker image) and some host platforms do not have the capability to set shared mem size on launching the MAP docker, e.g. Clara Deploy 0.8.1 (it is further noted that the default docker shared_mem size is 64MB while this can be modified in Docker daemon config), the MAP docker execution would fail on lack of shared_mem.

In order to allow the MAP docker to run even when the host platform does not bump the shared_mem size, shared_mem use is to be disabled as a workaround. Setting the num_workers to 0 may work per this discussion.

[slbryson]

I agree. It is not a bug, but I couldn't find a better label. Enhancement would have been a better choice. Thank you.

…

On Wed, Apr 6, 2022, 12:14 AM Ming M Qin ***@***.***> wrote: I changed this from bug to the closest label that I can find, because using shared_mem in dataloader is NOT a bug. It had become known that when the application is packaged in a MAP (docker image) and some host platforms do not have the capability to set shared mem size on launching the MAP docker, e.g. Clara Deploy 0.8.1 (it is further noted that the default docker shared_mem size is 64MB while this can be modified in Docker daemon config), the MAP docker execution would fail on lack of shared_mem. In order to allow the MAP docker to run even when the host platform does not bump the shared_mem size, shared_mem use is to be disabled as a workaround. Setting the num_workers to 0 may work per this discussion <https://discuss.pytorch.org/t/can-we-disable-share-memory-usage-in-pytorch/47342> . — Reply to this email directly, view it on GitHub <#283 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSWDBOBC4CIXIK57CPOK43VDUMRZANCNFSM5SURW5UA> . You are receiving this because you were assigned.Message ID: ***@***.***>

[MMelQin]

Some additional information from NGC on the pytorch base image (for MAP) with regard to shared memory use in torch dataloader, for reference only when we re-enable shared memory use and have hosting platform capable of supporting the docker settings.

Note: DIGITS uses shared memory to share data between processes. For example, if you use Torch multiprocessing for multi-threaded data loaders, the default shared memory segment size that the container runs with may not be enough. Therefore, you should increase the shared memory size by issuing either:

  --ipc=host

or

  --shm-size=

[aihsani]

I move to file another enhancement for MONAI app sdk to parameterize this via an environment variable or programmatically so that we may choose whether to use the feature or not. There's an obvious drawback to not using a variable number of workers.

[MMelQin]

Thanks. Given the current use case, i.e. inference workflow limited to one image fully loaded before progressing, there is not much difference, but for sure configurability is definitely good.
We have created an issue to support MONAI Bundle in the next release with the highest priority, and this operator will likely see major changes, but the class API will be backward compatible as much as feasible. Will supply an optional parameter for num_workers on the constructor, with the same semantics as the one on torch dataloader.

[aihsani]

Great, thank you