(torchx/scheduler) support mounts for docker_scheduler

[kiukchung]

Description

Support a runopt in docker_scheduler to specify volume mounts

Motivation/Background

Additional run option that is useful to the end user who wants to run torchx run -s local_docker with an optional volume mount for data

Detailed Proposal

1. Need to add volume/bind-mount specifications. Options:

   1. Add to Role. As Role.mount parameter (similar to Role.port: https://github.com/pytorch/torchx/blob/main/torchx/specs/api.py#L257) or as Role.metadata
   2. Add it as runopt on the local_docker and kubernetes schedulers

2. Add it to docker request here:

   torchx/torchx/schedulers/docker_scheduler.py

   Line 207 in 571fd78

   c = DockerContainer(

Alternatives

N/A

Additional context/links

• https://pytorch.slack.com/archives/C0345N79QMP/p1646851975420669

[d4l3k]

Looks like we can support the same mounts for docker, k8s and aws batch via similar interfaces.

• docker mount https://docker-py.readthedocs.io/en/stable/containers.html#docker.models.containers.ContainerCollection.run
• ECS/Batch mount points https://aws.amazon.com/premiumsupport/knowledge-center/batch-fsx-lustre-file-system-mount/
• Kubernetes volumes https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Volume.md https://aws.amazon.com/blogs/opensource/using-fsx-lustre-csi-driver-amazon-eks/