Devices Flag not Set if No Volumes Needed

[cancan101]

Right now GenerateLocalArgs does not get called if no volumes are needed which means the devices flag is not set.

This causes issues for images that expect the CUDA devices to exist. (see for example Tensorflow).

This is different from the original version which did set the flag.

[3XX0]

This is an intended behavior as non-GPU images shouldn't have devices set.
Moreover, device files are unusable without their corresponding volumes.

Tensorflow uses the official CUDA images which sets the volumes.needed label. You are probably encountering this issue because you used an outdated Tensorflow image.

[cancan101]

If you take a look at: https://github.com/tensorflow/tensorflow/blob/4c717c6ac35519e6d5eef11b9b5384847e5fdd89/tensorflow/stream_executor/cuda/cuda_diagnostics.cc#L100-L112 you can see tensorflow is explicitly looking for the /dev/.

[flx42]

The problem is that the Tensorflow 0.6.0 images were built with an older version of nvidia-docker and cuda images that didn't include the volume tags.
If you use nvidia-docker with this image, it will be interpreted as a CPU image.

You should either rebuild the Tensorflow image yourself or use the official instructions for Tensorflow (which mean exporting 2 environment variables, not recommended because it's not portable).
For the next Tensorflow release, they will probably regenerate images based on our new images.

[3XX0]

@cancan101: The code you linked is in the CUDA code path, so devices should be there at that point.
However, this code relies on /proc/driver/nvidia/version which might not be there depending on the driver version you are using and /dev/nvidia0 which might not be in the container if you didn't isolate it (device files are not renumbered).