Gpu support

[windreamer]

@mckelvin pls review

@bhack this is what i can guess for a GPU cluster

cf: #1

[windreamer]

Note: Mesos should be configured as below:
NVIDIA/nvidia-docker#60 (comment)

[mckelvin]

👍

@bhack Would you please try this? Since we don't have GPU cluster yet.

[bhack]

Yes sure. /cc @mtamburrano @lenlen

[xunzhang]

🍺

[windreamer]

a major issue i found is TF always use the first a few GPU cores, so when u get several TF compute at the same time, they will all run in the same cores, with other cores in idle.

i can not find a way to specify GPU UUIDs of a TF programs to run for now.

cf: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/common_runtime/gpu/gpu_device.cc#L553

[wsxiaozhang]

I'm curious which mesos release you are working on for GPU resource. Seems mesos hasn't finished https://issues.apache.org/jira/browse/MESOS-4424

[windreamer]

@wsxiaozhang we can add GPU cores as custom resources for Mesos before MESOS-4424 ships.
of coz it is not perfect, no isolator for now, but barely usable i think.

[bhack]

@windreamer With manual placement GPU can be addresses with id i.e. in the form of /gpu:0. See https://www.tensorflow.org/versions/r0.8/how_tos/using_gpu/index.html

[windreamer]

@bhack yes, but imagine u have 2 TF programs running, ur machine have 4 GPUS. Ideally we allocate 2 GPUs to the first program as "gpu:0" and "gpu:1" and the other 2 GPUs to the other program also as "gpu:0" and "gpu:1".

But in fact, the two program will both use GPU0 and GPU1, and GPU2, GPU3 will be idle.

[bhack]

But the id of the GPU free resources offered by mesos don't change? Or Is it ignored by the sequential internal cycle of TF?

[windreamer]

it is ignored by TF, TF only cares about the count. u can see the TF code i mentioned. no gpu-id is aware there unfortunately. maybe a docker isolator can do the magic to reassign ids to allocated gpus? but i'm not optimistic.

[bhack]

Do you mean i.e. that magically what it is presented as GPU 0 in the container it is GPU 3 in the host etc..?

[bhack]

Probably @3XX0 could give us some hint on this for nvidia-docker.

[3XX0]

You can use nvidia-docker instead of docker in the mesos-slave configuration.
Regarding Mesos resources, either use the GPGPU support in Mesos upstream

http://c99.millennium.berkeley.edu/documentation/latest/gpu-support/
https://issues.apache.org/jira/browse/MESOS-4424

or if you need more information about which GPUs to pick, use the custom resource advertised by nvidia-docker-plugin:

NVIDIA/nvidia-docker#60

[bhack]

In the link I read that "Filesystem isolation must be disabled. This means No Docker Support at the moment. Only the mesos containerizer can currently be used."

[3XX0]

No Docker support through the Mesos containerizer.
If you use the Docker containerizer with nvidia-docker or the Mesos containerizer without Docker and without filesystem isolation you're fine.

[windreamer]

@3XX0 Does nvidia-docker re-map GPUs inside containers? For example, /dev/nvidia3 in host machine will be /dev/nvidia0 inside container?

[windreamer]

According this anwser: http://stackoverflow.com/a/26568684
Seems we can do the re-mapping in tfmesos
But we should use gpu id instead of gpu uuid for resources.

[windreamer]

@bhack some idea updated here. Note: mesos gpu resources should be defined as below:

--resources=gpus:{0, 1, 2, ...}

[bhack]

As @3XX0 confirmed seems to me that with current limitation GPU cannot work in the case we set DOCKER_IMAGE env variable. Is this right?

[windreamer]

@bhack seems we can use Docker containerizer with nvidia-docker

[bhack]

"If you use the Docker containerizer with nvidia-docker or the Mesos containerizer without Docker and without filesystem isolation you're fine." This seems to me that on slave we cannot run Task inside docker or nvidia-docker containers. Do you have another interpretation?

[windreamer]

My interpretation is

If you use the Docker containerizer with nvidia-docker you're fine.

or

If you use the Mesos containerizer without Docker and without filesystem isolation you're fine.

so I think we can go with the first option.

[bhack]

If your interpretation is correct how can we let the mesos slave to use nvidia-docker? I think that slave simply invoke simply docker command when your slave run with --containerizers=docker,mesos

[windreamer]

@bhack nvidia-docker seems a plugin for docker provided by Nvidia for CUDA.
cf: https://github.com/NVIDIA/nvidia-docker/wiki/Installation

but I have no experience to build up a Mesos cluster with CUDA support, I am not sure about it.

[windreamer]

cf: NVIDIA/nvidia-docker#8

[3XX0]

@windreamer I just added support for UUID in NV_GPU, it should simplify the index lookup: NVIDIA/nvidia-docker@1dff2bb

And the exact Mesos slave command is:

mesos-slave --containerizers=docker,mesos \
            --docker=/usr/bin/nvidia-docker \
            --executor_environment_variables='{"NV_DOCKER": "/usr/bin/docker"}' \
            $(curl -s localhost:3476/mesos/cli)

[windreamer]

@lenlen thanks for the feed back, I've fix the errors you mentioned and commit to this PR, please review my fix and try again if it is OK to you.

libcudnn.so problem is a bit weird to me. I will try to investigate this problem and fix in dockerfile

further more can you provide more information abort how the task is failed when more gpus are allocated ?

[windreamer]

@3XX0 bravo! this is really convenient. thank you for this work.

but at present I may temporarily keep the code this way, I will simpfy the whole process when nvidia-docker v1.0 releases ASAP

[vitan]

@3XX0

You can use nvidia-docker instead of docker in the mesos-slave configuration.

Here I have a concern to the above given solution. The above configuration will make all of containers deployed by mesos use the nvidia-docker instead docker, isn't it? And I don't think it is a general solution.

[3XX0]

Yes and? It is a general solution until we upstream all the work in Mesos itself (which might take a while).
Right now in Mesos, we only support GPUs as a scalar with the Mesos containerizer and without filesystem isolation (see presentation).

[windreamer]

@bhack @3XX0 finally we go some GPU instances in AWS and are able to test this PR. and I made some changes:

1. upgrade tensorflow to 0.9 and upgrade other dependencies
2. thanks to nvidia-docker=1.0.0.rc.3, we can use UUIDs to map GPU into docker, and no longer need to decompress gpu infos
3. because when NV_GPU is empty, nvidia_docker map all gpu into the docker, but we are expecting NO GPU should be mapped, we try to use nvidia-docker-plugin to get docker parameters from remote and this also enable us to setup a cluster mixing gpu nodes and cpu nodes.

so this is what we currently get, and we are planning to deliver a simple script to help setup an CPU/GPU cluster of AWS G2 ubuntu nodes later.

pls feel free to comment, any suggestion is welcome.
thx

[bhack]

I think that you can expose this news in the related TF ticket.

[bhack]

@windreamer What about https://issues.apache.org/jira/plugins/servlet/mobile#issue/MESOS-540?

[windreamer]

@bhack sorry, anything related to this ?

[bhack]

Sorry, I've accidentally removed the last digit from URL... It is https://issues.apache.org/jira/plugins/servlet/mobile#issue/MESOS-5401

[windreamer]

@bhack OK, this is not a problem, MESOS-5401 is to resolve isolation problem in mesos containerizer, not in docker containerizer. we are using nvidia-docker-plugin, so far so good.

[bhack]

Probably it is part of unified containers that noto require Docker daemon anymore in 1.0. See http://mesos.apache.org/blog/mesos-1-0-0-released/

[windreamer]

Hmm, unified containers seems just we can inject new container implementation without changing Mesos code. when we need docker, we still need docker daemon. further more, if we need GPU support, i will choose nvidia docker not the embedded GPU support from mesos.

[bhack]

Are you sure? I'havent tried yet new Mesos 1.0 but i read

With the unified containerizer, we can now use a single container runtime engine, the MesosContainerizer to run containers off the most popular image formats (Docker, Appc, and soon OCI) without dependency on any external daemons.

[bhack]

And

As a bonus, we mimic the functionality of nvidia-docker so you can take your existing GPU docker containers and run them on Mesos without modification.

[windreamer]

from my [incomplete] code dig, YES, we still need docker installed.
cf: https://github.com/apache/mesos/blob/master/docs/mesos-containerizer.md#docker-runtime-isolator

[windreamer]

frankly speaking, i think everything is a MESS to me now: docker are integrating swarm to try beat mesos down, and mesos are supporting more container type and even try to parse docker image to control the launch process. why we can not just "God to God, Caesar to Caesar" ?

at present, i believe we should do containerization with docker, isolate GPUs using nvidia-docker, and organise containers using mesos. and for future? God knows!

[bhack]

@windreamer Mesos added native docker image support and nvidia-docker coverage so I think that with mesos 1.0 we don't require docker daemon anymore.
Basically? It is a war!

[windreamer]

OK, i think i found the so called native docker https://github.com/apache/mesos/blob/master/docs/docker-volume.md

this whole thing is too complicated, i do not think i want to give this a try.

[bhack]

There are some appreciable motivations. Read the introduction at http://mesos.apache.org/documentation/latest/container-image/