Add services and shell script to install artifacts in CoreOS #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dashesy
force-pushed
the
installer
branch
3 times, most recently
from
73fd70e to
01521ee
Compare
dashesy
force-pushed
the
installer
branch
2 times, most recently
from
9957664 to
c2fec90
Compare
mumoshu
pushed a commit
to kubernetes-retired/kube-aws
that referenced
this pull request
May 22, 2017
AWS offers Nvidia GPU ready instance type families (P2 and G2). And, of course Kubernetes supports GPU resource scheduling since 1.6. However Nvidia drivers is not installed in default coreos ami used in kube-aws. Then, let's support it! This implements auto installation support of Nvidia GPU driver. Some driver installation script are borrowed from [/Clarifai/coreos-nvidia](https://github.com/Clarifai/coreos-nvidia/). ## Design summary ### Configuration and what will happen New configuration for this feature is really simple. `worker.nodePool[i].gpu.nvidia.{enabled,version}` is introduced in `cluster.yaml`. - default value of `enabled` is false. - user will be warned if - user set `enabled: true` when `instanceType` doesn't support GPU. In this case the configuration will be ignored. - user set `enabled: false` when `instanceType` does support GPU - when `enabled: true` on GPU supported instance type, - nvidia driver will be installed automatically in each node in the nodepool. - The installation will happen just before `kubelet.service` starting (see below). - And, `kubelet` will start with [`--feature-gates="Accelerators=true"`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L212-L214) - then container can mount nvidia driver [like this](https://gist.github.com/everpeace/9e03050467d5ef5f66b7ce96b5fefa72#file-pod-yaml-L30-L53) - several tags are assigned to the node for enabling schedule on appropriate GPU model and its driver version by using `nodeAffinity`. - `alpha.kubernetes.io/nvidia-gpu-name=<GPU hardware type name>` - `kube-aws.coreos.com/gpu=nvidia`, - `kube-aws.coreos.com/nvidia-gpu-version=<version>` - Because substitution are not used in unit definition, I introduced `/etc/default/kubectl` for defining these label values in [this commit](5c59944). ### Driver installation process Most of installation script is borrowed from [/Clarifai/coreos-nvidia](https://github.com/Clarifai/coreos-nvidia/). Especially, for device node installation, I referenced to Clarifai/coreos-nvidia#4 . I just described summary of installation process. - [`kubelet.service`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L144-L147) ruires [`nvidia-start.service`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L456-L471) - [`nvidia-start.service`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L456-L471) invokes [`build-and-install.sh`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L918-L947), which installs nvidia drivers and kernel module files, via `ExecStartPre`. `nvidia-start.service` will create device nodes(`nvidiactl` and `nvidia0,1,...`). Other dynamic device nodes are controlled by`udevadam` (configuration is in [this rule file](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L905-L939)) - `nvidia-start.service` is `type=oneshot` because `kubelet.service` should wait until `nvidia-start.sh` completely succeeded. - `Restart` policy cannot be used with`type=oneshot`. `nvidia-start.service` doesn't use systemd's retry feature is not used but manual `retry.sh` is used. - [nvidia-persistenced](https://docs.nvidia.com/deploy/driver-persistence/#persistence-daemon) is also enabled for speeding up startup. this service is started/stopped via `udevadam` too. ## How to try 1. build `kube-aws` on this branch 2. `kube-aws up` with minimal nodepool configuration below ``` worker: nodePools: - name: p2xlarge count: 1 instanceType: p2.xlarge rootVolume: size: 30 type: gp2 gpu: nvidia: enabled: true version: "375.66" ``` 3. check `kubectl get nodes --show-labels`. Then you'll see one node with gpu related labels. 4. try starting this [pod](https://gist.github.com/everpeace/9e03050467d5ef5f66b7ce96b5fefa72#file-pod-yaml) ``` kubectl create -f pod.yaml ``` 5. log reports sample matrix multiplication is computed on gpus. ``` kubectl logs gpu-pod ``` ## Full changelog * add /etc/default/kubelet to worker nodes. * add nvidia driver installation support. * add gpu related config test. * it should be error when user gpu.nvidia.true with GPU unspported intance types. This change is caused by: #645 (comment) * add note which warns that driver may stop working when OS is updated. This change is caused by: #645 (comment) * move nvidia-{start, persisntenced}.service to `coreos.units` section. creation for nvidia-persistenced user to `users` section, too. This change is caused by: #645 (comment) * introduce unit dependency: kubelet --> nvidia-start --> nvidia-install deleted `systemctl` command from bash script. Instead, above unit dependency is introduced. nvidia-install.service, which just invokes build-and-install.sh is implemented type=oneshot because nvidia-start should wait until nvidia-install.service successed completely. Enabling retry build-and-install.sh, /opt/nvidia-build/util/retry.sh is introduced. It is because type=oneshot and Restart=always can't be used in systemd. * delete nvidia-install.service and now nvidia-start.service invoke build-and-install.sh via ExecStartPre with retry.sh kubelet.service 'Requires' and 'After' nvidia-star.service.
kylehodgetts
pushed a commit
to HotelsDotCom/kube-aws
that referenced
this pull request
Mar 27, 2018
…ed#645) AWS offers Nvidia GPU ready instance type families (P2 and G2). And, of course Kubernetes supports GPU resource scheduling since 1.6. However Nvidia drivers is not installed in default coreos ami used in kube-aws. Then, let's support it! This implements auto installation support of Nvidia GPU driver. Some driver installation script are borrowed from [/Clarifai/coreos-nvidia](https://github.com/Clarifai/coreos-nvidia/). ## Design summary ### Configuration and what will happen New configuration for this feature is really simple. `worker.nodePool[i].gpu.nvidia.{enabled,version}` is introduced in `cluster.yaml`. - default value of `enabled` is false. - user will be warned if - user set `enabled: true` when `instanceType` doesn't support GPU. In this case the configuration will be ignored. - user set `enabled: false` when `instanceType` does support GPU - when `enabled: true` on GPU supported instance type, - nvidia driver will be installed automatically in each node in the nodepool. - The installation will happen just before `kubelet.service` starting (see below). - And, `kubelet` will start with [`--feature-gates="Accelerators=true"`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L212-L214) - then container can mount nvidia driver [like this](https://gist.github.com/everpeace/9e03050467d5ef5f66b7ce96b5fefa72#file-pod-yaml-L30-L53) - several tags are assigned to the node for enabling schedule on appropriate GPU model and its driver version by using `nodeAffinity`. - `alpha.kubernetes.io/nvidia-gpu-name=<GPU hardware type name>` - `kube-aws.coreos.com/gpu=nvidia`, - `kube-aws.coreos.com/nvidia-gpu-version=<version>` - Because substitution are not used in unit definition, I introduced `/etc/default/kubectl` for defining these label values in [this commit](kubernetes-retired@5c59944). ### Driver installation process Most of installation script is borrowed from [/Clarifai/coreos-nvidia](https://github.com/Clarifai/coreos-nvidia/). Especially, for device node installation, I referenced to Clarifai/coreos-nvidia#4 . I just described summary of installation process. - [`kubelet.service`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L144-L147) ruires [`nvidia-start.service`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L456-L471) - [`nvidia-start.service`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L456-L471) invokes [`build-and-install.sh`](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L918-L947), which installs nvidia drivers and kernel module files, via `ExecStartPre`. `nvidia-start.service` will create device nodes(`nvidiactl` and `nvidia0,1,...`). Other dynamic device nodes are controlled by`udevadam` (configuration is in [this rule file](https://github.com/everpeace/kube-aws/blob/feature/nvidia-gpu/core/controlplane/config/templates/cloud-config-worker#L905-L939)) - `nvidia-start.service` is `type=oneshot` because `kubelet.service` should wait until `nvidia-start.sh` completely succeeded. - `Restart` policy cannot be used with`type=oneshot`. `nvidia-start.service` doesn't use systemd's retry feature is not used but manual `retry.sh` is used. - [nvidia-persistenced](https://docs.nvidia.com/deploy/driver-persistence/#persistence-daemon) is also enabled for speeding up startup. this service is started/stopped via `udevadam` too. ## How to try 1. build `kube-aws` on this branch 2. `kube-aws up` with minimal nodepool configuration below ``` worker: nodePools: - name: p2xlarge count: 1 instanceType: p2.xlarge rootVolume: size: 30 type: gp2 gpu: nvidia: enabled: true version: "375.66" ``` 3. check `kubectl get nodes --show-labels`. Then you'll see one node with gpu related labels. 4. try starting this [pod](https://gist.github.com/everpeace/9e03050467d5ef5f66b7ce96b5fefa72#file-pod-yaml) ``` kubectl create -f pod.yaml ``` 5. log reports sample matrix multiplication is computed on gpus. ``` kubectl logs gpu-pod ``` ## Full changelog * add /etc/default/kubelet to worker nodes. * add nvidia driver installation support. * add gpu related config test. * it should be error when user gpu.nvidia.true with GPU unspported intance types. This change is caused by: kubernetes-retired#645 (comment) * add note which warns that driver may stop working when OS is updated. This change is caused by: kubernetes-retired#645 (comment) * move nvidia-{start, persisntenced}.service to `coreos.units` section. creation for nvidia-persistenced user to `users` section, too. This change is caused by: kubernetes-retired#645 (comment) * introduce unit dependency: kubelet --> nvidia-start --> nvidia-install deleted `systemctl` command from bash script. Instead, above unit dependency is introduced. nvidia-install.service, which just invokes build-and-install.sh is implemented type=oneshot because nvidia-start should wait until nvidia-install.service successed completely. Enabling retry build-and-install.sh, /opt/nvidia-build/util/retry.sh is introduced. It is because type=oneshot and Restart=always can't be used in systemd. * delete nvidia-install.service and now nvidia-start.service invoke build-and-install.sh via ExecStartPre with retry.sh kubelet.service 'Requires' and 'After' nvidia-star.service.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I took the
udevapproach instead of manuallymknoding the modules, this will be more similar to the official approach taken by desktop distros like Ubuntu.