Support for nvidia-runtime on containerd and k3s on ARM64 machines due to docker deprecation

[Shaked]

Hey all,

I'm not sure if this should be here but as k3s seems to be quite used among ARM based machines, I thought I should post this both for others to be familiar with and maybe to figure if this is the best solution.

As most of the community has heard, Kubernetes 1.20, support of the Docker container engine is deprecated.

During our development cycle, we have been heavily dependent on docker as we needed to use nvidia-docker. This allowed us to use our devices - mainly Jetson Xavier - GPU within docker and later on within our k3s cluster.

Now, as kubernetes is dropping docker support and moving on with containerd, we figured that we should quickly fix our dependency in docker.

What we did:

1. Although for some of you this might be irrelevant, I do want to state that we moved our rancher/k3s root directory from /var/lib/rancher/k3s to /ssd/lib/rancher/k3s and the main reason was because we want containerd to save the downloaded images on an external SSD drive instead of the main harddrive which is quite low on space. (Before we did the same with /var/lib/docker)
2. We updated /etc/rancher/k3s/registries.yaml with our own settings. Before we did this in /etc/docker/daemon.json.
3. We of course no longer install k3s with the --docker argument.
4. We have added a template file /ssd/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl:

[plugins.opt]
  path = "{{ .NodeConfig.Containerd.Opt }}"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = {{ .NodeConfig.SELinux }}
{{- if .IsRunningInUserNS }}
  disable_cgroup = true
  disable_apparmor = true
  restrict_oom_score_adj = true
{{end}}
{{- if .NodeConfig.AgentConfig.PauseImage }}
  sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
{{end}}
{{- if .NodeConfig.AgentConfig.Snapshotter }}
[plugins.cri.containerd]
  disable_snapshot_annotations = true
  snapshotter = "{{ .NodeConfig.AgentConfig.Snapshotter }}"
{{end}}
{{- if not .NodeConfig.NoFlannel }}
[plugins.cri.cni]
  bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
  conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
{{end}}
[plugins.cri.containerd.default_runtime]
    privileged_without_host_devices = false
    runtime_engine = ""
    runtime_root = ""
    runtime_type = "io.containerd.runtime.v1.linux"

    [plugins.cri.containerd.default_runtime.options]
        Runtime = "/usr/bin/nvidia-container-runtime"

[plugins.cri.containerd.runtimes.nvidia]
    privileged_without_host_devices = false
    runtime_engine = ""
    runtime_root = ""
    runtime_type = "io.containerd.runtime.v1.linux"

    [plugins.cri.containerd.runtimes.nvidia.options]
        Runtime = "/usr/bin/nvidia-container-runtime"
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
{{ if .PrivateRegistryConfig }}
{{ if .PrivateRegistryConfig.Mirrors }}
[plugins.cri.registry.mirrors]{{end}}
{{range $k, $v := .PrivateRegistryConfig.Mirrors }}
[plugins.cri.registry.mirrors."{{$k}}"]
  endpoint = [{{range $i, $j := $v.Endpoints}}{{if $i}}, {{end}}{{printf "%q" .}}{{end}}]
{{end}}
{{range $k, $v := .PrivateRegistryConfig.Configs }}
{{ if $v.Auth }}
[plugins.cri.registry.configs."{{$k}}".auth]
  {{ if $v.Auth.Username }}username = {{ printf "%q" $v.Auth.Username }}{{end}}
  {{ if $v.Auth.Password }}password = {{ printf "%q" $v.Auth.Password }}{{end}}
  {{ if $v.Auth.Auth }}auth = {{ printf "%q" $v.Auth.Auth }}{{end}}
  {{ if $v.Auth.IdentityToken }}identitytoken = {{ printf "%q" $v.Auth.IdentityToken }}{{end}}
{{end}}
{{ if $v.TLS }}
[plugins.cri.registry.configs."{{$k}}".tls]
  {{ if $v.TLS.CAFile }}ca_file = "{{ $v.TLS.CAFile }}"{{end}}
  {{ if $v.TLS.CertFile }}cert_file = "{{ $v.TLS.CertFile }}"{{end}}
  {{ if $v.TLS.KeyFile }}key_file = "{{ $v.TLS.KeyFile }}"{{end}}
  {{ if $v.TLS.InsecureSkipVerify }}insecure_skip_verify = true{{end}}
{{end}}
{{end}}
{{end}}

We got nvidia-runtime's configuration from @klueska. Details at NVIDIA/nvidia-docker#1468 (comment)

5. Restart of k3s is required after settings this up.
6. We ran a deviceQuery container which showed that the GPU worked:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-test
spec:
  containers:
  - name: nvidia-test
    image: docker.io/jitteam/devicequery:latest
  restartPolicy: Never
EOF

I hope this helps others as it helped us,
Shaked

[Shaked]

I am following up on this topic after stumbling across https://k3d.io/v4.4.8/usage/guides/cuda/.

So the correct setup seems to be:

# Generated by Ansible
# Support for nvidia runtime through containerd: https://github.com/NVIDIA/nvidia-docker/issues/1468#issuecomment-791430335
[plugins.opt]
  path = "/xavier_ssd/var/lib/rancher/k3s/agent/containerd"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "docker.io/rancher/pause:3.1"

[plugins.cri.containerd]
  disable_snapshot_annotations = true
  snapshotter = "overlayfs"

[plugins.cri.cni]
  bin_dir = "/var/lib/rancher/k3s/data/477a397d5ed8de53c8cee46d5f565f81e93de8dadbff870dc27452b47238f6f5/bin"
  conf_dir = "/xavier_ssd/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runtime.v1.linux"
[plugins.linux]
  runtime = "nvidia-container-runtime"


[plugins.cri.registry.mirrors]

[plugins.cri.registry.mirrors."greeneye.azurecr.io"]
  endpoint = ["http://xaviershaked1:5000", "http://XavierShakedTX:5000"]

Or for k3s:

# Generated by Ansible
# Support for nvidia runtime through containerd: https://github.com/NVIDIA/nvidia-docker/issues/1468#issuecomment-791430335
[plugins.opt]
  path = "{{ .NodeConfig.Containerd.Opt }}"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = {{ .NodeConfig.SELinux }}
{{- if .IsRunningInUserNS }}
  disable_cgroup = true
  disable_apparmor = true
  restrict_oom_score_adj = true
{{end}}
{{- if .NodeConfig.AgentConfig.PauseImage }}
  sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
{{end}}
{{- if .NodeConfig.AgentConfig.Snapshotter }}
[plugins.cri.containerd]
  disable_snapshot_annotations = true
  snapshotter = "{{ .NodeConfig.AgentConfig.Snapshotter }}"
{{end}}
{{- if not .NodeConfig.NoFlannel }}
[plugins.cri.cni]
  bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
  conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
{{end}}
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runtime.v1.linux"
[plugins.linux]
  runtime = "nvidia-container-runtime"
{{ if .PrivateRegistryConfig }}
{{ if .PrivateRegistryConfig.Mirrors }}
[plugins.cri.registry.mirrors]{{end}}
{{range $k, $v := .PrivateRegistryConfig.Mirrors }}
[plugins.cri.registry.mirrors."{{$k}}"]
  endpoint = [{{range $i, $j := $v.Endpoints}}{{if $i}}, {{end}}{{printf "%q" .}}{{end}}]
{{end}}
{{range $k, $v := .PrivateRegistryConfig.Configs }}
{{ if $v.Auth }}
[plugins.cri.registryc.configs."{{$k}}".auth]
  {{ if $v.Auth.Username }}username = {{ printf "%q" $v.Auth.Username }}{{end}}
  {{ if $v.Auth.Password }}password = {{ printf "%q" $v.Auth.Password }}{{end}}
  {{ if $v.Auth.Auth }}auth = {{ printf "%q" $v.Auth.Auth }}{{end}}
  {{ if $v.Auth.IdentityToken }}identitytoken = {{ printf "%q" $v.Auth.IdentityToken }}{{end}}
{{end}}
{{ if $v.TLS }}
[plugins.cri.registry.configs."{{$k}}".tls]
  {{ if $v.TLS.CAFile }}ca_file = "{{ $v.TLS.CAFile }}"{{end}}
  {{ if $v.TLS.CertFile }}cert_file = "{{ $v.TLS.CertFile }}"{{end}}
  {{ if $v.TLS.KeyFile }}key_file = "{{ $v.TLS.KeyFile }}"{{end}}
  {{ if $v.TLS.InsecureSkipVerify }}insecure_skip_verify = true{{end}}
{{end}}
{{end}}
{{end}}

Then run the following to test this:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-test
spec:
  containers:
  - name: nvidia-test
    image: icetekio/jetson-nano-tensorflow
    command: [python3]
    args:
      - -c
      - 'import tensorflow'
  restartPolicy: Never
EOF

However, I'm looking where I could learn a bit more about these settings in regards to Nvidia, as I couldn't really find an explanation of why this works.

Shaked

CC:
@iwilltry42