CUDA setup not working?

[Data-drone]

I followed the instructions as per the cuda setup article but I noticed that the nvidia-device-plugin-daemonset pods are crashing.

The error message is:

Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request: unknown

How can I debug this?

I have going into the k3d docker container and nvidia-smi works so the GPU is being made available.

I am running:

Ubuntu 20.04
nvidia-docker2 is installed and working
Driver on host machine is: 470.63.01
k3d version v4.4.8
k3s version v1.21.3-k3s1 (default)

Docker is Docker-ce 20.10.8

[Data-drone]

Okay from the original documentation I updated the device-plugin-daemonset.yaml to the latest one as per

https://github.com/NVIDIA/k8s-device-plugin/blob/master/nvidia-device-plugin.yml

and now it seems to work.