WSL2 & GPU Operator

[Azerothian]

This is in insight for anyone trying to use RKE2 on wsl2 with the gpu-operator

---

NVIDIA/gpu-operator#318

System: Ubuntu 24.04.1 LTS
kernel: 5.15.167.4-microsoft-standard-WSL2+
RKE2 Version: v1.30.6+rke2r1

If anyone is trying to use RKE2 as a base, I found that it is forcing the nvidia runtime to use SystemCgroups = true in its container config. But my guess this seems to be failing back to cgroups v1?

Add this to your .wslconfig to force your kernel to use cgroups v2

[wsl2]
kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1

also unsure as to why but just worked for me patching the nvidia-device-plugin-daemonset to just use the nvidia-device-plugin command fixed the NVML failed to load error

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: gpu-operator
spec:
  containers:
    - command:
        - nvidia-device-plugin
      args: []

Node Labels

    feature.node.kubernetes.io/pci-10de.present: 'true'
    nvidia.com/device-plugin.config: NVIDIA-GeForce-RTX-4080
    nvidia.com/gpu.count: '1'
    nvidia.com/gpu.deploy.container-toolkit: 'false'
    nvidia.com/gpu.deploy.dcgm: 'true'
    nvidia.com/gpu.deploy.dcgm-exporter: 'true'
    nvidia.com/gpu.deploy.device-plugin: 'true'
    nvidia.com/gpu.deploy.driver: 'false'
    nvidia.com/gpu.deploy.gpu-feature-discovery: 'false'
    nvidia.com/gpu.deploy.node-status-exporter: 'true'
    nvidia.com/gpu.deploy.operator-validator: 'true'
    nvidia.com/gpu.present: 'true'
    nvidia.com/gpu.replicas: '16'

HelmChart

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  targetNamespace: gpu-operator
  createNamespace: true
  bootstrap: false
  valuesContent: |-
    cdi:
      enabled: false
    driver:
      enabled: false
    toolkit:
      enabled: true
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock
    validator:
      driver:
        env:
        - name: DISABLE_DEV_CHAR_SYMLINK_CREATION
          value: "true"
    devicePlugin:
      enabled: true

/etc/wsl2.conf

[boot]
systemd=true
command="/etc/startup.sh"

/etc/startup.sh

mount --make-rshared /
mkdir -p /run/nvidia/validations
touch /run/nvidia/validations/host-driver-ready
touch /run/nvidia/validations/toolkit-ready
touch /run/nvidia/validations/cuda-ready
touch /run/nvidia/validations/plugin-ready

Not sure if the next part is necessary, i need to test without it

/var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl

version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/rke2/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "index.docker.io/rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/rke2/agent/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
  runtime_type = "io.containerd.runc.v2"
  privileged_without_host_devices = false
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"
  Runtime = "/usr/bin/nvidia-container-runtime"
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-cdi"]
  runtime_type = "io.containerd.runc.v2"
  privileged_without_host_devices = false
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-cdi".options]
  BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi"
  SystemdCgroup = true

c:\Users\Profile.wslconfig

[wsl2]
kernel=C:\\bzImage-iscsi
swap=0
localhostForwarding=true
kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1

also a hint for anyone trying to use longhorn in wsl and having issues (iscsi support is not compiled in to the current kernel) look into compiling and using custom wsl kernels