NVIDIA Device Plugin Only Exposes One GPU Out of Two GPUs Installed on Single Node

[amir-bialek]

Hey all,

"I have an on-premises Kubernetes cluster with multiple nodes. One of these nodes is equipped with two different GPU models:
NVIDIA GeForce RTX 3090 and NVIDIA GeForce RTX 4090

When I SSH into this node and run nvidia-smi, both GPUs are properly detected and displayed.
I have installed the NVIDIA Device Plugin using gpu-operator Helm chart (https://github.com/NVIDIA/gpu-operator/tree/main/deployments/gpu-operator).
However, only the RTX 4090 is being exposed as a resource to Kubernetes.
Here is my current configuration:

devicePlugin:
  config:
    name: time-slicing-config-all
    create: true
    default: "any"
    data:
      any: |-
        version: v1
        flags:
          migStrategy: none
        sharing:
          timeSlicing:
            resources:
            - name: nvidia.com/gpu
              replicas: 5
I have tried different type of the configuration, but it always show only one type.

Any help ?

[klueska]

Multiple GPU types per node is not supported with the standard device plugin.

[amir-bialek]

omg. I have been breaking my head on this one.
Is there anyway around it?
any way I can tell to one pod to run gpu A, and one pod to run gpu B?

[amir-bialek]

Or any other plugin that does support both gpu?

[klueska]

Please see the thread starting here:
#424 (comment)

[amir-bialek]

Hey @klueska , thanks for the info.
At the moment the GPU is already installed on the Node, so the nvidia.com/gpu.product keep changing between the 3090 and 4090. is there anyway in the configuration I can tell it to discover only the 4090? something like:

      multi-gpu-node: |-
        version: v1
        flags:
          migStrategy: none
        resources:
           gpus:
               pattern: "*4090"
        sharing:
          timeSlicing:
            resources:
            - name: nvidia.com/gpu
              replicas: 20

?

[klueska]

Unfortunately not.

[amir-bialek]

Hey @klueska Some update:

Following the discussion here: #424 and information here: https://docs.google.com/document/d/1dL67t9IqKC2-xqonMi6DV7W2YNZdkmfX7ibB6Jb-qmk/edit?tab=t.0

(Many thanks for writing this ! it really helps :)

I downloaded the release version of 0.14.5 of Nvidia device plugin:
https://github.com/NVIDIA/k8s-device-plugin/releases/tag/v0.14.5

Made the following changes:

1. remove line 239 disableResourceRenamingInConfig(config) in file cmd/nvidia-device-plugin/main.go

• The line number may be different on every release. In addition, in release 0.16.2 it does not exists, but there is spec.DisableResourceNamingInConfig(logger.ToKlog, config). I assume this is the one that should be remove on newer versions.

2. In file /deployments/container/Dockerfile.ubuntu change line 26 from ARG GOLANG_VERSION=x.x.x to ARG GOLANG_VERSION=1.20.5

• Version 1.20.5 appear at line 15 so it seems most appropriate. Without this change the build command will fail.

Build with docker build -t my-image:nvidia -f deployments/container/Dockerfile.ubuntu .

Push to dockerhub

Edit gpu-operator helm in our cluster changes:

devicePlugin:
  repository: nvcr.io/nvidia
  image: k8s-device-plugin
  version: v0.14.5-ubi8
  config:
    name: device-plugin-config
    create: true
    default: "any"
    data:
      any: |-
        version: v1
        flags:
          migStrategy: none
        sharing:
          timeSlicing:
            resources:
            - name: nvidia.com/gpu
              replicas: 10
      mix-gpu-node: |-
        version: v1
        flags:
          migStrategy: none
        sharing:
          timeSlicing:
            resources:
            - name: nvidia.com/gpu
              replicas: 20

to:

devicePlugin:
  repository: <my_repo>
  image: <my image>
  version: <my version>
  config:
    name: device-plugin-config
    create: true
    default: "any"
    data:
      any: |-
        version: v1
        flags:
          migStrategy: none
        sharing:
          timeSlicing:
            resources:
            - name: nvidia.com/gpu
              replicas: 10

       mix-gpu-node: |-
         version: v1
         resources:
           gpus:
           - pattern: "*RTX*4090*"
             name: rtx4090
         flags:
           migStrategy: none
         sharing:
           timeSlicing:
             resources:
             - name: nvidia.com/rtx4090 
               replicas: 10
             - name: nvidia.com/gpu
               replicas: 20

• Apply the changes, restart the pods, etc..
• Checking log of nvidia-device-plugin-daemonset-pod on mix-gpu-node, It seems to work:

...
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "none",
    "failOnInitError": true,
    "nvidiaDriverRoot": "/",
    "gdsEnabled": false,
    "mofedEnabled": false,
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "envvar"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/bin/nvidia-ctk",
      "containerDriverRoot": "/host"
    }
  },
  "resources": {
    "gpus": [
      {
        "pattern": "*RTX*4090*",
        "name": "nvidia.com/rtx4090"
      },
      {
        "pattern": "*",
        "name": "nvidia.com/gpu"
      }
    ]
  },
  "sharing": {
    "timeSlicing": {
      "resources": [
        {
          "name": "nvidia.com/rtx4090",
          "devices": "all",
          "replicas": 10
        },
        {
          "name": "nvidia.com/gpu",
          "devices": "all",
          "replicas": 20
        }
      ]
    }
  }
}
...

• mix-gpu-node node also have the resrouces available:

  Resource               Requests          Limits
  --------                       --------          ------
..
  nvidia.com/gpu         9                 9
  nvidia.com/rtx4090     0                 0

• Running a pod on mix-gpu-node node with:

    resources:
      limits:
        nvidia.com/rtx4090: 1
      requests:
        nvidia.com/rtx4090: 1

work, and when I exec into the pod and run nvidia-smi it shows the gpu.
And the node resources show

  Resource               Requests          Limits
  --------                       --------          ------
... 
  nvidia.com/gpu         9                 9
  nvidia.com/rtx4090     1                 1

New issue, however on gpu-feature-discovery pod that running on the mix-gpu-node, logs showing:

..
30 18:10:54.019134      28 main.go:247] Customizing the 'resources' field is not yet supported in the config. Ignoring...
I1030 18:10:54.019196      28 main.go:139] 
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "none",
    "failOnInitError": true,
    "gdsEnabled": null,
    "mofedEnabled": null,
    "gfd": {
      "oneshot": false,
      "noTimestamp": false,
      "sleepInterval": "1m0s",
      "outputFile": "/etc/kubernetes/node-feature-discovery/features.d/gfd",
      "machineTypeFile": "/sys/class/dmi/id/product_name"
    }
  },
  "resources": {
    "gpus": null
  },
  "sharing": {
    "timeSlicing": {
      "resources": [
        {
          "name": "nvidia.com/rtx4090",
          "devices": "all",
          "replicas": 10
        },
        {
          "name": "nvidia.com/gpu",
          "devices": "all",
          "replicas": 20
        }
      ]
    }
  }
}
...

In addition the nvidia-operator-validator pod was in crash-loop-bach.
Maybe because the resrouces gpu show as null?
I assume will need to build custom version of them too. (?)
Current image running in the cluster:

nvcr.io/nvidia/gpu-feature-discovery:v0.8.2-ubi8
nvcr.io/nvidia/cloud-native/gpu-operator-validator:v23.9.2

..

• We disconnected the 3090 gpu from mix-gpu-node node , so for now I can only tested it on ‘dry’ until we will be sure of the configuration.

Greatly appreciate any help here.