Questions about GPU time-sharing on Kubernetes

[jxl4650152]

1. Issue or feature description

Is it possible to enable time-sharing on select GPUs within a single node, rather than all of them?
If not all GPUs on a single node support time-slicing, like Kepler K80 GPU, what behavior can be expected from the plugin?

[elezar]

The device plugin does not currently have a mechanism to expose different devices as different resource types. It is also not possible to apply sharing settings on a per-GPU basis.

Note that the plugin will still allow a sharing setting to be applied to GPUs that may not support this feature and effectively reports the same device multiple times to the Kubelet. The behaviour of containers that are both started on a device where time-slicing is not supported will depend on the application and the device, but should mirror what happens when two applications which access the same device are started on the host.

[jxl4650152]

Thank you for your response. We now have a clear understanding of the limitations of this plugin. We are using this repo for managing GPUs on Kubernetes, and we have also noticed another repo related to DRA（Dynamic Resource Allocation）, which offers greater flexibility and richer functionality. Regarding this, we have another question: Will there be strong support for using GPUs on Kubernetes through the DRA approach in the future? I'm not sure if it's appropriate to ask this question here. If not, could you please let me know where I should ask it?

[klueska]

Yes DRA will be well supported. We see it as the future of GPU support in Kubernetes.

Please see:
https://m.youtube.com/watch?v=_fi9asserLE&ab_channel=CNCF%5BCloudNativeComputingFoundation%5D

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed.

[frittentheke]

As @elezar said this is not (yet possible), see

k8s-device-plugin/api/config/v1/replicas.go

Line 61 in 35ad180

if setsDevices {

I myself would also like to only enable time-slicing on a subset of GPUs. Is there any chance we could make this issue here a feature request @elezar? Or is this capability never coming to the device plugin in favor of DRA? That seems quite a while out though - NVIDIA/k8s-dra-driver#131