[Bug]: GPUs cannot be used with Tainted Nodes

[andrewballantyne]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The difference is that I usually add a Taint to my nodes called "notebooksonly". I did add it to the GPU node, because:
A) that node has no taint by default
B) I had noticed random pods landing on it and hogging it for nothing
However, as soon as I add the taint, a few minutes later, the drop down disappears.
When I remove the taint, the drop-down re-appears.

From the downstream issue https://issues.redhat.com/browse/RHODS-4769

Expected Behavior

GPUs should work for both using tolerations & not using tolerations.

Steps To Reproduce

See the current behaviour / downstream ticket.

This is likely related to the Prometheus query omitting results -- but that is just a guess. Might be impacted by #573 and supporting of scaling (which could change the "show the dropdown" logic -- may not work with unscalable nodes that have tolerations)

Workaround (if any)

No response

OpenShift Infrastructure Version

No response

Openshift Version

No response

What browsers are you seeing the problem on?

No response

Open Data Hub Version

2.3.0

Relevant log output

No response

[guimou]

As it is, tolerations added manually to a Notebook definition are "erased" when you start the Notebook through the UI. This is incompatible with multiple pools of GPUs that you have to taint to properly schedule specific workloads, or just with standard management of workloads placement through taints.
We should either:

• Leave the tolerations in the Notebook CR alone, or only reconcile the ones we provide access to trough the UI, like GPU request = nvidia.com/gpu toleration.
• Or provide a mechanism in the UI to fully managed tolerations on Notebooks.

[andrewballantyne]

So today we have a toleration for nvidia.com/gpu (code link) -- this should be covered if you taint your node with the same. We also have the "NotebooksOnly" (configurable) default toleration for Notebooks if you check out the cluster settings as an admin -- but that's not tied to (Nvidia) GPU, that's additional.

I would imagine this is handled already to some degree 🤔

In the work for accelerators, I imagine this will be more flexible and seamless for what is being tainted/tolerated... effectively this should be done, no?

• See [UI] Habana Support Part 1 #1450
  • More specifically the CR we will create to allow customization for accelerators (Create a custom resource to track accelerator devices #1371)

[dgutride]

This is no longer valid due to Habana work - moving to closed after talking to Gage. Please reopen with questions if there are any misunderstandings.