DRA: Extend PodResources to include resources from Dynamic Resource Allocation

[klueska]

Enhancement Description

• One-line enhancement description (can be used as a release note):
  An enhancement to the PodResources API to include resources allocated by drivers of Dynamic Resource Allocation (DRA).

• Kubernetes Enhancement Proposal:
  https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3695-pod-resources-for-dra/README.md

• Discussion Link:
  dynamic resource allocation kubernetes#111023 (review)
  dynamic resource allocation: add DRA-managed resources to the PodResources API kubernetes#113783
  (As well as private slack messages)

• Primary contact (assignee):
  @klueska, @moshe010, @adrianchiris

• Responsible SIGs:
  /sig node

• Enhancement target (which target equals to which milestone):

  • Alpha release target: 1.27
  • Beta release target: 1.33
  • Stable release target: 1.36

• Alpha

  • KEP (k/enhancements) update PR(s):
    • KEP-3695: extend pod resource API to for DRA #3738
    • PodResources API for DRA: match API with actual implementation in 1.27 #3915
    • KEP-3695: Update podResourceV1 API #4913
    • 1.32 KEP-3695: Update podResourceV1 API #4913
  • Code (k/k) update PR(s):
    • kubelet DRA: Add checkpointing mechanism in the DRA Manager kubernetes#115912
    • Extend the PodResources API to include resources allocated by DRA kubernetes#115847
  • Docs (k/website) update PR(s):
    • Extend PodResources API for Dynamic Resource Allocation website#39978

• Beta

  • KEP (k/enhancements) update PR(s):
    • TBD
  • Code (k/k) update PR(s):
    • TBD
  • Docs (k/website) update(s):
    • TBD

[klueska]

/milestone v1.27
/label lead-opted-in

[k8s-ci-robot]

@klueska: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.27
/label lead-opted-in

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@klueska: Can not set label lead-opted-in: Must be member in one of these teams: [release-team-enhancements release-team-leads sig-api-machinery-leads sig-apps-leads sig-architecture-leads sig-auth-leads sig-autoscaling-leads sig-cli-leads sig-cloud-provider-leads sig-cluster-lifecycle-leads sig-contributor-experience-leads sig-docs-leads sig-instrumentation-leads sig-k8s-infra-leads sig-multicluster-leads sig-network-leads sig-node-leads sig-release-leads sig-scalability-leads sig-scheduling-leads sig-security-leads sig-storage-leads sig-testing-leads sig-windows-leads]

In response to this:

/milestone v1.27
/label lead-opted-in

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SergeyKanzhelev]

/milestone v1.27
/label lead-opted-in

[dchen1107]

/label lead-opted-in

I had trouble to add lead-opted-in last couple of days. Trying it one more time ...

[SergeyKanzhelev]

/stage alpha

[marosset]

Hello @klueska 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.27
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this enhancement, it looks like #3738 will address the remaining requirements.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well.
Thank you!

[marosset]

This enhancement meets all of the requirements to be tracked in v1.27.
Thanks!

[marosset]

Hi @klueska 👋,

Checking in as we approach 1.27 code freeze at 17:00 PDT on Tuesday 14th March 2023.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze:

• Extend the PodResources API to include resources allocated by DRA kubernetes#115847

Please let me know if there are any other PRs in k/k I should be tracking for this KEP.

As always, we are here to help should questions come up. Thanks!

[klueska]

This is a dependent PR for the one you listed -- I have updated the description to include it:
kubernetes/kubernetes#115912

[Rishit-dagli]

Hi @klueska 👋, I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release.
Please feel free to reach out with any questions. Thanks!

[klueska]

Docs placeholder added in description

[SergeyKanzhelev]

Based on SIG Node meeting on 05/02/2023 we do NOT plan this for 1.28 release. Please comment otherwise.

[klueska]

@moshe010 I don't think we can progress this to beta until DRA itself progresses to beta.

[moshe010]

@klueska I wasn't in the SIG Node meeting on 05/02/2023 which it was discussed and I never request this to be beta in 1.28. In the kep we stated that following: [1]
alpha: "v1.27"
beta: "v1.30"
stable: "v1.32"

[1] - https://github.com/kubernetes/enhancements/pull/3915/files#diff-11e83115a85d63622d7dcdb3732b43f918caa972ff7f034a431f642227d22b2aL30-L32

[aojea]

my understanding is that this KEP seems to target a networking out of band plugin approach, that makes sense on a DRA classic environment IIUIC.

Since then, DRA evolved and we are now making networking requirements part of the DRA efforts https://github.com/LionelJouin/kubernetes/commits/KEP-4817/ and making easy to integrate networking plugins directly into DRA https://github.com/aojea/dra-network-driver-template .

My questions is, with the new approach , removing DRA classic and that networking plugins can hook directly into DRA, is this still needed?

[klueska]

From my perspective, this KEP is important independent of its (possible) usage for networking. The PodResourcesAPI is designed to surface the full set of resources that are allocated to a pod, and thus including DRA allocated resources is a natural extension of this. For example, NVIDIA's DCGM prometheus exporter relies on the information provided via this API to link GPU metrics back to the pods that are consuming those GPUs.

[ArangoGutierrez]

/cc

[aojea]

I don't disagree on exposing the resources through the PodResourcesAPI and let the consumers work with that.

I was commenting on the second goal that is related to networking, we are making the integration with networking native, so encouraging that as a goal sounds contradictory with the current efforts, so may complain is not about the KEP is about this paragraph ...

To allow the DRA feature to work with CNIs that require complex network devices such as RDMA. DRA resource drivers will allocate the resources, and the meta-plugin will read the allocated CDI Devices using the PodResources API. The meta-plugin will then inject the device-id of these CDI Devices as CNI arguments and invoke other CNIs (just as it does for devices allocated by the device plugin today).

If you remove that goal and use a more generic text as "allow node components to use the PodResourcesAPI to use the DRA information to develop new features and integrations", then this is a SIG Node only KEP 😄

[johnbelamaric]

@haircommander can you un-opt-in this one? I will remove from my PRR board. Thanks

[ArangoGutierrez]

@ArangoGutierrez had said he was going to take this one over.

That said, there is no implementation work needed at the moment. All that is needed is to update the KEP to be in line with the latest code that is already merged.

I'm working on an update PR for the KEP, hope to have the PR link by tomorrow

[haircommander]

/remove-milestone v1.32
/remove-label lead-opted-in

[ArangoGutierrez]

#4913

[pacoxu]

According to #4913, beta is planned to be "v1.33"and stable: "v1.36".

So this is not milestone v1.32?

[klueska]

Correct -- #4913 was just an update to the KEP to reflect the current state of the code base while remaining in alpha.

[tjons]

/milestone clear

[pohly]

/wg device-management

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale