Support Kubeflow Jobs in MultiKueue

[alculquicondor]

What would you like to be added:

Support for Kubeflow Jobs in MultiKueue, in particular, for TFJob and PyTorchJob.
Ideally, the implementation should be mostly common among all job types.

Kubeflow Job doesn't have support for managedBy, so, for now, we can only support the scenario where the manager cluster doesn't have the controller installed.

Why is this needed:

To continue the incremental improvement of MK and satisfy the needs of early adopters.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[alculquicondor]

/assign @mszadkow

[kannon92]

The hope is to implement a new version of Trainingoperator called TrainJob that would use JobSet as the base.

I think if rhat is done then one could use the ManagedField in JobSet for this.

[alculquicondor]

Right, but we have users requesting this feature today.

[kannon92]

Should there be some work done to add managedField to the KubeFlow API? Or are we trying to avoid that to satisfy a solution that bypasses KubeFlow releases?

[alculquicondor]

We don't need it in the current version of MultiKueue. We can just recommend users not to install the operator in the dispatcher cluster.

[mimowo]

/reopen
Let's close it when the ongoing effort of supporting managedBy is complete for the training-operator and MPIJob.

[k8s-ci-robot]

@mimowo: Reopened this issue.

In response to this:

/reopen
Let's close it when the ongoing effort of supporting managedBy is complete for the training-operator and MPIJob.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tenzen-y]

FYI: kubeflow/trainer#2203 was merged right now.
We will include the feature in the next training-operator minor release.

RC.0 with the managedBy feature will be released on January 20th, 2025.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mimowo]

/remove-lifecycle stale

[mimowo]

RC.0 with the managedBy feature will be released on January 20th, 2025.

@tenzen-y does the plan for Jan 20th still holds?

cc @mszadkow

[tenzen-y]

/reopen Let's close it when the ongoing effort of supporting managedBy is complete for the training-operator and MPIJob.

We plan to release the final Kubeflow v1 API release within this year.

[mimowo]

FYI we already have the training-operator rc1 with the managedBy field: https://github.com/kubeflow/training-operator/releases/tag/v1.9.0-rc.0.

So, we can prepare the PR for the integration already. Depending on the timeline of the full release we can either merge the support using rc1 or wait for the full release, but starting the work early and discovering potential roadblocks would be great.

cc @mszadkow

[mszadkow]

FYI we already have the training-operator rc1 with the managedBy field: https://github.com/kubeflow/training-operator/releases/tag/v1.9.0-rc.0.

So, we can prepare the PR for the integration already. Depending on the timeline of the full release we can either merge the support using rc1 or wait for the full release, but starting the work early and discovering potential roadblocks would be great.

cc @mszadkow

ACK

[tenzen-y]

I guess that this has already been completed. @mszadkow @mimowo Do you have any remaining tasks?

[mszadkow]

I think only documentation left, I will push PR today.

[mimowo]

Please also x-ref the documentation in Kueue with the pending PR for the documentation in kubeflow: kubeflow/website#3956

[tenzen-y]

I think only documentation left, I will push PR today.

Thank you!