Support TFJob(kubeflow) in Multikueue #2626

mszadkow · 2024-07-16T13:27:21Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

The PR introduces a new Multikueue adapter to handle TFJob (Kubeflow).
We want to extend Multikueue capabilities to satisfy the needs of early adopters.

Which issue(s) this PR fixes:

Relates #2552

Special notes for your reviewer:

Does this PR introduce a user-facing change?

MultiKueue: Support for the Kubeflow TFJob

k8s-ci-robot · 2024-07-16T13:27:23Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

netlify · 2024-07-16T13:27:36Z

✅ Deploy Preview for kubernetes-sigs-kueue ready!

Name	Link
🔨 Latest commit	`dd4dbe9`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/66a9dc76fcc1c900081a8911
😎 Deploy Preview	https://deploy-preview-2626--kubernetes-sigs-kueue.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

mszadkow · 2024-07-17T06:04:53Z

/ok-to-test

mszadkow · 2024-07-18T11:58:13Z

/retest

mszadkow · 2024-07-22T11:17:52Z

/retest

mszadkow · 2024-07-22T15:02:33Z

/retest

pkg/controller/admissionchecks/multikueue/indexer_test.go

hack/multikueue-e2e-test.sh

test/e2e/multikueue/e2e_test.go

mszadkow · 2024-07-23T07:55:35Z

/retest

mszadkow · 2024-07-23T08:12:04Z

/retest

test/e2e/multikueue/e2e_test.go

mimowo · 2024-07-30T09:51:17Z

@mszadkow please ping us in a comment when the PR is ready for the second pass after the updates.

mszadkow · 2024-07-30T10:47:03Z

@tenzen-y @mimowo
I think it's ready

mszadkow · 2024-07-30T10:47:09Z

/retest

trasc

some nits, otherwise LGTM

pkg/controller/jobs/job/job_multikueue_adapter.go

pkg/controller/jobs/jobset/jobset_multikueue_adapter.go

site/content/en/docs/concepts/multikueue.md

trasc · 2024-07-30T12:38:03Z

site/content/en/docs/tasks/manage/setup_multikueue.md

@@ -74,6 +74,16 @@ kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs
 ```
 {{% /alert %}}

+### Kubeflow Installation


Add a link to the kubeflow installation guide, to be done in the worker clusters.

test/e2e/multikueue/e2e_test.go

tenzen-y

Some minor change suggestions.
/approve

tenzen-y · 2024-07-31T05:49:22Z

site/content/en/docs/concepts/multikueue.md

@@ -68,6 +68,12 @@ The `managedBy` field is available as an Alpha feature staring Kubernetes 1.30.0

 We recommend using JobSet v0.5.1 or newer.

+### Kubeflow
+
+The supported version of the Kubeflow Training Operator is v1.7.0. 


Suggested change

The supported version of the Kubeflow Training Operator is v1.7.0.

The supported version of the Kubeflow Training Operator is v1.7.0, or a newer version.

tenzen-y · 2024-07-31T05:49:58Z

site/content/en/docs/tasks/manage/setup_multikueue.md

+### Kubeflow Installation
+
+Install Kubeflow Training-operator in the Worker cluster (see [Kubeflow Training-operator Installation](https://www.kubeflow.org/docs/components/training/installation/)
+for more details). Please use version v1.7.0 for MultiKueue.


Suggested change

for more details). Please use version v1.7.0 for MultiKueue.

for more details). Please use version v1.7.0 or a newer version for MultiKueue.

tenzen-y · 2024-07-31T05:52:48Z

site/content/en/docs/tasks/manage/setup_multikueue.md

+### Kubeflow Installation
+
+{{% alert title="Warning" color="warning" %}}
+Make sure to only install the Kubeflow TFJobs CRD of version v1.7.0 on the management cluster.


Suggested change

Make sure to only install the Kubeflow TFJobs CRD of version v1.7.0 on the management cluster.

Make sure to install only the Kubeflow TFJobs CRD of version v1.7.0 or newer on the management cluster.

tenzen-y · 2024-07-31T05:53:23Z

site/content/en/docs/tasks/manage/setup_multikueue.md

+Make sure to only install the Kubeflow TFJobs CRD of version v1.7.0 on the management cluster.
+
+```bash
+  kubectl apply --server-side -f https://github.com/kubeflow/training-operator/blob/v1.7.0/manifests/base/crds/kubeflow.org_tfjobs.yaml


Suggested change

kubectl apply --server-side -f https://github.com/kubeflow/training-operator/blob/v1.7.0/manifests/base/crds/kubeflow.org_tfjobs.yaml

kubectl apply --server-side -f https://github.com/kubeflow/training-operator/blob/v1.8.0/manifests/base/crds/kubeflow.org_tfjobs.yaml

Let's use the latest Kubeflow Training Operator version.

k8s-ci-robot · 2024-07-31T05:54:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mszadkow, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

trasc

/lgtm

k8s-ci-robot · 2024-07-31T07:18:19Z

LGTM label has been added.

Git tree hash: c69f1a8dde6f7c068ec57fb0f2838318a4fb45a7

tenzen-y · 2024-07-31T08:09:46Z

/release-note-edit

MultiKueue: Support for the Kubeflow TFJob

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels Jul 16, 2024

k8s-ci-robot requested review from alculquicondor and denkensk July 16, 2024 13:27

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 16, 2024

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jul 16, 2024

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from 51ef408 to 7845aa5 Compare July 16, 2024 13:53

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jul 17, 2024

mszadkow force-pushed the feature/support-kubeflow-in-mk branch 2 times, most recently from fed964d to c8ce51a Compare July 18, 2024 10:28

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 18, 2024

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from c8ce51a to 4c252f9 Compare July 18, 2024 10:29

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from 03e2736 to 7848c26 Compare July 22, 2024 11:14

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 22, 2024

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from 7848c26 to 3e314a8 Compare July 22, 2024 14:54

trasc reviewed Jul 23, 2024

View reviewed changes

pkg/controller/admissionchecks/multikueue/indexer_test.go Outdated Show resolved Hide resolved

hack/multikueue-e2e-test.sh Outdated Show resolved Hide resolved

test/e2e/multikueue/e2e_test.go Outdated Show resolved Hide resolved

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from 3e314a8 to 74aeec5 Compare July 23, 2024 07:52

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from 74aeec5 to 702e214 Compare July 23, 2024 08:11

mszadkow marked this pull request as ready for review July 23, 2024 16:37

mimowo reviewed Jul 25, 2024

View reviewed changes

test/e2e/multikueue/e2e_test.go Outdated Show resolved Hide resolved

mszadkow mentioned this pull request Jul 29, 2024

Update Kubeflow image tag #2709

Closed

mszadkow and others added 5 commits July 30, 2024 12:44

add kubeflow tfjob adapter

4547619

update multikueue e2e test scripts to support running tfjob

98a836f

add multikueue tests for tfjob

921b12b

add kubeflow tfjobs integration tests

3b563fc

Tests fix-up

e73894f

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from 83ea5c9 to 3b81bb6 Compare July 30, 2024 10:44

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 30, 2024

k8s-ci-robot requested a review from trasc July 30, 2024 10:44

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from 3b81bb6 to 63305cf Compare July 30, 2024 11:58

trasc reviewed Jul 30, 2024

View reviewed changes

rework after code review

bd4e440

mszadkow force-pushed the feature/support-kubeflow-in-mk branch from 63305cf to bd4e440 Compare July 30, 2024 14:59

tenzen-y reviewed Jul 31, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 31, 2024

update docs

dd4dbe9

trasc reviewed Jul 31, 2024

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 31, 2024

k8s-ci-robot merged commit 3b8d828 into kubernetes-sigs:main Jul 31, 2024
16 checks passed

k8s-ci-robot added this to the v0.9 milestone Jul 31, 2024

trasc deleted the feature/support-kubeflow-in-mk branch July 31, 2024 07:46

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jul 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support TFJob(kubeflow) in Multikueue #2626

Support TFJob(kubeflow) in Multikueue #2626

mszadkow commented Jul 16, 2024 •

edited by k8s-ci-robot

Loading

k8s-ci-robot commented Jul 16, 2024

netlify bot commented Jul 16, 2024 •

edited

Loading

mszadkow commented Jul 17, 2024

mszadkow commented Jul 18, 2024

mszadkow commented Jul 22, 2024

mszadkow commented Jul 22, 2024

mszadkow commented Jul 23, 2024

mszadkow commented Jul 23, 2024

mimowo commented Jul 30, 2024

mszadkow commented Jul 30, 2024

mszadkow commented Jul 30, 2024

trasc left a comment

trasc Jul 30, 2024

tenzen-y left a comment

tenzen-y Jul 31, 2024

tenzen-y Jul 31, 2024

tenzen-y Jul 31, 2024

tenzen-y Jul 31, 2024

k8s-ci-robot commented Jul 31, 2024

trasc left a comment

k8s-ci-robot commented Jul 31, 2024

tenzen-y commented Jul 31, 2024

	The supported version of the Kubeflow Training Operator is v1.7.0.
	The supported version of the Kubeflow Training Operator is v1.7.0, or a newer version.

	for more details). Please use version v1.7.0 for MultiKueue.
	for more details). Please use version v1.7.0 or a newer version for MultiKueue.

	Make sure to only install the Kubeflow TFJobs CRD of version v1.7.0 on the management cluster.
	Make sure to install only the Kubeflow TFJobs CRD of version v1.7.0 or newer on the management cluster.

	kubectl apply --server-side -f https://github.com/kubeflow/training-operator/blob/v1.7.0/manifests/base/crds/kubeflow.org_tfjobs.yaml
	kubectl apply --server-side -f https://github.com/kubeflow/training-operator/blob/v1.8.0/manifests/base/crds/kubeflow.org_tfjobs.yaml

Support TFJob(kubeflow) in Multikueue #2626

Support TFJob(kubeflow) in Multikueue #2626

Conversation

mszadkow commented Jul 16, 2024 • edited by k8s-ci-robot Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Jul 16, 2024

netlify bot commented Jul 16, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue ready!

mszadkow commented Jul 17, 2024

mszadkow commented Jul 18, 2024

mszadkow commented Jul 22, 2024

mszadkow commented Jul 22, 2024

mszadkow commented Jul 23, 2024

mszadkow commented Jul 23, 2024

mimowo commented Jul 30, 2024

mszadkow commented Jul 30, 2024

mszadkow commented Jul 30, 2024

trasc left a comment

Choose a reason for hiding this comment

trasc Jul 30, 2024

Choose a reason for hiding this comment

tenzen-y left a comment

Choose a reason for hiding this comment

tenzen-y Jul 31, 2024

Choose a reason for hiding this comment

tenzen-y Jul 31, 2024

Choose a reason for hiding this comment

tenzen-y Jul 31, 2024

Choose a reason for hiding this comment

tenzen-y Jul 31, 2024

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 31, 2024

trasc left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 31, 2024

tenzen-y commented Jul 31, 2024

mszadkow commented Jul 16, 2024 •

edited by k8s-ci-robot

Loading

netlify bot commented Jul 16, 2024 •

edited

Loading