Add E2E test for gang-scheduling #1736

tenzen-y · 2023-01-22T00:45:56Z

Signed-off-by: Yuki Iwai yuki.iwai.tz@gmail.com

What this PR does / why we need it:
I added the E2E test for gang-scheduling.
Also, I added functions to apply patches to CustomJob resources to Python SDK.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Follow up on #1724.

Checklist:

Docs included if any changes are user facing

coveralls · 2023-01-22T00:49:13Z

Pull Request Test Coverage Report for Build 3994952641

0 of 0 changed or added relevant lines in 0 files are covered.
9 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.06%) to 39.095%

Files with Coverage Reduction	New Missed Lines	%
pkg/controller.v1/pytorch/pytorchjob_controller.go	9	58.16%

Totals
Change from base Build 3980850357:	0.06%
Covered Lines:	2687
Relevant Lines:	6873

💛 - Coveralls

tenzen-y · 2023-01-22T15:53:19Z

Blocked by #1737.

/hold

tenzen-y · 2023-01-22T16:09:37Z

sdk/python/kubeflow/training/api/training_client.py

        Args:
            namespace: Namespace to list the MPIJobs.
+            timeout: Optional, Kubernetes API server timeout in seconds
+                to execute the request.

        Returns:
            list[KubeflowOrgV1MPIJob]: List of MPIJobs objects. It returns
            empty list if MPIJobs cannot be found.
-            timeout: Optional, Kubernetes API server timeout in seconds
-                to execute the request.


The timeout seems to be Args, not Returns.

Nice catch!

tenzen-y · 2023-01-22T16:27:08Z

.github/workflows/integration-tests.yaml

+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true


Adding these parameters, when we push commits to the same PR, the already running Jobs are forcefully stopped, and new Jobs are started.

So we can reduce CI run times.

tenzen-y · 2023-01-22T19:02:23Z

The mxjob-controller watches Pods created by other FrameworkJob resources and updates MXJob Status :(

tenzen-y · 2023-01-22T19:05:32Z

The mxjob-controller watches Pods created by other FrameworkJob resources and updates MXJob Status :(

Ah, this is my misunderstanding.

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y · 2023-01-22T21:29:24Z

@johnugeorge This PR is ready for review, and all tests were passed. Please take a look.

tenzen-y · 2023-01-22T21:31:08Z

/hold cancel

johnugeorge · 2023-01-23T05:11:43Z

.github/workflows/integration-tests.yaml

 jobs:
  integration-test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        kubernetes-version: ["v1.23.12", "v1.24.6", "v1.25.2"]
+        # TODO (tenzen-y): Add volcano.
+        gang-scheduler-name: ["none", "scheduler-plugins"]


Good idea to extend to volcano in the future

Yes, we should run E2E with volcano. Although, we can follow up on other PRs.
So, I will create an issue to keep tracking this.

I created #1738.

sdk/python/test/e2e/utils.py

johnugeorge · 2023-01-23T05:22:27Z

Great work @tenzen-y for adding this e2e-test

/cc @andreyvelich PTAL at SDK changes

google-oss-prow · 2023-01-23T05:22:31Z

@johnugeorge: GitHub didn't allow me to request PR reviews from the following users: PTAL, at, SDK, changes.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Great work @tenzen-y for adding this e2e-test

/cc @andreyvelich PTAL at SDK changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

andreyvelich

Thank you for this feature @tenzen-y !
I left few comments.

sdk/python/test/e2e/test_e2e_tfjob.py

sdk/python/kubeflow/training/api/training_client.py

andreyvelich · 2023-01-23T14:56:25Z

sdk/python/kubeflow/training/api/training_client.py

        Args:
            namespace: Namespace to list the MPIJobs.
+            timeout: Optional, Kubernetes API server timeout in seconds
+                to execute the request.

        Returns:
            list[KubeflowOrgV1MPIJob]: List of MPIJobs objects. It returns
            empty list if MPIJobs cannot be found.
-            timeout: Optional, Kubernetes API server timeout in seconds
-                to execute the request.


Nice catch!

sdk/python/kubeflow/training/utils/utils.py

sdk/python/test/e2e/constants.py

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

andreyvelich · 2023-01-24T11:20:36Z

Thanks for the updates @tenzen-y!
/lgtm
/assign @johnugeorge @terrytangyuan

johnugeorge · 2023-01-24T11:47:10Z

Thanks @tenzen-y
/approve

google-oss-prow · 2023-01-24T11:47:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot added the do-not-merge/work-in-progress label Jan 22, 2023

google-oss-prow bot requested review from jinchihe and kuizhiqing January 22, 2023 00:46

google-oss-prow bot added the size/XXL label Jan 22, 2023

tenzen-y force-pushed the add-coscheduling-e2e branch 5 times, most recently from db995b5 to a976ca6 Compare January 22, 2023 15:10

tenzen-y mentioned this pull request Jan 22, 2023

Fix XGBoost conditions bug #1737

Merged

1 task

tenzen-y force-pushed the add-coscheduling-e2e branch from a976ca6 to eb319db Compare January 22, 2023 15:52

google-oss-prow bot added the do-not-merge/hold label Jan 22, 2023

tenzen-y changed the title ~~[WIP] Add E2E test for Scheduler Plugins with coscheduling~~ [WIP] Add E2E test for gang-scheduling Jan 22, 2023

tenzen-y force-pushed the add-coscheduling-e2e branch 2 times, most recently from 306e607 to 9a7a3ac Compare January 22, 2023 16:08

tenzen-y commented Jan 22, 2023

View reviewed changes

tenzen-y changed the title ~~[WIP] Add E2E test for gang-scheduling~~ Add E2E test for gang-scheduling Jan 22, 2023

google-oss-prow bot removed the do-not-merge/work-in-progress label Jan 22, 2023

tenzen-y force-pushed the add-coscheduling-e2e branch from 9a7a3ac to 0912922 Compare January 22, 2023 16:19

tenzen-y commented Jan 22, 2023

View reviewed changes

tenzen-y force-pushed the add-coscheduling-e2e branch 3 times, most recently from 30afcaa to 977d2f9 Compare January 22, 2023 18:48

tenzen-y changed the title ~~Add E2E test for gang-scheduling~~ [WIP] Add E2E test for gang-scheduling Jan 22, 2023

google-oss-prow bot added the do-not-merge/work-in-progress label Jan 22, 2023

tenzen-y force-pushed the add-coscheduling-e2e branch from 977d2f9 to 6d90d36 Compare January 22, 2023 20:02

tenzen-y force-pushed the add-coscheduling-e2e branch 2 times, most recently from 50c3004 to 4f4c0da Compare January 22, 2023 21:00

Add E2E test for gang-scheduling

758c31d

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y force-pushed the add-coscheduling-e2e branch from 4f4c0da to 758c31d Compare January 22, 2023 21:10

tenzen-y changed the title ~~[WIP] Add E2E test for gang-scheduling~~ Add E2E test for gang-scheduling Jan 22, 2023

google-oss-prow bot removed the do-not-merge/work-in-progress label Jan 22, 2023

google-oss-prow bot removed the do-not-merge/hold label Jan 22, 2023

johnugeorge reviewed Jan 23, 2023

View reviewed changes

sdk/python/test/e2e/utils.py Show resolved Hide resolved

google-oss-prow bot requested a review from andreyvelich January 23, 2023 05:22

tenzen-y mentioned this pull request Jan 23, 2023

Add E2E with volcano for ganga-scheduling #1738

Closed

Add a TODO comment for volcano

c5b09b5

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

andreyvelich reviewed Jan 23, 2023

View reviewed changes

Fix some nit

a43a29b

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

google-oss-prow bot assigned johnugeorge, terrytangyuan and andreyvelich Jan 24, 2023

google-oss-prow bot added the lgtm label Jan 24, 2023

google-oss-prow bot added the approved label Jan 24, 2023

google-oss-prow bot merged commit 307f17a into kubeflow:master Jan 24, 2023

tenzen-y deleted the add-coscheduling-e2e branch January 24, 2023 11:50

lowang-bh mentioned this pull request Jun 13, 2023

e2e testing support volcano gang-scheduler #1831

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add E2E test for gang-scheduling #1736

Add E2E test for gang-scheduling #1736

tenzen-y commented Jan 22, 2023 •

edited

Loading

coveralls commented Jan 22, 2023 •

edited

Loading

tenzen-y commented Jan 22, 2023 •

edited

Loading

tenzen-y Jan 22, 2023

andreyvelich Jan 23, 2023

tenzen-y Jan 22, 2023 •

edited

Loading

tenzen-y commented Jan 22, 2023

tenzen-y commented Jan 22, 2023

tenzen-y commented Jan 22, 2023 •

edited

Loading

tenzen-y commented Jan 22, 2023

johnugeorge Jan 23, 2023

tenzen-y Jan 23, 2023

tenzen-y Jan 23, 2023

johnugeorge commented Jan 23, 2023

google-oss-prow bot commented Jan 23, 2023

andreyvelich left a comment

andreyvelich Jan 23, 2023

andreyvelich commented Jan 24, 2023

johnugeorge commented Jan 24, 2023

google-oss-prow bot commented Jan 24, 2023

Add E2E test for gang-scheduling #1736

Add E2E test for gang-scheduling #1736

Conversation

tenzen-y commented Jan 22, 2023 • edited Loading

coveralls commented Jan 22, 2023 • edited Loading

Pull Request Test Coverage Report for Build 3994952641

💛 - Coveralls

tenzen-y commented Jan 22, 2023 • edited Loading

tenzen-y Jan 22, 2023

Choose a reason for hiding this comment

andreyvelich Jan 23, 2023

Choose a reason for hiding this comment

tenzen-y Jan 22, 2023 • edited Loading

Choose a reason for hiding this comment

tenzen-y commented Jan 22, 2023

tenzen-y commented Jan 22, 2023

tenzen-y commented Jan 22, 2023 • edited Loading

tenzen-y commented Jan 22, 2023

johnugeorge Jan 23, 2023

Choose a reason for hiding this comment

tenzen-y Jan 23, 2023

Choose a reason for hiding this comment

tenzen-y Jan 23, 2023

Choose a reason for hiding this comment

johnugeorge commented Jan 23, 2023

google-oss-prow bot commented Jan 23, 2023

andreyvelich left a comment

Choose a reason for hiding this comment

andreyvelich Jan 23, 2023

Choose a reason for hiding this comment

andreyvelich commented Jan 24, 2023

johnugeorge commented Jan 24, 2023

google-oss-prow bot commented Jan 24, 2023

tenzen-y commented Jan 22, 2023 •

edited

Loading

coveralls commented Jan 22, 2023 •

edited

Loading

tenzen-y commented Jan 22, 2023 •

edited

Loading

tenzen-y Jan 22, 2023 •

edited

Loading

tenzen-y commented Jan 22, 2023 •

edited

Loading