Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ray 2.3 Release] stress_test_many_tasks.json stage_3_time has 17% regression in 2.3.0 #32235

Closed
cadedaniel opened this issue Feb 6, 2023 · 5 comments · Fixed by #32251
Closed

[Ray 2.3 Release] stress_test_many_tasks.json stage_3_time has 17% regression in 2.3.0 #32235

cadedaniel opened this issue Feb 6, 2023 · 5 comments · Fixed by #32251
Assignees
@rickyyx
Labels
core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release

Comments

@cadedaniel
Copy link
Member

From preliminary performance comparison PR: https://github.com/ray-project/ray/pull/32086/files

  • stage_0_time 54% regression
  • stage_3_time 17% regression
@cadedaniel cadedaniel added release-blocker P0 Issue that blocks the release P0 Issues that should be fixed in short order core Issues that should be addressed in Ray Core labels Feb 6, 2023
@cadedaniel
Copy link
Member Author

@rickyyx: The 54% regression might just be due to high variance

image

@scv119 scv119 assigned scv119 and unassigned rickyyx Feb 6, 2023
@cadedaniel
Copy link
Member Author

The release test is timing out, likely because of this https://buildkite.com/ray-project/release-tests-branch/builds/1333#0186150b-32aa-4666-aaa2-688171f796f3

@rkooo567 rkooo567 assigned scv119 and rickyyx and unassigned scv119 Feb 6, 2023
@rkooo567
Copy link
Contributor

rkooo567 commented Feb 6, 2023

We will see if @rickyyx's work on microbenchmark will fix stage 3

@scv119 scv119 removed their assignment Feb 7, 2023
@scv119
Copy link
Contributor

scv119 commented Feb 7, 2023

confirmed regression is caused by d74e4c4

@scv119
Copy link
Contributor

scv119 commented Feb 7, 2023 •
edited
Loading

d74e4c42eb [core][state] Adjust worker side reporting with batches && add debugstring (remerging #31840) (#32057)

INFO:main:Finished stage 1 after 383.62558245658875 seconds.

b2c5e63bf7 [Core] Join Ray Jobs API JobInfo with GCS JobTableData (#31046)

INFO:main:Finished stage 1 after 237.37327575683594 seconds.
INFO:main:Finished stage 2 after 257.5613226890564 seconds.

@scv119 scv119 linked a pull request Feb 8, 2023 that will close this issue
[core][state] Task backend improve performance #32251
Merged
7 tasks
@scv119 scv119 mentioned this issue Feb 8, 2023
Closed
@cadedaniel cadedaniel mentioned this issue Feb 9, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rickyyx rickyyx

Labels
core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[core][state] Task backend improve performance rickyyx/ray
4 participants
@scv119 @cadedaniel @rickyyx @rkooo567