Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] linux://python/ray/data:tests/test_dataset is failing/flaky on master. #32067

Closed
rickyyx opened this issue Jan 30, 2023 · 8 comments
Closed

[CI] linux://python/ray/data:tests/test_dataset is failing/flaky on master. #32067

rickyyx opened this issue Jan 30, 2023 · 8 comments
Assignees
@c21 @jianoaix
Labels
flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/

Comments

@rickyyx
Copy link
Contributor

rickyyx commented Jan 30, 2023

....
Generated from flaky test tracker. Please do not edit the signature in this section.
DataCaseName-linux://python/ray/data:tests/test_dataset-END
....
@rickyyx rickyyx added the flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ label Jan 30, 2023
@rickyyx
Copy link
Contributor Author

rickyyx commented Jan 30, 2023

cc data oncall

@jianoaix
Copy link
Contributor

This should have been fixed at: #31930. Is it not the case?

@rickyyx
Copy link
Contributor Author

rickyyx commented Jan 30, 2023

Looks like the test is kind of flaky?

@rickyyx
Copy link
Contributor Author

rickyyx commented Jan 30, 2023

image

@jianoaix
Copy link
Contributor

Yeah, it however doesn't seem to have any failed tests, so unclear what's failing, e.g. https://buildkite.com/ray-project/oss-ci-build-branch/builds/2061#0186036d-cb12-4fb7-a4a7-b04ea63c8ab3 cc @krfricke

@cadedaniel
Copy link
Member

This is causing false-positives in the 2.3 Ray release. I haven't seen it fail, just timing out 3/3.

@jianoaix jianoaix mentioned this issue Feb 3, 2023
Merged
7 tasks
@jianoaix
Copy link
Contributor

jianoaix commented Feb 3, 2023

Got a failure with useful message: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2165#_

It looks there was a running cluster when this test tried to start a new one, fix: #32200

@jianoaix jianoaix mentioned this issue Feb 13, 2023
Merged
7 tasks
ericl pushed a commit that referenced this issue Feb 14, 2023
@jianoaix
Deflake test_dataset.py: split torch tests (#32487)
b89457a 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the torch tests from this big test file.

#32067
@jianoaix jianoaix mentioned this issue Feb 14, 2023
Merged
7 tasks
ericl pushed a commit that referenced this issue Mar 1, 2023
@jianoaix
Deflake test_dataset.py: split tensor tests (#32540)
371b330 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the tensor tests from this big test file.

#32067
@c21 c21 assigned c21 and unassigned scottjlee Mar 7, 2023
@c21
Copy link
Contributor

c21 commented Mar 7, 2023

update: #33101 is merged, and should fix the flaky completely.

Seeing promising trend now as below. Will wait for 1 day before closing this issue.

Screen Shot 2023-03-07 at 12 35 22 PM

ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this issue Mar 21, 2023
@jianoaix @ProjectsByJackHe
Deflake test_dataset.py: split tensor tests (ray-project#32540)
c1632f6 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the tensor tests from this big test file.

ray-project#32067
Signed-off-by: Jack He <jackhe2345@gmail.com>
edoakes pushed a commit to edoakes/ray that referenced this issue Mar 22, 2023
@jianoaix @edoakes
Deflake test_dataset.py: split torch tests (ray-project#32487)
1c444f1 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the torch tests from this big test file.

ray-project#32067
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
edoakes pushed a commit to edoakes/ray that referenced this issue Mar 22, 2023
@jianoaix @edoakes
Deflake test_dataset.py: split tensor tests (ray-project#32540)
6e1cf2f 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the tensor tests from this big test file.

ray-project#32067
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
peytondmurray pushed a commit to peytondmurray/ray that referenced this issue Mar 22, 2023
@jianoaix @peytondmurray
Deflake test_dataset.py: split tensor tests (ray-project#32540)
bc4a6ae 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the tensor tests from this big test file.

ray-project#32067
elliottower pushed a commit to elliottower/ray that referenced this issue Apr 22, 2023
@jianoaix @elliottower
Deflake test_dataset.py: split torch tests (ray-project#32487)
777cd09 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the torch tests from this big test file.

ray-project#32067
Signed-off-by: elliottower <elliot@elliottower.com>
elliottower pushed a commit to elliottower/ray that referenced this issue Apr 22, 2023
@jianoaix @elliottower
Deflake test_dataset.py: split tensor tests (ray-project#32540)
7d244ba 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the tensor tests from this big test file.

ray-project#32067
Signed-off-by: elliottower <elliot@elliottower.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this issue May 4, 2023
@jianoaix @ProjectsByJackHe
Deflake test_dataset.py: split tensor tests (ray-project#32540)
1c1dfd6 
One of the flakiness of test_dataset.py is due to the timeout. This splits out the tensor tests from this big test file.

ray-project#32067
Signed-off-by: Jack He <jackhe2345@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@c21 c21

@jianoaix jianoaix

Labels
flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@cadedaniel @c21 @scottjlee @rickyyx @jianoaix