[ci][core] Mark python tests flaky on CI #41195
Conversation
| @@ -327,6 +327,10 @@ def has_client_deprecation_warn(warning: Warning, expected_replacement: str) -> | |||
| @pytest.mark.filterwarnings( | |||
| "default:Starting a connection through `ray.client` will be deprecated" | |||
| ) | |||
| @pytest.mark.skipif( | |||
There was a problem hiding this comment.
can you make a wrapper like "run_only_in_postmerge()"?
There was a problem hiding this comment.
Also @can-anyscale is there some sort of guarantee this env var won't change? Would like to avoid the env var name/behavior is changed, and we stop tracking flaky tests
There was a problem hiding this comment.
I think it can only be guarantee through code reviews. But unless core change it, I don't think a random engineer will change it though.
There was a problem hiding this comment.
can you make a wrapper like "run_only_in_postmerge()"?
I thought this would actually bring in additional dependency such as a test_util.py of where we define this run_only_in_postmerge() func
There was a problem hiding this comment.
Okay. I don't think we will ever change the env. If there are times that this need to be changed, let us know!
@rickyyx I think that's probably fine? We don't have that many tests anyway. I want to make sure we have corresponding issue (so we can ensure reason is always passed properly) whenever we mark it like this.
There was a problem hiding this comment.
I want to make sure we have corresponding issue (so we can ensure reason is always passed properly) whenever we mark it like this.
that's a good point.
There was a problem hiding this comment.
actually - it's required to pass in reason with skipif using booleans.
|
Can we also skip this one https://buildkite.com/ray-project/postmerge/builds/1690#018bd506-9d94-4111-bd86-c64149eb997a/6-565; then I don't need this PR (#41189), thankks |
rickyyx
changed the title
|
docbuild is broken and blocking all merges :( |
|
you probably cannot merge because the docbuild is broken on master; let me know if you want to bypass and merge it |
|
@can-anyscale will you be making a PR to enable this flag and actual remove the test names from the flakey test file? |
|
@rickyyx yes, will work on it today |
--------- Signed-off-by: rickyyx <rickyx@anyscale.com>
Why are these changes needed?
We are now tracking core flaky tests at functions level:
linux://python/ray/tests:test_client_builderis failing/flaky on master. #38224linux://python/ray/tests:test_object_assign_owner_client_modeis failing/flaky on master. #41175linux://python/ray/tests:test_placement_groupis failing/flaky on master. #38726linux://python/ray/tests:test_runtime_env_working_dir_3is failing/flaky on master. #40781linux://python/ray/tests:test_state_api_logis failing/flaky on master. #40959linux://python/ray/tests:test_failure_3is failing/flaky on master. #41188Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.