[release] update if xgboost test suite requires result or not.

[xwjiang2010]

Some xgboost tests don't generate result, and we should not fail those tests. This is to fix a regression introduced by https://github.com/ray-project/ray/pull/32055/files

See failing case: https://buildkite.com/ray-project/release-tests-branch/builds/1381#0186332c-8e3c-4199-9315-359a6e1fecf9

Signed-off-by: xwjiang2010 xwjiang2010@gmail.com

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[xwjiang2010]

running at : https://buildkite.com/ray-project/release-tests-pr/builds/27828

[Yard1]

It looks like we just need to change alert to default in the release_tests.yaml for those tests? Not sure if we need any python code changes. Would be great if we could avoid adding special cases there.

[xwjiang2010]

It looks like we just need to change alert to default in the release_tests.yaml for those tests? Not sure if we need any python code changes. Would be great if we could avoid adding special cases there.

some of the xgboost tests do require result, which is captured in the require_result() function logic. For those tests, if result is fetched wrong, we would want to surface infra error instead of application error.

[Yard1]

Ok but we can just change it to alert: default for the tests that do not require it (xgboost_distributed_api_test, xgboost_ft_small_elastic, xgboost_ft_small_non_elastic), right? alert is set on the test level, after all.

[xwjiang2010]

Ok but we can just change it to alert: default for the tests that do not require it (xgboost_distributed_api_test, xgboost_ft_small_elastic, xgboost_ft_small_non_elastic), right? alert is set on the test level, after all.

ah updated.

[Yard1]

Thanks!