[Ray 2.3 release] Last update to results json was too long ago CI flaky failure

[cadedaniel]

• long_running_many_actor_tasks [smoke test] https://buildkite.com/ray-project/release-tests-branch/builds/1309#0185e5f7-46d3-4965-86c5-d30d800ad47e
• long_running_actor_deaths [smoke test] https://buildkite.com/ray-project/release-tests-branch/builds/1307#0185e0b3-c5af-4020-9ed0-9b33d84e131a

Traceback (most recent call last):
--
  | File "ray_release/scripts/run_release_test.py", line 153, in main
  | no_terminate=no_terminate,
  | File "/tmp/release-y0g2zN9kOn/release/ray_release/glue.py", line 404, in run_release_test
  | raise pipeline_exception
  | File "/tmp/release-y0g2zN9kOn/release/ray_release/glue.py", line 382, in run_release_test
  | handle_result(test, result)
  | File "/tmp/release-y0g2zN9kOn/release/ray_release/alerts/handle.py", line 39, in handle_result
  | raise ResultsAlert(error)
  | ray_release.exception.ResultsAlert: Last update to results json was too long ago (inf > 300)

[cadedaniel]

from Kai:

[2:29 PM] The main problem is that the results file can’t be fetched:

ray_release.exception.FileDownloadError: Error downloading file
/tmp/release_test_out.json to /tmp/tmpx3c1t6pi

[2:29] And then the alerting step checks the last update time, but it defaults to inf if no data is found. That raises the error
[2:30] So the main problem is that either the file was not written or the download did not work correctly

[krfricke]

To add to this, this result fetching procedure works for other tests. So if the "download did not work correctly" it could be e.g. because the jobs server died

[cadedaniel]

for long_running_many_actor_tasks , the root cause exception is swallowed by the exception handler in an exponential backoff handler. Have a fix in #32014 but ofc not the root cause

Will look into long_running_actor_deaths now

[cadedaniel]

The long_running_actor_deaths failure looks to have the same issue

[cadedaniel]

Update: I thought that the exception handler around Executing pip install -q awscli && aws s3 cp /tmp/release_test_out.json s3://ray-release-automation-results/tmp/qjlzkodchv --acl bucket-owner-full-control with {} via ray job submit was failing but actually looks like the failure is in a different codepath -- in the Anyscale SDK itself (Could not fetch results from session). Will look more later.

In any case, these tests aren't crashing, and we currently don't check for performance regressions on them. So the tests are passing, we're just missing metrics from them with this bug.