[observability][export-api] Write node events #47221

nikitavemuri · 2024-08-20T18:22:56Z

Why are these changes needed?

Write node events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false.
Event write is called whenever a value in the node event data schema is modified. Typically this occurs in the callback after writing NodeTable to the GCS table

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

can-anyscale · 2024-08-29T16:42:40Z

This broke #47395, i'm reverting to unblock

Same code changes as [observability][export-api] Write node events ray-project#47221 Move test into a separate file to create a separate bazel target that can be skipped on Windows Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Write node events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the node event data schema is modified. Typically this occurs in the callback after writing NodeTable to the GCS table Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

…7405) Reverts ray-project#47221 This broke ray-project#47395 Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Same code changes as [observability][export-api] Write node events ray-project#47221 Move test into a separate file to create a separate bazel target that can be skipped on Windows Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Write node events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the node event data schema is modified. Typically this occurs in the callback after writing NodeTable to the GCS table Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

…7405) Reverts ray-project#47221 This broke ray-project#47395 Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Same code changes as [observability][export-api] Write node events ray-project#47221 Move test into a separate file to create a separate bazel target that can be skipped on Windows Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Write node events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the node event data schema is modified. Typically this occurs in the callback after writing NodeTable to the GCS table Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

…7405) Reverts ray-project#47221 This broke ray-project#47395 Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Same code changes as [observability][export-api] Write node events ray-project#47221 Move test into a separate file to create a separate bazel target that can be skipped on Windows Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Write node events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the node event data schema is modified. Typically this occurs in the callback after writing NodeTable to the GCS table Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

…7405) Reverts ray-project#47221 This broke ray-project#47395 Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Same code changes as [observability][export-api] Write node events ray-project#47221 Move test into a separate file to create a separate bazel target that can be skipped on Windows Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Write node events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the node event data schema is modified. Typically this occurs in the callback after writing NodeTable to the GCS table Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

…7405) Reverts ray-project#47221 This broke ray-project#47395 Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Same code changes as [observability][export-api] Write node events ray-project#47221 Move test into a separate file to create a separate bazel target that can be skipped on Windows Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Write node events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the node event data schema is modified. Typically this occurs in the callback after writing NodeTable to the GCS table Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

…7405) Reverts ray-project#47221 This broke ray-project#47395 Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Same code changes as [observability][export-api] Write node events ray-project#47221 Move test into a separate file to create a separate bazel target that can be skipped on Windows Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Write node events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the node event data schema is modified. Typically this occurs in the callback after writing NodeTable to the GCS table Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

…7405) Reverts ray-project#47221 This broke ray-project#47395 Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Same code changes as [observability][export-api] Write node events ray-project#47221 Move test into a separate file to create a separate bazel target that can be skipped on Windows Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

Nikita Vemuri added 8 commits August 20, 2024 11:20

write event

fdb14ff

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

wip test - memory issue

418c611

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

add test

8941b59

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

move test util fn

144fd26

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

add feature flag

7c13a8d

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

formatting

196e016

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

formatting

b7305d6

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

formatting

8354907

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

nikitavemuri added the go add ONLY when ready to merge, run all tests label Aug 24, 2024

Nikita Vemuri added 12 commits August 23, 2024 18:11

formatting

10ebbf6

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

run tests

8bd7a86

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

separate test class for export

a7a7709

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

run test

398c691

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

run test

3a74aae

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

run tests

c241f39

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

fix test

ec1d976

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

synchronize with poll

5ed7152

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

synchronize with run

e1245b7

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

separate add and remove node tests

80af0d0

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

call io_service manually without separate thread

7f401ec

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

remove retry logic

6b4de92

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

nikitavemuri changed the title ~~[WIP] Write node events~~ [observability][export-api] Write node events Aug 28, 2024

update

b6fe6cc

Signed-off-by: Nikita Vemuri <nikitavemuri@anyscale.com>

nikitavemuri marked this pull request as ready for review August 28, 2024 17:54

nikitavemuri requested a review from a team as a code owner August 28, 2024 17:54

nikitavemuri requested a review from rkooo567 August 28, 2024 17:54

rkooo567 self-assigned this Aug 28, 2024

rkooo567 approved these changes Aug 28, 2024

View reviewed changes

rkooo567 merged commit d425f31 into ray-project:master Aug 28, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[observability][export-api] Write node events #47221

[observability][export-api] Write node events #47221

nikitavemuri commented Aug 20, 2024 •

edited

Loading

can-anyscale commented Aug 29, 2024

[observability][export-api] Write node events #47221

[observability][export-api] Write node events #47221

Conversation

nikitavemuri commented Aug 20, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

can-anyscale commented Aug 29, 2024

nikitavemuri commented Aug 20, 2024 •

edited

Loading