Signed-off-by: pdmurray <peynmurray@gmail.com>

Signed-off-by: pdmurray <peynmurray@gmail.com>

This PR adds log rotation for Ray Serve, letting it inherit rotation parameters (max_bytes, backup_count) from Ray Core, bringing a more consistent logging experience to Ray (as opposed to having the serve/ folder grow forever while the other logs rotate.

This PR adds additional information to the driver task event, namely, driver task type, and it's running/finished timestamps. This allows users (i.e. the dashboard) to inspect driver task more easily.
This PR also exposes the exclude_driver flag to state API, allowing requests through https and ListAPiOptions to get driver tasks, while the default behaviour from state API will still be excluding it.
This PR also filters out any tasks w/o task_info to prevent missing data issue.

If deployment is repeatedly failing, perform exponential backoff so as to not repeatedly try to restart the replica at a very fast rate.

Related issue number
Closes #31121

Co-authored-by: Cade Daniel <edacih@gmail.com>
Closes #31880

Adds back the metrics page
Adds button to visit new dashboard and to go back
Adds buttons for leaving feedback and viewing docs
Add color to status badges of tasks and placement groups table
Add alert when grafana is not running
Fix copy button icon
Separate metrics page into sections (both new IA and old IA)

…es to specify it (#31959)

This PR clarifies where RunConfig can be specified. Also, when multiple configs are specified in different locations (in the Tuner and Trainer), this PR also logs information about which RunConfig is actually used.

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

581cd4e moved some test files, breaking a link from the documentation. cc @iycheng

3343c76 changed the MapBatches string representation, breaking a docstring test. cc @peytondmurray

Signed-off-by: Kai Fricke <kai@anyscale.com>

…cution state, and task submitters. (#31986)

…d management (#31979)

Before this PR, stalls in the consumer thread would fully block the control loop. This provides backpressure, but at the cost of performance.

This PR fully decouples the consumer thread from the control loop thread, allowing execution to proceed so long as there is sufficient object_store_memory budget remaining. It also adds a progress bar for the output queue, showing the number of output bundles consumed and the number of queued bundles for output:

…2010)

#31669 changed the `Trial.__dict__` by moving `local_dir` to `_local_dir`, which resulted in an error in our tune cloud tests. This PR updates the signature of the `TrialStub` class to resolve the issue.

Signed-off-by: Kai Fricke <kai@anyscale.com>

…1993)

Remove legacy memory monitor from worker submission code path, as that was already disabled by default in Ray 2.2

The structure of the content looks good. My main request is (like with the scheduling refactor), that we make this discoverable with links from the main task/actor sections. Could we add 2-3 links each from the main tasks/actors/objects content to the appropriate fault tolerance sections?

_Originally posted by @ericl in #27573 (review)

Co-authored-by: Yi Cheng <74173148+iycheng@users.noreply.github.com>
Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com>

The long_running_serve_failure test uses a long-running actor, RandomKiller, to randomly kill Serve actors. This change sets the RandomKiller's max_restarts and max_task_retries to -1, so it can restart after crashes.

Related issue number
Addresses #31741

…tring (#31840)

Signed-off-by: rickyyx <rickyx@anyscale.com>

This PR introduces a flag RAY_task_events_send_batch_size that controls the number of task events sent to GCS in a batch. With default setting, each core worker will send 10K task events per second to GCS, where GCS could handle 10K task events in ~50 milliseconds.

This PR also adjust the worker side buffer limit to 1M with the new batching setting.

The PR adds some debug informations as well.

…e_node` release test (#31904)

The release test read_parquet_benchmark_single_node fails, due to using Python 3.7 and not having the pickle5 package installed. A similar issue is discussed in #26225. We found that the test failure is contained to the portion which tests a Dataset with a filter expression (the error is related to pickling with this filter expression).

Therefore, we will temporarily disable this portion of the test, while keeping the rest of the release test (which I verified passes on the same cluster). We can come back to this in the future and fix the case with filter. Example of release test successfully running with the filter case removed.

Signed-off-by: Scott Lee <sjl@anyscale.com>

…imizer (#31985)

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

The following operations call map_batches directly: add_column, drop_columns, select_columns, random_sample.

In this PR we add e2e tests for these examples with the new optimizer. In a future PR, we should refactor so that these operations do not call into map_batches and instead have their own logical operator.

…info on failure (#32014)

It appears the root cause of flaky failures described in #31981 is suppressed because we're not logging exceptions in `exponential_backoff_retry`.

Signed-off-by: Cade Daniel <cade@anyscale.com>

… node manager. (#31917)" (#31995)

This reverts commit a32b9b1.

…`MapOperator` actor pool. (#31987)

This PR adds support for autoscaling to the actor pool implementation of `MapOperator` (this PR is stacked on top of #31986).

The same autoscaling policy as the legacy `ActorPoolStrategy` is maintained, as well as providing more aggressive and sensible downscaling via:
* If there are more idle actors than running/pending actors, scale down.
* Once we're done submitting tasks, cancel pending actors and kill idle actors.

In addition to autoscaling, `max_tasks_in_flight` capping is also implemented.

<img width="1731" alt="Screen Shot 2023-01-24 at 1 01 25 AM" src="https://user-images.githubusercontent.com/18510752/214250430-9bac7b06-56fb-44b3-a044-3eaf726d1469.png">

This PR adds the cluster utilization page in the landing view

Co-authored-by: Alan Guo <aguo@anyscale.com>

This PR adds logical operator for randomize_block_order(). The change includes:

Introduce AbstractAllToAll for all logical operators converted to AllToAllOperator
RandomizeBlocks logical operator for randomize_block_order().
_internal/planner to move logic for Planner here and have generated function for randomize_blocks. This can be used later to create MapOperator/AllToAllOperator.

Add code owner to GCS module.

Signed-off-by: Cheng Su <scnju13@gmail.com>

Co-authored-by: Amog Kamsetty <amogkam@users.noreply.github.com>
Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>

Signed-off-by: SangBin Cho <rkooo567@gmail.com>

This PR implements the timeline to the ray dashboard using new task backend.

Implement the task events -> chrome tracing logic. Most of code is copied from existing code. TODO add unit tests (although we already have one, it is a pretty weak test).
Create a timeline endpoint that can 1. download the json file (to download & upload manually) 2. return the json array buffer (to load onto perfetto directly)
Create a subsection that has 3 features. 1. Download button. 2. Open perfetto button. 3. Instruction accordion.

Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>

…d debugstring (#31840)" (#32024)

This reverts commit 5d1f2e4.

This PR adds seealso notes to help users distinguish between map, flat_map, and map_batches.

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>

Signed-off-by: Eric Liang <ekhliang@gmail.com>

Without this patch, several of the help text are missing whitespace. For
example, `--dashboard-host` appears as follows:

  --dashboard-host TEXT           the host to bind the dashboard server to,
                                  either localhost (127.0.0.1) or 0.0.0.0
                                  (available from all interfaces). By default,
                                  thisis localhost.

This patch adds the correct trailing whitespace so there are spaces.

Signed-off-by: Luke Hsiao <luke.hsiao@numbersstation.ai>

…igurable. (#31960)

…rs (#31991)

* trying out a new configuration pattern for trainer runner and rl trainers

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

#32036)

… a MultiAgent env. (#31480)

…2002)

This PR adds a DrainAndKillNode endpoint to the monitor service. It has the exact same semantics as the GcsNodeManager::HandleDrainNode.


---------

Co-authored-by: Alex <alex@anyscale.com>

Checkpointable actor is already removed in #10333

…moved by node manager."" (#32019)

This reverts commit 51c5eda.

Reverts #31995

Skip the windows test.

Signed-off-by: Yi Cheng <74173148+iycheng@users.noreply.github.com>

Mixins don't work well with reuse_actors because the init is only called on construction. In the case of mlflow, this means that reused actors will try to overwrite state from the trials that previously ran on them. This is incorrect behavior and errors on the mlflow server side.

Thus, we should default to not reuse actors for mixins.

Signed-off-by: Kai Fricke <kai@anyscale.com>

Signed-off-by: Eric Liang <ekhliang@gmail.com>

Fixes:
- Properly wire max tasks per actor to pool
- Account for internal queue size in scheduling algorithm
- Small improvements to progress bar UX

#31337 has become flaky again due to a low timeout. This PR follows #31338 and increases the timeout.

We currently have no canonical way to await actors. Users can define their own _is-ready_ methods, schedule a future, and await these, but this has to be done for every actor class separately.

This does not match other patterns - e.g. we have `actor.__ray_terminate__.remote()` for actor termination and `placement_group.ready()` for placement group ready futures.

This PR adds a new `__ray_ready__` magic actor method that just returns `True`. It can be used to await actors becoming ready (newly scheduled actors), and actors having processed all of their other enqueued tasks.

Signed-off-by: Kai Fricke <kai@anyscale.com>

The long_running_serve_failure release test is marked as unstable due to recent failures. Recently, #31945 and #32011 have resolved the root causes of these failures. After those changes, the test ran successfully for 15+ hours without failure. This change limits the test's iterations, so it doesn't run forever, and it marks the test as stable.

Reduce the timeout for many nodes actor test given that a test should finish within 1h.
It can save some cost for problematic runs.

This PR is a quick fix to remove the non-useful comment introduced in #31526, probably during debugging.

Replace the comment with a meaningful one.

Signed-off-by: Eric Liang <ekhliang@gmail.com>

Combine tasks and actors sections
Move object store memory back up to the logical section (it's one of the most useful metrics, it shouldn't be buried)
Improve titles

Currently, the dropdown menu "Resources" in the Ray documentation contains a link called "Training." This link points to the [same site](https://www.anyscale.com/events) as "Events." However, we want this to direct to the repository of [technical training content](https://github.com/ray-project/ray-educational-materials).

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>

…ult tolerance (#31949)

This PR adds the documentation and sample config files for deploying Ray to K8S without using KubeRay. As KubeRay CRDs need cluster-scoped permissions, this PR helps those users who do not have cluster-scoped permissions to install Ray Cluster in their K8S.

Signed-off-by: Alan Guo <aguo@anyscale.com>

This progress bar automatically shows progress by groupings.

Things that belong to the same parent are all put in a group.
If a group has multiple children with the same name, those are merged together into a virtual group.

These virtual groups have different visual treatment because a virtual group should not add an additional level of nesting.

… execute commands on databricks notebook for a long time (#31962)

Databricks Runtime provides an API:
dbutils.entry_point.getIdleTimeMillisSinceLastNotebookExecution() that returns elapsed milliseconds since last databricks notebook code execution.
This PR code calls this interface to monitor notebook activity and shut down Ray cluster on timeout.

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

…` and `ArrowVariableShapedTensorArray` (#31817)

Add support for creating ArrowTensorArrays and ArrowVariableShapedTensorArrays with string typed columns.

Signed-off-by: Scott Lee <sjl@anyscale.com>

…apes` (but `reduce_retracing` instead). (#29214)

Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>

…e w/ pettingzoo. (#31820)

… foundation's gymnasium" (from "OpenAI gym"). (#32061)

…32101)

This PR is to fix master with resolving the conflict between #32080 and #32081, i.e.

- Pass TaskContext in random_shuffle.py:generate_random_shuffle_fn()
- Add AllToAllTransformFn and rename TransformFn to MapTransformFn
- Update the function return type in generate_map_xxx_fn().

Signed-off-by: Cheng Su <scnju13@gmail.com>

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* [air] Add test for remote_storage with real hdfs backend.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* try a different syntax.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* change `install-hdfs.sh` permission.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* -hdfs in air tests.

update ssh-kengen command.

fix a few typos.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* test_env=

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* cat hdfs_env

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* move `PATH` as well to a separate file.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* setting env vars in test only.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* nit

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix fixture

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

…xfail (#32072)

* Marking RLLib release tests as unstable if xfail

This PR adds logical operator for `repartition()`. Only implement shuffle repartition (`repartition.py:generate_repartition_fn()`).

Non-shuffle repartition is left as TODO, as the corresponding code in [fast_repartition.py](https://github.com/ray-project/ray/blob/master/python/ray/data/_internal/fast_repartition.py) involves `BlockList`, `ExecutionPlan` and `Dataset.split()`, so it needs a deeper refactoring and code change.

This PR exposes the MultiGet operation to the InternalKVInterface. The MultiGet operation is already supported in the two backends (InMemory and Redis), so this PR is just plumbing.

This change is needed to support getting multiple keys from the Internal KV in a single RPC.

…sorArray` and `ArrowVariableShapedTensorArray` (#31817)" (#32123)

This reverts commit 1fdf24e.

This adds an option to the AIR DatasetConfig for a preprocessor that gets reapplied on each epoch. Currently the implementation uses DatasetPipeline to ensure that the extra preprocessing step is overlapped with training.

Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>

…scaling (#32085)

The previous way pending_nodes was calculated was prone to race conditions, instead, let's just always publish it in the main thread with other metrics.

Closes #31982

---------

Co-authored-by: Alex <alex@anyscale.com>

…r lookup (#32087)

In #30016 we migrated Ray Tune to use a new resource management interface. In the same PR, we simplified the resource consolidation logic. This lead to a performance regression first identified in #31337.

After manual profiling, the regression seems to come from `RayTrialExecutor._count_staged_resources`. We have 1000 staged trials, and this function is called on every step, executing a linear scan through all trials.

This PR fixes this performance bottleneck by keeping state of the resource counter instead of dynamically recreating it every time. This is simple as we can just add/subtract the resources whenever we add/remove from the `RayTrialExecutor._staged_trials` set.

Manual testing confirmed this improves the runtime of `tune_scalability_result_throughput_cluster` from ~132 seconds to ~122 seconds, bringing it back to the same level as before the refactor.

Signed-off-by: Kai Fricke <kai@anyscale.com>

…RLTrainers (#31991)" (#32130)

Reverts #31991

This PR seems to have broken CI.

Screenshot 2023-01-31 at 1 39 09 PM

The error is https://buildkite.com/ray-project/oss-ci-build-branch/builds/2099#01860972-e02e-47c4-8f86-8be28ea18d92/3786-3992
AttributeError: '_TFStub' object has no attribute 'Tensor'

. So instead of averaging out, we should do sum(gpu_utillization) / (sum(num_gpus)) to cap the max percentage to 100%.

With the recent updating of the nightly tests, update the data here.

In the nightly tests, we use 2k nodes (2cpus per node) and 20k actors, but if better node is used, we can run more than 40k actors.

https://buildkite.com/ray-project/release-tests-branch/builds/1321#018604d7-86a3-4fad-ac6c-803db73821d3

Signed-off-by: Alan Guo <aguo@anyscale.com>

fix lint #31750

…on planner. (#32095)

This PR adds operation fusion to the new execution planner.

* Remove empty parser.add_argument() in test file
* remove --framework=torch
* fix BUILD
* use training_iteration as stopping cirterion

Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>

* [release] minor fix to pytorch_pbt_failure test when using gpu. (#32070)

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

Datasets depends on ray.air for several key features (tensor extensions, Arrow transformations, data batch conversions), and not running the Datasets test suite in PR builds on ray.air changes has caused breaks to go undetected. This PR changes this so when files under python/ray/air change, we trigger the Datasets test suite in CI.

Signed-off-by: Clark Zinzow <clarkzinzow@gmail.com>

…n` for remote URI (#32110)

At least two users reported encountering

ImportError(
                            "You must `pip install smart_open` and "
                            "`pip install boto3` to fetch URIs in s3 "
                            "bucket. "
and trying to fix it by specifying them in the pip field of runtime_env, which won't work because the runtime_env setup code doesn't run inside the runtime_env. This PR clarifies the error message to say that they must be preinstalled on the cluster, and adds a note to the docs.

…32126)

…d. (#31664)

Signed-off-by: SangBin Cho <rkooo567@gmail.com>

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

Why are these changes needed?
Fail the task if it is the last task of the group, per the new (group by parent) worker killing policy

Related issue number
#32149 32078


Co-authored-by: Clarence Ng <clarence@anyscale.com>

…me/ray to Ray Dockerfile." #32026

Signed-off-by: kaihsun <kaihsun@anyscale.com>

#31956

Upgrade to a version of gRPC that GHSA-cfmr-vrgj-vqwv in Zlib
1.46.6 has this patch: grpc/grpc#31845

Why are these changes needed?
Add a new protobuf for JobInfo from the Ray Job API
Augment the existing GCS GetAllJobInfo endpoint to return this information, if available (not all GCS jobs were submitted via the Ray Job API; these jobs won't have this extra JobInfo.)
Related issue number
Closes #29621

…tring (remerging #31840) (#32057)

Remerging #31840

This is the initial prototype of integrating ray status to the frontend.

I think we could've returned structured data from the backend, but I decided to parse ray status output from the frontend for quick implementation (so that we can support if from ray 2.3).

)

Signed-off-by: SangBin Cho <rkooo567@gmail.com>

<!-- Thank you for your contribution! Please review https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. -->

## Why are these changes needed?

This PR unpins the version of open telemetry as it is too strict for an experimental tracing feature.  

## Related issue number

Closes #32051

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
- [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

* Revert "Revert "[RLlib] Reparameterize the construction of TrainerRunner and RLTrainers (#31991)" (#32130)"

This reverts commit d15ccfc.

* added bool evaluation to tf stub so that if tf returns false

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

This PR takes over #28179

Why are these changes needed?
Today with the default scheduling policy, Ray will try to pack tasks on nodes until the resource utilization is beyond a certain threshold and spread tasks afterwards.
This has caused slow down the scheduling speed for embarrassingly parallel jobs:

we will only move on to another node before the current node's resource if sufficiently utilized,
for each node, the overhead of accepting new job and starting of a new workers is not negligible.
the overall scheduling speed doesn't scale with the number of nodes;
This PR is one proposal to address the problem: instead of stick to one node, we randomly choose one node from top-k nodes for the default scheduling, where the node is sorted by it's resource utilization in reverse order.

Intuitively, this allows us to kick off the workers startup on multiple node in parallel of the scheduling.

benchmark result:

baseline: 10 parallelism, top 1, 25 tasks/second
10 parallelism, top 6, 30 tasks/second
64 parallelism, top 6, 126
100 parallelism, top 6, 150
1000 parallelism, top 6, 374.8676886257549
10 concurrent, top 12, 176
64 concurrent, top 12, 182.59477988042443 tasks/s
128 concurrent, top 12, 245.9862948998163
256 concurrent, top 12, 298…

This PR adds actor detail page.

Other than the detail page, it also

Add pg id to task/actor
Add profiling links to job detail & job row & actor detail

This PR is to add logical operator for `sort()`, the change includes:
* `Sort` logical operator
* `SortTaskSpec` to copy from `sort.py`
* `generate_sort_fn` is generated function for sort