From 579995c54181b6a4d830ed1f93bc7398f7202f42 Mon Sep 17 00:00:00 2001 From: Peyton Murray Date: Mon, 26 Aug 2024 17:39:44 -0700 Subject: [PATCH] [Doc] Run pre-commit on core docs (#47004) ## Why are these changes needed? Currently we have no linting on any part of the docs code. This PR runs pre-commit on the core docs. This PR fixes the following issues: ``` trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing doc/source/ray-core/examples/overview.rst Fixing doc/source/ray-core/scheduling/placement-group.rst Fixing doc/source/ray-core/doc_code/streaming_generator.py Fixing doc/source/ray-core/tasks.rst Fixing doc/source/ray-core/actors/terminating-actors.rst Fixing doc/source/ray-core/scheduling/ray-oom-prevention.rst Fixing doc/source/ray-core/objects/serialization.rst Fixing doc/source/ray-core/using-ray-with-jupyter.rst Fixing doc/source/ray-core/doc_code/ray-dag.py Fixing doc/source/ray-core/fault_tolerance/tasks.rst Fixing doc/source/ray-core/actors.rst fix end of files.........................................................Failed - hook id: end-of-file-fixer - exit code: 1 - files were modified by this hook Fixing doc/source/ray-core/user-spawn-processes.rst Fixing doc/source/ray-core/api/cli.rst Fixing doc/source/ray-core/images/oom_killer_example_killed_two.svg Fixing doc/source/ray-core/examples/lm/preprocess.sh Fixing doc/source/ray-core/examples/plot_hyperparameter.ipynb Fixing doc/source/ray-core/patterns/concurrent-operations-async-actor.rst Fixing doc/source/ray-core/images/oom_killer_example_killed_one.svg Fixing doc/source/ray-core/images/memory.svg Fixing doc/source/ray-core/actors/terminating-actors.rst Fixing doc/source/ray-core/scheduling/resources.rst Fixing doc/source/ray-core/images/oom_killer_example.svg Fixing doc/source/ray-core/accelerator-types.rst Fixing doc/source/ray-core/examples/images/batch-training.svg Fixing doc/source/ray-core/images/physical_resources_vs_logical_resources.svg Fixing doc/source/ray-core/images/pong-arch.svg Fixing doc/source/ray-core/fault_tolerance/actors.rst Fixing doc/source/ray-core/examples/images/automl.svg Fixing doc/source/ray-core/examples/BUILD check for added large files..............................................Passed check python ast.........................................................Passed check json...............................................................Passed check toml...........................................(no files to check)Skipped black....................................................................Passed flake8...................................................................Passed prettier.............................................(no files to check)Skipped mypy.................................................(no files to check)Skipped isort (python)...........................................................Passed rst directives end with two colons.......................................Passed rst ``inline code`` next to normal text..................................Failed - hook id: rst-inline-touching-normal - exit code: 1 doc/source/ray-core/actors.rst:427:Ray tracks all child and Actor Tasks. When the``recursive=True`` argument is given, use logger.warning(......................................................Passed check for not-real mock methods..........................................Passed ShellCheck v0.9.0........................................................Passed clang-format.........................................(no files to check)Skipped Google Java Formatter................................(no files to check)Skipped Check for Ray docstyle violations........................................Passed Check for Ray import order violations....................................Passed ``` ## Related issue number Part of #42587. ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( Signed-off-by: pdmurray --- doc/source/ray-core/accelerator-types.rst | 2 +- doc/source/ray-core/actors.rst | 14 +++++----- .../ray-core/actors/terminating-actors.rst | 6 ++-- doc/source/ray-core/api/cli.rst | 2 +- doc/source/ray-core/doc_code/ray-dag.py | 4 +-- .../ray-core/doc_code/streaming_generator.py | 2 +- doc/source/ray-core/examples/BUILD | 2 +- .../ray-core/examples/images/automl.svg | 2 +- .../examples/images/batch-training.svg | 2 +- doc/source/ray-core/examples/lm/preprocess.sh | 2 +- doc/source/ray-core/examples/overview.rst | 4 +-- .../examples/plot_hyperparameter.ipynb | 2 +- .../ray-core/fault_tolerance/actors.rst | 1 - doc/source/ray-core/fault_tolerance/tasks.rst | 2 +- doc/source/ray-core/images/memory.svg | 2 +- .../ray-core/images/oom_killer_example.svg | 2 +- .../images/oom_killer_example_killed_one.svg | 2 +- .../images/oom_killer_example_killed_two.svg | 2 +- ...hysical_resources_vs_logical_resources.svg | 2 +- doc/source/ray-core/images/pong-arch.svg | 2 +- doc/source/ray-core/objects/serialization.rst | 2 +- .../concurrent-operations-async-actor.rst | 1 - .../ray-core/scheduling/placement-group.rst | 4 +-- .../scheduling/ray-oom-prevention.rst | 6 ++-- doc/source/ray-core/scheduling/resources.rst | 2 +- doc/source/ray-core/tasks.rst | 4 +-- doc/source/ray-core/user-spawn-processes.rst | 2 +- .../ray-core/using-ray-with-jupyter.rst | 28 +++++++++---------- 28 files changed, 53 insertions(+), 55 deletions(-) diff --git a/doc/source/ray-core/accelerator-types.rst b/doc/source/ray-core/accelerator-types.rst index a4559a038ca4..91f3fdaaeb46 100644 --- a/doc/source/ray-core/accelerator-types.rst +++ b/doc/source/ray-core/accelerator-types.rst @@ -8,4 +8,4 @@ Accelerator Types Ray supports the following accelerator types: .. literalinclude:: ../../../python/ray/util/accelerators/accelerators.py - :language: python \ No newline at end of file + :language: python diff --git a/doc/source/ray-core/actors.rst b/doc/source/ray-core/actors.rst index 67474959c5a9..b581921f312d 100644 --- a/doc/source/ray-core/actors.rst +++ b/doc/source/ray-core/actors.rst @@ -104,7 +104,7 @@ Use `ray list actors` from :ref:`State API ` to see acto Stats: ------------------------------ Total: 1 - + Table: ------------------------------ ACTOR_ID CLASS_NAME STATE JOB_ID NAME NODE_ID PID RAY_NAMESPACE @@ -403,8 +403,8 @@ Cancel Actor Tasks by calling :func:`ray.cancel() ` on the returned In Ray, Task cancellation behavior is contingent on the Task's current state: **Unscheduled Tasks**: -If the Actor Task hasn't been scheduled yet, Ray attempts to cancel the scheduling. -When successfully cancelled at this stage, invoking ``ray.get(actor_task_ref)`` +If the Actor Task hasn't been scheduled yet, Ray attempts to cancel the scheduling. +When successfully cancelled at this stage, invoking ``ray.get(actor_task_ref)`` produce a :class:`TaskCancelledError `. **Running Actor Tasks (Regular Actor, Threaded Actor)**: @@ -412,8 +412,8 @@ For tasks classified as a single-threaded Actor or a multi-threaded Actor, Ray offers no mechanism for interruption. **Running Async Actor Tasks**: -For Tasks classified as `async Actors <_async-actors>`, Ray seeks to cancel the associated `asyncio.Task`. -This cancellation approach aligns with the standards presented in +For Tasks classified as `async Actors <_async-actors>`, Ray seeks to cancel the associated `asyncio.Task`. +This cancellation approach aligns with the standards presented in `asyncio task cancellation `__. Note that `asyncio.Task` won't be interrupted in the middle of execution if you don't `await` within the async function. @@ -424,7 +424,7 @@ the Task might not be cancelled. You can check if a Task was successfully cancelled using ``ray.get(actor_task_ref)``. **Recursive Cancellation**: -Ray tracks all child and Actor Tasks. When the``recursive=True`` argument is given, +Ray tracks all child and Actor Tasks. When the ``recursive=True`` argument is given, it cancels all child and Actor Tasks. Scheduling @@ -466,7 +466,7 @@ define can run. This also implies that tasks are scheduled more flexibly, and that if you don't need the stateful part of an actor, you're mostly better off using tasks. -Task Events +Task Events ----------- By default, Ray traces the execution of actor tasks, reporting task status events and profiling events diff --git a/doc/source/ray-core/actors/terminating-actors.rst b/doc/source/ray-core/actors/terminating-actors.rst index 1500c3664125..3ac8dc22eefb 100644 --- a/doc/source/ray-core/actors/terminating-actors.rst +++ b/doc/source/ray-core/actors/terminating-actors.rst @@ -87,7 +87,7 @@ Use `ray list actors --detail` from :ref:`State API ` to serialized_runtime_env: '{}' required_resources: {} death_cause: - actor_died_error_context: # <---- You could see the error message w.r.t why the actor exits. + actor_died_error_context: # <---- You could see the error message w.r.t why the actor exits. error_message: The actor is dead because `ray.kill` killed it. owner_id: 01000000ffffffffffffffffffffffffffffffffffffffffffffffff owner_ip_address: 127.0.0.1 @@ -153,7 +153,7 @@ Note that this method of termination waits until any previously submitted tasks finish executing and then exits the process gracefully with sys.exit. - + You could see the actor is dead as a result of the user's `exit_actor()` call: .. code-block:: bash @@ -190,4 +190,4 @@ You could see the actor is dead as a result of the user's `exit_actor()` call: never_started: false is_detached: false placement_group_id: null - repr_name: '' \ No newline at end of file + repr_name: '' diff --git a/doc/source/ray-core/api/cli.rst b/doc/source/ray-core/api/cli.rst index 7f5ee4c3ef57..85c764a9e0e2 100644 --- a/doc/source/ray-core/api/cli.rst +++ b/doc/source/ray-core/api/cli.rst @@ -50,4 +50,4 @@ This section contains commands to enable/disable :ref:`Ray usage stats -
Data
Data
Compute (hyperparamter configurations)
Compute (h...
Fit & evaluate models
Fit & eval...
Get metrics back
Get metric...
Dataset
DatasetText is not SVG - cannot display \ No newline at end of file +
Data
Data
Compute (hyperparamter configurations)
Compute (h...
Fit & evaluate models
Fit & eval...
Get metrics back
Get metric...
Dataset
DatasetText is not SVG - cannot display diff --git a/doc/source/ray-core/examples/images/batch-training.svg b/doc/source/ray-core/examples/images/batch-training.svg index 83fb2b36efe1..c6d04d309463 100644 --- a/doc/source/ray-core/examples/images/batch-training.svg +++ b/doc/source/ray-core/examples/images/batch-training.svg @@ -1,4 +1,4 @@ -
Data
Data
Compute
ComputeText is not SVG - cannot display \ No newline at end of file +
Data
Data
Compute
ComputeText is not SVG - cannot display diff --git a/doc/source/ray-core/examples/lm/preprocess.sh b/doc/source/ray-core/examples/lm/preprocess.sh index 46efce24ce27..df69a257cc8d 100644 --- a/doc/source/ray-core/examples/lm/preprocess.sh +++ b/doc/source/ray-core/examples/lm/preprocess.sh @@ -30,4 +30,4 @@ fairseq-preprocess \ --validpref wikitext-103-raw/wiki.valid.bpe \ --testpref wikitext-103-raw/wiki.test.bpe \ --destdir data-bin/wikitext-103 \ - --workers 60 \ No newline at end of file + --workers 60 diff --git a/doc/source/ray-core/examples/overview.rst b/doc/source/ray-core/examples/overview.rst index 290f522dc866..5803eaa39a1d 100644 --- a/doc/source/ray-core/examples/overview.rst +++ b/doc/source/ray-core/examples/overview.rst @@ -22,7 +22,7 @@ Beginner * - :doc:`A Gentle Introduction to Ray Core by Example ` * - :doc:`Using Ray for Highly Parallelizable Tasks ` * - :doc:`Monte Carlo Estimation of π ` - + Intermediate ------------ @@ -37,7 +37,7 @@ Advanced -------- .. list-table:: - + * - :doc:`Build Simple AutoML for Time Series Using Ray ` * - :doc:`Build Batch Prediction Using Ray ` * - :doc:`Build a Simple Parameter Server Using Ray ` diff --git a/doc/source/ray-core/examples/plot_hyperparameter.ipynb b/doc/source/ray-core/examples/plot_hyperparameter.ipynb index f84765f6df57..645980c88d31 100644 --- a/doc/source/ray-core/examples/plot_hyperparameter.ipynb +++ b/doc/source/ray-core/examples/plot_hyperparameter.ipynb @@ -314,4 +314,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} \ No newline at end of file +} diff --git a/doc/source/ray-core/fault_tolerance/actors.rst b/doc/source/ray-core/fault_tolerance/actors.rst index c15431642cf9..6f46a6d67412 100644 --- a/doc/source/ray-core/fault_tolerance/actors.rst +++ b/doc/source/ray-core/fault_tolerance/actors.rst @@ -183,4 +183,3 @@ Retry behavior depends on the value you set ``retry_exceptions`` to: - The default value,`0`. For example, if a method sets `max_task_retries=5` and `retry_exceptions=True`, and the actor sets `max_restarts=2`, Ray executes the method up to 6 times: once for the initial invocation, and 5 additional retries. The 6 invocations may include 2 actor crashes. After the 6th invocation, a `ray.get` call to the result Ray ObjectRef raises the exception raised in the last invocation, or `ray.exceptions.RayActorError` if the actor crashed in the last invocation. - diff --git a/doc/source/ray-core/fault_tolerance/tasks.rst b/doc/source/ray-core/fault_tolerance/tasks.rst index 2fa9720fac1b..3fcd6d9ea2d2 100644 --- a/doc/source/ray-core/fault_tolerance/tasks.rst +++ b/doc/source/ray-core/fault_tolerance/tasks.rst @@ -53,7 +53,7 @@ Use `ray list tasks` from :ref:`State API CLI ` to query .. code-block:: bash # This API is only available when you download Ray via `pip install "ray[default]"` - ray list tasks + ray list tasks .. code-block:: bash diff --git a/doc/source/ray-core/images/memory.svg b/doc/source/ray-core/images/memory.svg index 593b5330a803..09974db2c149 100644 --- a/doc/source/ray-core/images/memory.svg +++ b/doc/source/ray-core/images/memory.svg @@ -1 +1 @@ - \ No newline at end of file + diff --git a/doc/source/ray-core/images/oom_killer_example.svg b/doc/source/ray-core/images/oom_killer_example.svg index b1977b023522..5859d7745c99 100644 --- a/doc/source/ray-core/images/oom_killer_example.svg +++ b/doc/source/ray-core/images/oom_killer_example.svg @@ -1 +1 @@ - \ No newline at end of file + diff --git a/doc/source/ray-core/images/oom_killer_example_killed_one.svg b/doc/source/ray-core/images/oom_killer_example_killed_one.svg index 31e5f448674e..adf8217bf54a 100644 --- a/doc/source/ray-core/images/oom_killer_example_killed_one.svg +++ b/doc/source/ray-core/images/oom_killer_example_killed_one.svg @@ -1 +1 @@ - \ No newline at end of file + diff --git a/doc/source/ray-core/images/oom_killer_example_killed_two.svg b/doc/source/ray-core/images/oom_killer_example_killed_two.svg index 1ae1d01e3cdd..d607ac09e823 100644 --- a/doc/source/ray-core/images/oom_killer_example_killed_two.svg +++ b/doc/source/ray-core/images/oom_killer_example_killed_two.svg @@ -1 +1 @@ - \ No newline at end of file + diff --git a/doc/source/ray-core/images/physical_resources_vs_logical_resources.svg b/doc/source/ray-core/images/physical_resources_vs_logical_resources.svg index 640db36c8c2f..8524bfdb8110 100644 --- a/doc/source/ray-core/images/physical_resources_vs_logical_resources.svg +++ b/doc/source/ray-core/images/physical_resources_vs_logical_resources.svg @@ -1 +1 @@ - \ No newline at end of file + diff --git a/doc/source/ray-core/images/pong-arch.svg b/doc/source/ray-core/images/pong-arch.svg index f51a201ad23a..0955a27fdd87 100644 --- a/doc/source/ray-core/images/pong-arch.svg +++ b/doc/source/ray-core/images/pong-arch.svg @@ -1 +1 @@ - \ No newline at end of file + diff --git a/doc/source/ray-core/objects/serialization.rst b/doc/source/ray-core/objects/serialization.rst index 5f1ec2162c66..b41c14f4b43f 100644 --- a/doc/source/ray-core/objects/serialization.rst +++ b/doc/source/ray-core/objects/serialization.rst @@ -33,7 +33,7 @@ Ray `ObjectRefs` can be serialized using `ray.cloudpickle`. The `ObjectRef` can When serialized, the `ObjectRef`'s value will remain pinned in Ray's shared memory object store. The object must be explicitly freed by calling `ray._private.internal_api.free(obj_ref)`. .. warning:: - + `ray._private.internal_api.free(obj_ref)` is a private API and may be changed in future Ray versions. This code example demonstrates how to serialize an `ObjectRef`, store it in external storage, deserialize and use it, and lastly free its object. diff --git a/doc/source/ray-core/patterns/concurrent-operations-async-actor.rst b/doc/source/ray-core/patterns/concurrent-operations-async-actor.rst index 0c1162f07301..656c3215243d 100644 --- a/doc/source/ray-core/patterns/concurrent-operations-async-actor.rst +++ b/doc/source/ray-core/patterns/concurrent-operations-async-actor.rst @@ -31,4 +31,3 @@ We can solve this problem by using :ref:`async actors ` and use `` :end-before: __async_actor_end__ Here, instead of using the blocking :func:`ray.get() ` to get the value of an ObjectRef, we use ``await`` so it can yield the control while we are waiting for the object to be fetched. - diff --git a/doc/source/ray-core/scheduling/placement-group.rst b/doc/source/ray-core/scheduling/placement-group.rst index 339c23bb4875..cc7b9c7b49da 100644 --- a/doc/source/ray-core/scheduling/placement-group.rst +++ b/doc/source/ray-core/scheduling/placement-group.rst @@ -578,9 +578,9 @@ child tasks and actors to the same placement group, specify ``PlacementGroupSche :end-before: __child_capture_disable_pg_end__ .. warning:: - The value of ``placement_group_capture_child_tasks`` for a given actor isn't inherited from its parent. If you're creating nested actors of depth greater than 1 + The value of ``placement_group_capture_child_tasks`` for a given actor isn't inherited from its parent. If you're creating nested actors of depth greater than 1 and should all use the same placement group, you should explicitly set ``placement_group_capture_child_tasks`` explicitly set for each actor. - + [Advanced] Named Placement Group -------------------------------- diff --git a/doc/source/ray-core/scheduling/ray-oom-prevention.rst b/doc/source/ray-core/scheduling/ray-oom-prevention.rst index b64f8c7892a0..faddad1393c5 100644 --- a/doc/source/ray-core/scheduling/ray-oom-prevention.rst +++ b/doc/source/ray-core/scheduling/ray-oom-prevention.rst @@ -25,7 +25,7 @@ It's available on Linux and is tested with Ray running inside a container that i How do I disable the memory monitor? -------------------------------------- -The memory monitor is enabled by default and can be disabled by setting the environment variable ``RAY_memory_monitor_refresh_ms`` to zero when Ray starts (e.g., RAY_memory_monitor_refresh_ms=0 ray start ...). +The memory monitor is enabled by default and can be disabled by setting the environment variable ``RAY_memory_monitor_refresh_ms`` to zero when Ray starts (e.g., RAY_memory_monitor_refresh_ms=0 ray start ...). How do I configure the memory monitor? -------------------------------------- @@ -97,7 +97,7 @@ If, at this point, the node still runs out of memory, the process will repeat: RAY_event_stats_print_interval_ms=1000 python oom.py (raylet) node_manager.cc:3040: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 2c82620270df6b9dd7ae2791ef51ee4b5a9d5df9f795986c10dd219c, IP: 172.31.183.172) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.31.183.172` - (raylet) + (raylet) (raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero. task failed with OutOfMemoryError, which is expected Verify the task was indeed executed twice via ``task_oom_retry``: @@ -125,7 +125,7 @@ If, at this point, the node still runs out of memory, the process will repeat: .. code-block:: bash $ python two_actors.py - + First started actor, which is retriable, was killed by the memory monitor. Second started actor, which is not-retriable, finished. diff --git a/doc/source/ray-core/scheduling/resources.rst b/doc/source/ray-core/scheduling/resources.rst index b7c92a9d1c8f..f3a06761c872 100644 --- a/doc/source/ray-core/scheduling/resources.rst +++ b/doc/source/ray-core/scheduling/resources.rst @@ -197,4 +197,4 @@ The precision of the fractional resource requirement is 0.0001 so you should avo .. tip:: Besides resource requirements, you can also specify an environment for a task or actor to run in, - which can include Python packages, local files, environment variables, and more. See :ref:`Runtime Environments ` for details. \ No newline at end of file + which can include Python packages, local files, environment variables, and more. See :ref:`Runtime Environments ` for details. diff --git a/doc/source/ray-core/tasks.rst b/doc/source/ray-core/tasks.rst index fd31daba2c3e..390636240a31 100644 --- a/doc/source/ray-core/tasks.rst +++ b/doc/source/ray-core/tasks.rst @@ -97,8 +97,8 @@ Use `ray summary tasks` from :ref:`State API ` to see r total_actor_scheduled: 0 total_actor_tasks: 0 total_tasks: 5 - - + + Table (group by func_name): ------------------------------------ FUNC_OR_CLASS_NAME STATE_COUNTS TYPE diff --git a/doc/source/ray-core/user-spawn-processes.rst b/doc/source/ray-core/user-spawn-processes.rst index 1b06b7f9f7a0..dc049e0de23b 100644 --- a/doc/source/ray-core/user-spawn-processes.rst +++ b/doc/source/ray-core/user-spawn-processes.rst @@ -102,4 +102,4 @@ When the ``the worker`` dies, ``Raylet`` kills the ``user process A``, because i An edge case is, if the ``the worker`` is still alive but the ``user process A`` is dead, then ``user process B`` gets reparented and risks being killed. To mitigate, ``Ray`` also sets the ``the worker`` as a subreaper, so it can adopt the reparented processes. ``Core worker`` does not kill unknown children processes, so a user "daemon" process e.g. ``user process B`` that outlives ``user process A`` can live along. However if the ``the worker`` dies, the user daemon process gets reparented to ``raylet`` and gets killed. -Related PR: `Use subreaper to kill unowned subprocesses in raylet. (#42992) `_ \ No newline at end of file +Related PR: `Use subreaper to kill unowned subprocesses in raylet. (#42992) `_ diff --git a/doc/source/ray-core/using-ray-with-jupyter.rst b/doc/source/ray-core/using-ray-with-jupyter.rst index fb59aca877d8..b6ff96886551 100644 --- a/doc/source/ray-core/using-ray-with-jupyter.rst +++ b/doc/source/ray-core/using-ray-with-jupyter.rst @@ -1,7 +1,7 @@ Working with Jupyter Notebooks & JupyterLab =========================================== -This document describes best practices for using Ray with Jupyter Notebook / +This document describes best practices for using Ray with Jupyter Notebook / JupyterLab. We use AWS for the purpose of illustration, but the arguments should also apply to other Cloud providers. @@ -10,24 +10,24 @@ Feel free to contribute if you think this document is missing anything. Setting Up Notebook ------------------- -1. Ensure your EC2 instance has enough EBS volume if you plan to run the +1. Ensure your EC2 instance has enough EBS volume if you plan to run the Notebook on it. -The Deep Learning AMI, pre-installed libraries and environmental set-up +The Deep Learning AMI, pre-installed libraries and environmental set-up will by default consume ~76% of the disk prior to any Ray work. With additional applications running, the Notebook could fail frequently -due to full disk. -Kernel restart loses progressing cell outputs, especially if we rely on -them to track experiment progress. -Related issue: `Autoscaler should allow configuration of disk space and +due to full disk. +Kernel restart loses progressing cell outputs, especially if we rely on +them to track experiment progress. +Related issue: `Autoscaler should allow configuration of disk space and should use a larger default. `_. 2. Avoid unnecessary memory usage. IPython stores the output of every cell in a local Python variable indefinitely. This causes Ray to pin the objects even though you application may not actually be using them. -Therefore, explicitly calling ``print`` or ``repr`` is better than letting +Therefore, explicitly calling ``print`` or ``repr`` is better than letting the Notebook automatically generate the output. -Another option is to just altogether disable IPython caching with the +Another option is to just altogether disable IPython caching with the following (run from bash/zsh): .. code-block:: console @@ -39,21 +39,21 @@ following (run from bash/zsh): This will still allow printing, but stop IPython from caching altogether. .. tip:: - While the above settings help reduce memory footprint, it's always a good + While the above settings help reduce memory footprint, it's always a good practice to remove references that are no longer needed in your application to free space in the object store. -3. Understand the node’s responsibility. +3. Understand the node’s responsibility. Assuming the Notebook runs on a EC2 instance, do you plan to start a ray runtime locally on this instance, -or do you plan to use this instance as a cluster launcher? -Jupyter Notebook is more suitable for the first scenario. +or do you plan to use this instance as a cluster launcher? +Jupyter Notebook is more suitable for the first scenario. CLI’s such as ``ray exec`` and ``ray submit`` fit the second use case better. 4. Forward the ports. Assuming the Notebook runs on an EC2 instance, you should forward both the Notebook port and the Ray Dashboard port. -The default ports are 8888 and 8265 respectively. +The default ports are 8888 and 8265 respectively. They will increase if the default ones are not available. You can forward them with the following (run from bash/zsh):