Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Doc] Restructure core API docs #32236

Merged
merged 59 commits into from
Feb 14, 2023
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
749c8f8
Restructure core API docs
jjyao Feb 6, 2023
9cdfc3e
up
jjyao Feb 6, 2023
8e31911
up
jjyao Feb 6, 2023
9ad9ada
up
jjyao Feb 6, 2023
b81b9aa
up
jjyao Feb 6, 2023
55ac49e
up
jjyao Feb 6, 2023
175a824
up
jjyao Feb 6, 2023
b34a733
up
jjyao Feb 7, 2023
9286053
up
jjyao Feb 7, 2023
f84e755
up
jjyao Feb 7, 2023
d97a2d3
up
jjyao Feb 7, 2023
fe9d1db
up
jjyao Feb 7, 2023
31903e3
up
jjyao Feb 7, 2023
bf86626
up
jjyao Feb 7, 2023
298de25
up
jjyao Feb 7, 2023
aaa5f00
up
jjyao Feb 7, 2023
0771fcf
up
jjyao Feb 7, 2023
f8cbdc8
up
jjyao Feb 7, 2023
7aa8d6f
up
jjyao Feb 8, 2023
9239df9
up
jjyao Feb 8, 2023
96ae8cf
Revert "up"
jjyao Feb 8, 2023
d0d5424
test
jjyao Feb 8, 2023
d0a9562
up
jjyao Feb 8, 2023
741332c
lint
jjyao Feb 8, 2023
6908a2c
fix
jjyao Feb 8, 2023
7e6a480
lint
jjyao Feb 8, 2023
2f910ec
up
jjyao Feb 9, 2023
f306a66
test
jjyao Feb 9, 2023
19e8ad7
Revert "test"
jjyao Feb 9, 2023
2979edb
fix
jjyao Feb 9, 2023
27e8ad1
Merge branch 'master' of github.com:ray-project/ray into jjyao/api
jjyao Feb 9, 2023
c9b66e4
warn
jjyao Feb 10, 2023
203241f
try
jjyao Feb 10, 2023
db0261d
comments
jjyao Feb 10, 2023
beb1bf2
up
jjyao Feb 10, 2023
e241183
up
jjyao Feb 10, 2023
019fc85
try
jjyao Feb 10, 2023
a24f3ba
up
jjyao Feb 10, 2023
7f1b7ae
Revert "try"
jjyao Feb 10, 2023
b7b6b5a
try
jjyao Feb 10, 2023
d915e3b
try
jjyao Feb 12, 2023
16b0d82
Merge branch 'master' of github.com:ray-project/ray into jjyao/api
jjyao Feb 12, 2023
fe9f44d
fix
jjyao Feb 13, 2023
2387859
fix
jjyao Feb 13, 2023
a6d319b
reduce time
jjyao Feb 13, 2023
87d6710
fix
jjyao Feb 13, 2023
1272e72
fix
jjyao Feb 13, 2023
11333ba
reduce doc build time
jjyao Feb 13, 2023
bd2878e
Merge branch 'master' of github.com:ray-project/ray into jjyao/api
jjyao Feb 13, 2023
b80c383
try
jjyao Feb 13, 2023
2b78440
Revert "try"
jjyao Feb 14, 2023
ba4ca5e
up
jjyao Feb 14, 2023
a855a0a
up
jjyao Feb 14, 2023
166cdf7
Merge branch 'master' of github.com:ray-project/ray into jjyao/api
jjyao Feb 14, 2023
30cf81c
fix
jjyao Feb 14, 2023
8a29670
split
jjyao Feb 14, 2023
bba82cf
up
jjyao Feb 14, 2023
7a95a67
split
jjyao Feb 14, 2023
575a8dc
func
jjyao Feb 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions doc/source/_templates/autosummary/class.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{{ fullname | escape | underline}}

.. currentmodule:: {{ module }}

.. autoclass:: {{ objname }}

{% block methods %}
{% if methods %}
.. rubric:: {{ _('Methods') }}

.. autosummary::
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will generate stub page for each class method and attribute which will make the doc build even longer but that's what other libraries (e.g. pandas, numpy) are doing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @maxpumperla FYI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jjyao is there a compromise we could find here? this seems to more than double the build time now. I'm in touch with support from readthedocs.org right now to make longer builds happen, but for now we can't pull this off.

just asking if we can think of a trade-off and move towards a more extensive solution later?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can try to not autosummary class members

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, let's try this for now and see how it impacts build times. this PR increases rtd builds from 900s to 2000+, which will always time out

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems that doesn't work: https://buildkite.com/ray-project/oss-ci-build-pr/builds/11774#01863776-7701-40d6-974c-b6fe51e9d705 since we have places that manually autosummary class members which creates duplications with autoclass.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, so be it. we just have to make sure we stay below 3000s total, once all other libs merge their changes. might be risky.

:toctree:

{% for item in methods %}
{{ name }}.{{ item }}
{%- endfor %}

{% endif %}
{% endblock %}

{% block attributes %}
{% if attributes %}
.. rubric:: {{ _('Attributes') }}

.. autosummary::
:toctree:

{% for item in attributes %}
{{ name }}.{{ item }}
{%- endfor %}

{% endif %}
{% endblock %}
2 changes: 1 addition & 1 deletion doc/source/_toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ parts:
- file: ray-core/examples/batch_training
- file: ray-core/examples/automl_for_time_series
- file: ray-core/examples/web-crawler
- file: ray-core/api
- file: ray-core/reference/index

- file: cluster/getting-started
title: "Ray Clusters"
Expand Down
2 changes: 1 addition & 1 deletion doc/source/ray-core/actors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ By default, Ray actors won't be :ref:`restarted <fault-tolerance-actors>` and
actor tasks won't be retried when actors crash unexpectedly.
You can change this behavior by setting
``max_restarts`` and ``max_task_retries`` options
in :ref:`ray.remote() <ray-remote-ref>` and :ref:`.options() <ray-options-ref>`.
in :meth:`ray.remote() <ray.remote>` and :meth:`.options() <ray.actor.ActorClass.options>`.
See :ref:`Ray fault tolerance <fault-tolerance>` for more details.

FAQ: Actors, Workers and Resources
Expand Down
4 changes: 2 additions & 2 deletions doc/source/ray-core/actors/actor-utils.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Actor Pool

.. literalinclude:: ../doc_code/actor-pool.py

See the :ref:`package reference <ray-actor-pool-ref>` for more information.
See the :class:`package reference <ray.util.ActorPool>` for more information.

.. tabbed:: Java

Expand All @@ -25,7 +25,7 @@ Message passing using Ray Queue
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sometimes just using one signal to synchronize is not enough. If you need to send data among many tasks or
actors, you can use :ref:`ray.util.queue.Queue <ray-queue-ref>`.
actors, you can use :class:`ray.util.queue.Queue <ray.util.queue.Queue>`.

.. literalinclude:: ../doc_code/actor-queue.py

Expand Down
11 changes: 0 additions & 11 deletions doc/source/ray-core/api.rst

This file was deleted.

10 changes: 5 additions & 5 deletions doc/source/ray-core/configure.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,12 @@ If using the command line, connect to the Ray cluster as follow:
.. _omp-num-thread-note:

.. note::
Ray sets the environment variable ``OMP_NUM_THREADS=<num_cpus>`` if ``num_cpus`` is set on
the task/actor via :ref:`ray.remote() <ray-remote-ref>` and :ref:`.options() <ray-options-ref>`.
Ray sets ``OMP_NUM_THREADS=1`` if ``num_cpus`` is not specified; this
Ray sets the environment variable ``OMP_NUM_THREADS=<num_cpus>`` if ``num_cpus`` is set on
the task/actor via :meth:`ray.remote() <ray.remote>` and :meth:`task.options() <ray.remote_function.RemoteFunction.options>`/:meth:`actor.options() <ray.actor.ActorClass.options>`.
Ray sets ``OMP_NUM_THREADS=1`` if ``num_cpus`` is not specified; this
is done to avoid performance degradation with many workers (issue #6998). You can
also override this by explicitly setting ``OMP_NUM_THREADS`` to override anything Ray sets by default.
``OMP_NUM_THREADS`` is commonly used in numpy, PyTorch, and Tensorflow to perform multi-threaded
also override this by explicitly setting ``OMP_NUM_THREADS`` to override anything Ray sets by default.
``OMP_NUM_THREADS`` is commonly used in numpy, PyTorch, and Tensorflow to perform multi-threaded
linear algebra. In multi-worker setting, we want one thread per worker instead of many threads
per worker to avoid contention. Some other libraries may have their own way to configure
parallelism. For example, if you're using OpenCV, you should manually set the number of
Expand Down
10 changes: 5 additions & 5 deletions doc/source/ray-core/examples/monte_carlo_pi.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ To get started, install Ray via ``pip install -U ray``. See :ref:`Installing Ray

Starting Ray
------------
First, let's include all modules needed for this tutorial and start a local Ray cluster with :ref:`ray.init() <ray-init-ref>`:
First, let's include all modules needed for this tutorial and start a local Ray cluster with :meth:`ray.init() <ray.init>`:

.. literalinclude:: ../doc_code/monte_carlo_pi.py
:language: python
Expand All @@ -41,7 +41,7 @@ Ray actors are essentially stateful services that anyone with an instance (a han
:start-after: __defining_actor_start__
:end-before: __defining_actor_end__

We define a Ray actor by decorating a normal Python class with :ref:`ray.remote <ray-remote-ref>`.
We define a Ray actor by decorating a normal Python class with :meth:`ray.remote <ray.remote>`.
The progress actor has ``report_progress()`` method that will be called by sampling tasks to update their progress individually
and ``get_progress()`` method to get the overall progress.

Expand All @@ -55,7 +55,7 @@ Ray tasks are stateless functions. They execute asynchronously, and run in paral
:start-after: __defining_task_start__
:end-before: __defining_task_end__

To convert a normal Python function as a Ray task, we decorate the function with :ref:`ray.remote <ray-remote-ref>`.
To convert a normal Python function as a Ray task, we decorate the function with :meth:`ray.remote <ray.remote>`.
The sampling task takes a progress actor handle as an input and reports progress to it.
The above code shows an example of calling actor methods from tasks.

Expand Down Expand Up @@ -97,7 +97,7 @@ While sampling tasks are running, we can periodically query the progress by call
To call an actor method, use ``actor_handle.method.remote()``.
This invocation immediately returns an ``ObjectRef`` as a future
and then executes the method asynchronously on the remote actor process.
To fetch the actual returned value of ``ObjectRef``, we use the blocking :ref:`ray.get() <ray-get-ref>`.
To fetch the actual returned value of ``ObjectRef``, we use the blocking :meth:`ray.get() <ray.get>`.

Calculating π
-------------
Expand All @@ -108,7 +108,7 @@ Finally, we get number of samples inside the circle from the remote sampling tas
:start-after: __calculating_pi_start__
:end-before: __calculating_pi_end__

As we can see from the above code, besides a single ``ObjectRef``, :ref:`ray.get() <ray-get-ref>` can also take a list of ``ObjectRef`` and return a list of results.
As we can see from the above code, besides a single ``ObjectRef``, :meth:`ray.get() <ray.get>` can also take a list of ``ObjectRef`` and return a list of results.

If you run this tutorial, you will see output like:

Expand Down
2 changes: 1 addition & 1 deletion doc/source/ray-core/fault_tolerance/actors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ After the specified number of restarts, subsequent actor methods will
raise a ``RayActorError``.

By default, actor tasks execute with at-most-once semantics
(``max_task_retries=0`` in the ``@ray.remote`` :ref:`decorator <ray-remote-ref>`). This means that if an
(``max_task_retries=0`` in the ``@ray.remote`` :meth:`decorator <ray.remote>`). This means that if an
actor task is submitted to an actor that is unreachable, Ray will report the
error with ``RayActorError``, a Python-level exception that is thrown when
``ray.get`` is called on the future returned by the task. Note that this
Expand Down
2 changes: 1 addition & 1 deletion doc/source/ray-core/fault_tolerance/tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ If a task is hanging, you may want to cancel the task to continue to make
progress. You can do this by calling ``ray.cancel`` on an ``ObjectRef``
returned by the task. By default, this will send a KeyboardInterrupt to the
task's worker if it is mid-execution. Passing ``force=True`` to ``ray.cancel``
will force-exit the worker. See :ref:`the API reference <ray-cancel-ref>` for
will force-exit the worker. See :meth:`the API reference <ray.cancel>` for
``ray.cancel`` for more details.

Note that currently, Ray will not automatically retry tasks that have been
Expand Down
4 changes: 2 additions & 2 deletions doc/source/ray-core/objects.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ similar.
Object refs can be created in two ways.

1. They are returned by remote function calls.
2. They are returned by ``put`` (:ref:`docstring <ray-put-ref>`).
2. They are returned by :meth:`ray.put() <ray.put>`.

.. tabbed:: Python

Expand Down Expand Up @@ -48,7 +48,7 @@ Object refs can be created in two ways.
Fetching Object Data
--------------------

You can use the ``get`` method (:ref:`docstring <ray-get-ref>`) to fetch the result of a remote object from an object ref.
You can use the :meth:`ray.get() <ray.get>` method to fetch the result of a remote object from an object ref.
If the current node's object store does not contain the object, the object is downloaded.

.. tabbed:: Python
Expand Down
Loading