[RLlib; docs] RLlib documentation do-over (new API stack): Main index page. #48285

sven1977 · 2024-10-27T19:24:59Z

Update, refactor, fix the main RLlib index.html page (for the new API stack).

Fully geared towards new API stack.
Simplified (only mention a few high-value features).
Better overview tables within the tabs for algos, environments, and features.
Redo RLlib overview diagram at bottom of page (also simplified).

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <svenmika1977@gmail.com>

simonsays1980

LGTM.

simonsays1980 · 2024-10-27T19:32:03Z

rllib/examples/inference/policy_inference_after_training.py

@@ -147,7 +150,7 @@
        )
    )

-    # Create the env to do inference in.
+    # Create a env to do inference in.


We should bring in here loading the pipelines from checkpoint, too and using them.

We have another example, where we do that (the LSTM one, which requires the connector pipeline for a more sophisticated inference loop w/ state in/outs).

Signed-off-by: sven1977 <svenmika1977@gmail.com>

rllib/examples/inference/policy_inference_after_training.py

Signed-off-by: Sven Mika <sven@anyscale.io>

angelinalg

Some style nits. Please correct if the rewrites are inaccurate, esp the ones changing passive voice to active voice.

angelinalg · 2024-10-28T16:50:24Z

rllib/examples/inference/policy_inference_after_training.py

@@ -147,7 +150,7 @@
        )
    )

-    # Create the env to do inference in.
+    # Create a env to do inference in.


Suggested change

# Create a env to do inference in.

# Create an env to do inference in.

angelinalg · 2024-10-28T17:01:37Z

doc/source/rllib/index.rst

+RLlib is used in production by industry leaders in many different verticals, such as
+`gaming <https://www.anyscale.com/events/2021/06/22/using-reinforcement-learning-to-optimize-iap-offer-recommendations-in-mobile-games>`_,
+`robotics <https://www.anyscale.com/events/2021/06/23/introducing-amazon-sagemaker-kubeflow-reinforcement-learning-pipelines-for>`_,
+`finance <https://www.anyscale.com/events/2021/06/22/a-24x-speedup-for-reinforcement-learning-with-rllib-+-ray>`_,
 `climate control <https://www.anyscale.com/events/2021/06/23/applying-ray-and-rllib-to-real-life-industrial-use-cases>`_,


climate control and industrial control links point to the same link. Is that intentional?

fixed by merging them ...

angelinalg · 2024-10-28T17:14:00Z

doc/source/rllib/index.rst

-    <div class="termynal" data-termynal>
-        <span data-ty="input">pip install "ray[rllib]" tensorflow torch</span>
-    </div>
+    For installation on computers running Apple Silicon (such as M1),


Suggested change

For installation on computers running Apple Silicon (such as M1),

For installation on computers running Apple Silicon such as M1,

angelinalg · 2024-10-28T17:15:23Z

doc/source/rllib/index.rst

-        <span data-ty="input">pip install "ray[rllib]" tensorflow torch</span>
-    </div>
+    For installation on computers running Apple Silicon (such as M1),
+    `follow instructions here. <https://docs.ray.io/en/latest/ray-overview/installation.html#m1-mac-apple-silicon-support>`_


Suggested change

`follow instructions here. <https://docs.ray.io/en/latest/ray-overview/installation.html#m1-mac-apple-silicon-support>`_

see `M1 Mac Support. <https://docs.ray.io/en/latest/ray-overview/installation.html#m1-mac-apple-silicon-support>`_

angelinalg · 2024-10-28T17:16:10Z

doc/source/rllib/index.rst

-    `here. <https://docs.ray.io/en/latest/ray-overview/installation.html#m1-mac-apple-silicon-support>`_
-    To be able to run our Atari examples, you should also install
-    `pip install "gym[atari]" "gym[accept-rom-license]" atari_py`.
+    To be able to run our Atari or MuJoCo examples, you also need to run:


Suggested change

To be able to run our Atari or MuJoCo examples, you also need to run:

To run the Atari or MuJoCo examples, you also need to run:

doc/source/rllib/rllib-examples.rst

doc/source/rllib/rllib-algorithms.rst

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

…' into documentation_do_over_index_page # Conflicts: # doc/source/rllib/rllib-algorithms.rst

doc/source/rllib/rllib-algorithms.rst

Signed-off-by: Sven Mika <sven@anyscale.io>

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…' into documentation_do_over_index_page

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

angelinalg

Thanks for addressing the comments. I must've missed the "off-policy'ness" in the first review. It'll be good to fix that if you can.

doc/source/rllib/rllib-algorithms.rst

angelinalg · 2024-10-29T19:40:43Z

doc/source/rllib/rllib-algorithms.rst

@@ -217,13 +217,13 @@ Asynchronous Proximal Policy Optimization (APPO)
    **APPO architecture:** APPO is an asynchronous variant of :ref:`Proximal Policy Optimization (PPO) <ppo>` based on the IMPALA architecture,
    but using a surrogate policy loss with clipping, allowing for multiple SGD passes per collected train batch.
    In a training iteration, APPO requests samples from all EnvRunners asynchronously and the collected episode


"RLlib" was my guess and the point was just to clarify who's doing the returning, if needed. If it's obvious to the reader, just ignore my suggestion.

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

… page. (ray-project#48285)

… page. (ray-project#48285) Signed-off-by: JP-sDEV <jon.pablo80@gmail.com>

… page. (ray-project#48285) Signed-off-by: mohitjain2504 <mohit.jain@dream11.com>

sven1977 added 5 commits June 13, 2024 09:52

wip

eb4df7d

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

2eccb2c

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

69c69c0

Signed-off-by: sven1977 <svenmika1977@gmail.com>

merge

1bc41cf

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

b9f4427

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 requested review from maxpumperla, simonsays1980 and a team as code owners October 27, 2024 19:25

sven1977 assigned simonsays1980, angelinalg and peytondmurray Oct 27, 2024

simonsays1980 approved these changes Oct 27, 2024

View reviewed changes

peytondmurray removed their assignment Oct 27, 2024

sven1977 added 4 commits October 27, 2024 21:13

wip

1b6c4da

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

5971c33

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

8bc1534

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

37581bb

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 commented Oct 28, 2024

View reviewed changes

rllib/examples/inference/policy_inference_after_training.py Outdated Show resolved Hide resolved

Apply suggestions from code review

08a4910

Signed-off-by: Sven Mika <sven@anyscale.io>

angelinalg reviewed Oct 28, 2024

View reviewed changes

sven1977 commented Oct 29, 2024

View reviewed changes

doc/source/rllib/rllib-algorithms.rst Outdated Show resolved Hide resolved

sven1977 and others added 3 commits October 29, 2024 10:09

Apply suggestions from code review

f632af9

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

wip

e343986

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Apply suggestions from code review

1b248d2

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

sven1977 added 2 commits October 29, 2024 10:43

Merge remote-tracking branch 'origin/documentation_do_over_index_page…

9b7fd5c

…' into documentation_do_over_index_page # Conflicts: # doc/source/rllib/rllib-algorithms.rst

Merge remote-tracking branch 'origin/documentation_do_over_index_page…

60dbd7b

…' into documentation_do_over_index_page # Conflicts: # doc/source/rllib/rllib-algorithms.rst

sven1977 commented Oct 29, 2024

View reviewed changes

doc/source/rllib/rllib-algorithms.rst Outdated Show resolved Hide resolved

sven1977 and others added 4 commits October 29, 2024 12:12

Apply suggestions from code review

e5b9384

Signed-off-by: Sven Mika <sven@anyscale.io>

wip

5eaa584

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge remote-tracking branch 'origin/documentation_do_over_index_page…

560d02d

…' into documentation_do_over_index_page

Apply suggestions from code review

4809f0a

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

angelinalg approved these changes Oct 29, 2024

View reviewed changes

sven1977 enabled auto-merge (squash) October 29, 2024 19:48

github-actions bot added the go add ONLY when ready to merge, run all tests label Oct 29, 2024

Apply suggestions from code review

88fbcd7

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

github-actions bot disabled auto-merge October 29, 2024 19:51

sven1977 enabled auto-merge (squash) October 29, 2024 20:18

sven1977 merged commit 7f52a4e into ray-project:master Oct 29, 2024
5 of 6 checks passed

sven1977 deleted the documentation_do_over_index_page branch October 30, 2024 14:21

Jay-ju pushed a commit to Jay-ju/ray that referenced this pull request Nov 5, 2024

[RLlib; docs] RLlib documentation do-over (new API stack): Main index…

9850944

… page. (ray-project#48285)

JP-sDEV pushed a commit to JP-sDEV/ray that referenced this pull request Nov 14, 2024

[RLlib; docs] RLlib documentation do-over (new API stack): Main index…

b040525

… page. (ray-project#48285) Signed-off-by: JP-sDEV <jon.pablo80@gmail.com>

mohitjain2504 pushed a commit to mohitjain2504/ray that referenced this pull request Nov 15, 2024

[RLlib; docs] RLlib documentation do-over (new API stack): Main index…

71513de

… page. (ray-project#48285) Signed-off-by: mohitjain2504 <mohit.jain@dream11.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib; docs] RLlib documentation do-over (new API stack): Main index page. #48285

[RLlib; docs] RLlib documentation do-over (new API stack): Main index page. #48285

sven1977 commented Oct 27, 2024 •

edited

Loading

simonsays1980 left a comment

simonsays1980 Oct 27, 2024

sven1977 Oct 28, 2024

angelinalg left a comment

angelinalg Oct 28, 2024

angelinalg Oct 28, 2024

sven1977 Oct 29, 2024

angelinalg Oct 28, 2024

angelinalg Oct 28, 2024

angelinalg Oct 28, 2024

angelinalg left a comment

angelinalg Oct 29, 2024

	# Create a env to do inference in.
	# Create an env to do inference in.

	For installation on computers running Apple Silicon (such as M1),
	For installation on computers running Apple Silicon such as M1,

	`follow instructions here. <https://docs.ray.io/en/latest/ray-overview/installation.html#m1-mac-apple-silicon-support>`_
	see `M1 Mac Support. <https://docs.ray.io/en/latest/ray-overview/installation.html#m1-mac-apple-silicon-support>`_

	To be able to run our Atari or MuJoCo examples, you also need to run:
	To run the Atari or MuJoCo examples, you also need to run:

[RLlib; docs] RLlib documentation do-over (new API stack): Main index page. #48285

[RLlib; docs] RLlib documentation do-over (new API stack): Main index page. #48285

Conversation

sven1977 commented Oct 27, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

simonsays1980 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angelinalg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angelinalg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sven1977 commented Oct 27, 2024 •

edited

Loading