[AIR][Train][Doc] Restructure API reference #32360

justinvyu · 2023-02-09T10:49:55Z

Why are these changes needed?

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class.

This PR is a followup to #31204 and #32311, which made the same changes for Ray Data/Tune docs.

Depends on [Tune][Doc] Restructure API reference #32311 being merged.

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

maxpumperla

great stuff!

c21

Looks good by spot-checking several doc page. cc @matthewdeng for approval.

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

matthewdeng

Very nice!

matthewdeng · 2023-02-10T18:55:53Z

doc/source/ray-air/api/api.rst

+.. toctree::
+    :maxdepth: 2


Love this structure so much!

matthewdeng · 2023-02-10T18:57:15Z

doc/source/ray-air/api/api.rst

+    Dataset Ingest (Ray Data)<dataset-ingest.rst>
+    Trainers (Ray Train)<../../train/api/api>
+    Tuner (Ray Tune)<../../tune/api_docs/execution>
+    Results (Ray Train + Ray Tune)<../../tune/api_docs/result_grid>


This doesn't seem to be rendering properly on the base page or table of contents. I think it might be that the result_grid page is missing a title header or something?

This will be fixed by merging in Tune PR.

matthewdeng · 2023-02-10T19:09:58Z

doc/source/train/api/api.rst

+Ray Train Built-in Predictors
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    ~xgboost.XGBoostPredictor
+    ~lightgbm.LightGBMPredictor
+    ~tensorflow.TensorflowPredictor
+    ~torch.TorchPredictor
+    ~huggingface.HuggingFacePredictor
+    ~sklearn.SklearnPredictor
+    ~rl.RLPredictor
+
+Ray Train Framework-specific Checkpoints
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    ~xgboost.XGBoostCheckpoint
+    ~lightgbm.LightGBMCheckpoint
+    ~tensorflow.TensorflowCheckpoint
+    ~torch.TorchCheckpoint
+    ~huggingface.HuggingFaceCheckpoint
+    ~sklearn.SklearnCheckpoint
+    ~rl.RLCheckpoint


Personally feel like we should move these to the predictor/checkpoint pages even though they're in the train package...

I think checkpoints should stay in train, since they are still needed by users of Train only (w/o prediction). I can include another reference in the air checkpoint page for framework specific ckpts, but without toctree.

Predictors I agree should be moved out.

matthewdeng · 2023-02-10T20:12:24Z

doc/source/ray-air/api/serve.rst

@@ -0,0 +1,18 @@
+.. _air-serve-integration:
+
+Model Serving in AIR


Add ray.serve.air_integrations.PredictorWrapper?

PredictorWrapper should be on this page already.

Ahhh woops the rendering looked a bit strange because of the alias

My mistake!

Yeah, to be fair this page is kinda a mess. The see also tooltip + autosummary + autofunction is a bit much. But I had to do switch predictor deployment to an autofunction because it's an empty class that was erroring if I put it in autosummary.

matthewdeng · 2023-02-10T20:13:34Z

doc/source/ray-air/api/api.rst

+    AIR Checkpoint (All Libraries)<checkpoint.rst>
+    Predictors (Ray Data + Ray Train)<predictor.rst>
+    Model Serving in AIR (Ray Serve)<serve.rst>
+    External Library Integrations<../../tune/api_docs/integration>


Just double checking that we're intentionally keeping these unchanged?

The external libraries one will be updated with the Tune PR. The other ones are updated.

…n/api_restructure Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

…in air Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

…n/api_restructure

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

…n/api_restructure

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

c21

cc @richardliaw - can you merge this?

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

…n/api_restructure

justinvyu · 2023-02-13T20:17:28Z

@richardliaw Let's wait to merge after tests pass.

richardliaw · 2023-02-13T20:28:42Z

ping me when tests pass

…

On Mon, Feb 13, 2023 at 12:17 PM Justin Yu ***@***.***> wrote: @richardliaw <https://github.com/richardliaw> Let's wait to merge after tests pass. — Reply to this email directly, view it on GitHub <#32360 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCRZZNLX3C66YPR277ZZEDWXKJGJANCNFSM6AAAAAAUWMM5MI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

…n/api_restructure

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

…n/api_restructure

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to ray-project#31204 and ray-project#32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to ray-project#31204 and ray-project#32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <justinvyu@berkeley.edu> Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to ray-project#31204 and ray-project#32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <justinvyu@berkeley.edu> Signed-off-by: elliottower <elliot@elliottower.com>

justinvyu added 3 commits February 9, 2023 01:05

Update train api refs

2aeb40a

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Update AIR api refs

56ed281

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Remove unused tune.rst file

6f7d580

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

justinvyu requested review from richardliaw, gjoliver, krfricke, xwjiang2010, amogkam, matthewdeng, Yard1, maxpumperla and a team as code owners February 9, 2023 10:49

maxpumperla approved these changes Feb 9, 2023

View reviewed changes

justinvyu assigned matthewdeng and c21 Feb 9, 2023

justinvyu added release-blocker P0 Issue that blocks the release P0 Issues that should be fixed in short order docs An issue or change related to documentation labels Feb 9, 2023

c21 reviewed Feb 9, 2023

View reviewed changes

justinvyu added 2 commits February 10, 2023 11:43

Fix broken ray.air.checkpoint refs

239046f

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Fix remaining duplicate refs

1f65aa7

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

matthewdeng reviewed Feb 10, 2023

View reviewed changes

justinvyu added 7 commits February 10, 2023 12:29

Merge branch 'master' of https://github.com/ray-project/ray into trai…

b0a4f58

…n/api_restructure Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Fix todos

358f383

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Move predictor refs out of train, reference framework-specific ckpts …

396020e

…in air Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Merge branch 'master' of https://github.com/ray-project/ray into trai…

5f7bfee

…n/api_restructure

add missing toctree

b025c41

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Merge branch 'master' of https://github.com/ray-project/ray into trai…

b0b9c4e

…n/api_restructure

Final few broken ref fixes

327e02f

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

matthewdeng approved these changes Feb 11, 2023

View reviewed changes

c21 approved these changes Feb 13, 2023

View reviewed changes

c21 assigned richardliaw Feb 13, 2023

justinvyu added 3 commits February 13, 2023 12:15

Add dependencies to prevent docs import error

eaff6c1

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Add flaml dependency

39438d9

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Merge branch 'master' of https://github.com/ray-project/ray into trai…

e93ef3e

…n/api_restructure

justinvyu added 6 commits February 13, 2023 18:17

Install horovod with install-horovod.sh instead

45439e5

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Merge branch 'master' of https://github.com/ray-project/ray into trai…

06b2850

…n/api_restructure

Try pinning ruamel version

67d9c88

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Install mosaic separately

290d0ce

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Remove flaml

f51c6d9

Signed-off-by: Justin Yu <justinvyu@berkeley.edu>

Merge branch 'master' of https://github.com/ray-project/ray into trai…

f6c154b

…n/api_restructure

krfricke approved these changes Feb 14, 2023

View reviewed changes

krfricke merged commit d87d86f into ray-project:master Feb 14, 2023

This was referenced Feb 14, 2023

[2.3.0][AIR][Train][Doc] Restructure API reference #32535

Merged

[AIR][Doc] Add back important Ray Train integration methods (for Torch/TF) #32551

Merged

justinvyu deleted the train/api_restructure branch February 16, 2023 16:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIR][Train][Doc] Restructure API reference #32360

[AIR][Train][Doc] Restructure API reference #32360

justinvyu commented Feb 9, 2023 •

edited

Loading

maxpumperla left a comment

c21 left a comment

matthewdeng left a comment

matthewdeng Feb 10, 2023

matthewdeng Feb 10, 2023

justinvyu Feb 10, 2023

matthewdeng Feb 10, 2023

justinvyu Feb 10, 2023

matthewdeng Feb 10, 2023

justinvyu Feb 10, 2023

matthewdeng Feb 10, 2023

matthewdeng Feb 10, 2023

justinvyu Feb 10, 2023

matthewdeng Feb 10, 2023

justinvyu Feb 10, 2023

c21 left a comment

justinvyu commented Feb 13, 2023

richardliaw commented Feb 13, 2023 via email

		@@ -0,0 +1,18 @@
		.. _air-serve-integration:

		Model Serving in AIR

[AIR][Train][Doc] Restructure API reference #32360

[AIR][Train][Doc] Restructure API reference #32360

Conversation

justinvyu commented Feb 9, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

maxpumperla left a comment

Choose a reason for hiding this comment

c21 left a comment

Choose a reason for hiding this comment

matthewdeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c21 left a comment

Choose a reason for hiding this comment

justinvyu commented Feb 13, 2023

richardliaw commented Feb 13, 2023 via email

justinvyu commented Feb 9, 2023 •

edited

Loading