Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Forward fix for failing PPO Torch RLTrainer test #32308

Merged
merged 136 commits into from
Feb 8, 2023
Merged

[RLlib] Forward fix for failing PPO Torch RLTrainer test #32308

merged 136 commits into from
Feb 8, 2023

Conversation

kouroshHakha
Copy link
Contributor

Why are these changes needed?

image

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@kouroshHakha
added quick cleanups to trainer_runner.
28679ca 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
created test_trainer_runner
a2f9439 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
added TODO tag
d8b36c1 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
Merge branch 'master' into trainer-runner-quick-cleanups
d24bff5 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
fixed imports
2b67577 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
typo in BUILD
fe60e20 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
started to create torch_rl_trainer
916d674 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
added bc_rl_trainer
71026e5 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
torch trainer test works now
ae61014 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
lint
ef1ffb8 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
Merge branch 'master' into torch-trainer
f393cea 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
updated TODOs and BUILD
16f64f9 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
wip: trainer_runner multi-gpu test
9719518 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
torch version runs but the parameters are not synced
77730d8 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
wip
97573dc 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
got the multi-gpu gradient sync up working
091c406 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
fixed add/remove multi-gpu tests
a42b0f1 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
moved the DDPRLModuleWrapper outside of RLTrainer + lint
62fe11f 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
merged tf and torch train_runner tests
2cc5185 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
fixed trainer_runner auto-scaling on a cluster where autoscaling is e…
435c352 
…nabled

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
fix rl_trainer unittest failures.
ff845c3 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
1. renamed the DDP wrapper
7c3eed7 
2. don't do numpy conversion for batch on the base class

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
removed in_test from the production code
d56ce2c 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
clarified todo
200b5f7 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
comments
f747e50 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
renamed make_distributed to make_distributed_module
7ce81f0 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
fixed test torch rl_trainer lint
b2ddd2d 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
fixed marl_module stuff
5bc625c 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
fixed the import issue
3302db7 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
lint
5455e29 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
1. set resources for trainable 2. convert_to_numpy weights on RLTrain…
f28a385 
…er get_weights()

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
added examples as a unittest to BUILD kite
993932f 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
fixed test name conflict
05c8297 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
removed the wrong tag from docs
839ff90 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
Merge branch 'master' into ppo-torch-trainer
320b116
@kouroshHakha
fixed as test flag
8466fc8 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
Merge branch 'master' into ppo-torch-trainer
4c4f7cc
@kouroshHakha
made the sync_weights equivalent to the implementation before this PR
d78f3d5 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
addressed jun's comments, created a minibatchCycleIterator
7a68bb4 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
Merge branch 'ppo-torch-trainer' of github.com:kouroshHakha/ray into …
0008833 
…ppo-torch-trainer
@kouroshHakha
added annotations
edbf081 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
Merge branch 'master' into ppo-torch-trainer
2698973
@kouroshHakha
empty
4c8ce18
@kouroshHakha
empty
9f29038 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
empty
b1d3f63 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@kouroshHakha
Merge branch 'master' into ppo-torch-trainer
f345c70
@kouroshHakha
fwd fix for the failing test
653bf3d 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
@gjoliver gjoliver merged commit b85eb52 into ray-project:master Feb 8, 2023
edoakes pushed a commit to edoakes/ray that referenced this pull request Mar 22, 2023
@kouroshHakha @edoakes
[RLlib] Forward fix for failing PPO Torch RLTrainer test (ray-project…
a1b4507 
…#32308)

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gjoliver gjoliver gjoliver approved these changes

@sven1977 sven1977 Awaiting requested review from sven1977 sven1977 is a code owner

@avnishn avnishn Awaiting requested review from avnishn

@ArturNiederfahrenhorst ArturNiederfahrenhorst Awaiting requested review from ArturNiederfahrenhorst

@smorad smorad Awaiting requested review from smorad

@maxpumperla maxpumperla Awaiting requested review from maxpumperla

@krfricke krfricke Awaiting requested review from krfricke

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kouroshHakha @gjoliver @xwjiang2010