Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] PPO torch RLTrainer #31801

Merged
merged 134 commits into from
Feb 8, 2023
Merged

[RLlib] PPO torch RLTrainer #31801

merged 134 commits into from
Feb 8, 2023

Commits on Jan 10, 2023

  1. added quick cleanups to trainer_runner. 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    28679ca View commit details
    Browse the repository at this point in the history

  2. created test_trainer_runner 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    a2f9439 View commit details
    Browse the repository at this point in the history

  3. added TODO tag 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    d8b36c1 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2023

  1. Merge branch 'master' into trainer-runner-quick-cleanups 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    d24bff5 View commit details
    Browse the repository at this point in the history

  2. fixed imports 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    2b67577 View commit details
    Browse the repository at this point in the history

  3. typo in BUILD 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    fe60e20 View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2023

  1. started to create torch_rl_trainer 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    916d674 View commit details
    Browse the repository at this point in the history

  2. added bc_rl_trainer 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    71026e5 View commit details
    Browse the repository at this point in the history

  3. torch trainer test works now 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    ae61014 View commit details
    Browse the repository at this point in the history

  4. lint 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    ef1ffb8 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2023

  1. Merge branch 'master' into torch-trainer 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    f393cea View commit details
    Browse the repository at this point in the history

  2. updated TODOs and BUILD 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    16f64f9 View commit details
    Browse the repository at this point in the history

  3. wip: trainer_runner multi-gpu test 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    9719518 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2023

  1. torch version runs but the parameters are not synced 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 14, 2023
    Configuration menu
    Copy the full SHA
    77730d8 View commit details
    Browse the repository at this point in the history

  2. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 14, 2023
    Configuration menu
    Copy the full SHA
    97573dc View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2023

  1. got the multi-gpu gradient sync up working 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    091c406 View commit details
    Browse the repository at this point in the history

  2. fixed add/remove multi-gpu tests 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    a42b0f1 View commit details
    Browse the repository at this point in the history

  3. moved the DDPRLModuleWrapper outside of RLTrainer + lint 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    62fe11f View commit details
    Browse the repository at this point in the history

  4. merged tf and torch train_runner tests 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    2cc5185 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2023

  1. fixed trainer_runner auto-scaling on a cluster where autoscaling is e… 

    …nabled

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    435c352 View commit details
    Browse the repository at this point in the history

  2. fix rl_trainer unittest failures. 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    ff845c3 View commit details
    Browse the repository at this point in the history

  3. 1. renamed the DDP wrapper 

    2. don't do numpy conversion for batch on the base class

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    7c3eed7 View commit details
    Browse the repository at this point in the history

  4. removed in_test from the production code 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    d56ce2c View commit details
    Browse the repository at this point in the history

  5. clarified todo 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    200b5f7 View commit details
    Browse the repository at this point in the history

  6. comments 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    f747e50 View commit details
    Browse the repository at this point in the history

  7. renamed make_distributed to make_distributed_module 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    7ce81f0 View commit details
    Browse the repository at this point in the history

  8. fixed test torch rl_trainer lint 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    b2ddd2d View commit details
    Browse the repository at this point in the history

  9. fixed marl_module stuff 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    5bc625c View commit details
    Browse the repository at this point in the history

  10. fixed the import issue 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    3302db7 View commit details
    Browse the repository at this point in the history

  11. lint 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    5455e29 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2023

  1. Merge branch 'master' into torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    a2d042f View commit details
    Browse the repository at this point in the history

  2. fixed lint 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    b9159a8 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into torch-trainer 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    f3edd50 View commit details
    Browse the repository at this point in the history

  4. test trainer runner updated 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    2aec198 View commit details
    Browse the repository at this point in the history

  5. fixed the scaling config and in_test issues introduced after the merge. 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    873cdd5 View commit details
    Browse the repository at this point in the history

  6. fixed the scaling config and in_test issues introduced after the merge. 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    13e19aa View commit details
    Browse the repository at this point in the history

  7. Merge branch 'torch-trainer' of github.com:kouroshHakha/ray into torc… 

    …h-trainer
    @kouroshHakha
    kouroshHakha committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    68b72e4 View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2023

  1. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    ca3e225 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    ff3b335 View commit details
    Browse the repository at this point in the history

  3. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    3234aaf View commit details
    Browse the repository at this point in the history

  4. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    15b99ee View commit details
    Browse the repository at this point in the history

  5. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    8b9ae92 View commit details
    Browse the repository at this point in the history

  6. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    eb82e67 View commit details
    Browse the repository at this point in the history

  7. fixed trainer_runner config test 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    dac0d6b View commit details
    Browse the repository at this point in the history

  8. removed the stuff that got moved to SARLTrainer made easy PR 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    cfdaa04 View commit details
    Browse the repository at this point in the history

  9. fixed torch import 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    7b5938b View commit details
    Browse the repository at this point in the history

  10. removed the override decorator for nn.Module 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    f4cbe5a View commit details
    Browse the repository at this point in the history

  11. Merge branch 'master' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    9fb749c View commit details
    Browse the repository at this point in the history

  12. added unittest (wip) 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    9645137 View commit details
    Browse the repository at this point in the history

  13. fixed import torch in bc_module.py 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    eac5223 View commit details
    Browse the repository at this point in the history

  14. fixed the bazel bug where the working directory gets switched to wher… 

    …e the unittest is locateed and import torch would import the relative torch module instead of the global torch moduel

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    0baa3cf View commit details
    Browse the repository at this point in the history

  15. Merge branch 'torch-trainer' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    78860f8 View commit details
    Browse the repository at this point in the history

  16. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    c125e20 View commit details
    Browse the repository at this point in the history

Commits on Jan 21, 2023

  1. Merge branch 'master' into ppo-torch-trainer 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    5aaa603 View commit details
    Browse the repository at this point in the history

  2. fixed the unittest 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    c77be0c View commit details
    Browse the repository at this point in the history

  3. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    0e6f511 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2023

  1. Merge branch 'master' into policy-with-marl

    @kouroshHakha
    kouroshHakha committed Jan 23, 2023
    Configuration menu
    Copy the full SHA
    1778c44 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2023

  1. added dataclass specs for RLModule and MARLModule for easier construc… 

    …tion from RLTrainer side

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    4bdd949 View commit details
    Browse the repository at this point in the history

  2. test trainer runner local passed 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    58bbe82 View commit details
    Browse the repository at this point in the history

  3. add_module() api is now update to accept a module_spec instead of mod… 

    …ule_Cclass and module_kwargs

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    93b27ec View commit details
    Browse the repository at this point in the history

  4. get_trainer_runner_config() now gets an optional ModuleSpec object 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    7aaee18 View commit details
    Browse the repository at this point in the history

  5. Algorithm can now construct the trainer_runner based on the policy_maps 

    created on local_worker

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    7c831d3 View commit details
    Browse the repository at this point in the history

  6. lint and clean up 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    d3d610e View commit details
    Browse the repository at this point in the history

  7. fixed imports 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    97acdb7 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'policy-with-marl' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    f3ccf54 View commit details
    Browse the repository at this point in the history

  9. fixed the unittest for ppo_rl_trainer 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    93f3ce9 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2023

  1. got the PPO POC running 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    c58293a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    3f36f0d View commit details
    Browse the repository at this point in the history

  3. lint 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    77ff585 View commit details
    Browse the repository at this point in the history

  4. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    87bda01 View commit details
    Browse the repository at this point in the history

  5. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    6346a20 View commit details
    Browse the repository at this point in the history

  6. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    eb5106f View commit details
    Browse the repository at this point in the history

  7. multi-gpu test works now 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    b2c01ad View commit details
    Browse the repository at this point in the history

  8. removed left out api get_weight() 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    ea3d9c6 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2023

  1. get_weights() updated 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    45830c6 View commit details
    Browse the repository at this point in the history

  2. trying out a new configuration pattern for trainer runner and rl trai… 

    …ners

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    94ca772 View commit details
    Browse the repository at this point in the history

  3. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    29ac2fb View commit details
    Browse the repository at this point in the history

  4. lint 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    4714e20 View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2023

  1. rl_trainer tf test passes again 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    abd5e5e View commit details
    Browse the repository at this point in the history

  2. torch rl trainer test passed 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    a44c370 View commit details
    Browse the repository at this point in the history

  3. trainer_runner_config test works too 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    e496fcc View commit details
    Browse the repository at this point in the history

  4. tested the multigpu 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    477795d View commit details
    Browse the repository at this point in the history

  5. docstring updated 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    58fe5df View commit details
    Browse the repository at this point in the history

  6. updated the docstring 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    d5bcd3b View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2023

  1. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 28, 2023
    Configuration menu
    Copy the full SHA
    d8841d1 View commit details
    Browse the repository at this point in the history

  2. renamed the classes and variables to backend 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 28, 2023
    Configuration menu
    Copy the full SHA
    026899e View commit details
    Browse the repository at this point in the history

  3. renamed 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 28, 2023
    Configuration menu
    Copy the full SHA
    f04e99d View commit details
    Browse the repository at this point in the history

  4. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 28, 2023
    Configuration menu
    Copy the full SHA
    d280887 View commit details
    Browse the repository at this point in the history

  5. refactor 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 28, 2023
    Configuration menu
    Copy the full SHA
    97d80b1 View commit details
    Browse the repository at this point in the history

  6. lin 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 28, 2023
    Configuration menu
    Copy the full SHA
    85387e5 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2023

  1. fix the lint and tf_dependency test issue via adding tf stubs 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    1a70b6e View commit details
    Browse the repository at this point in the history

  2. wip on unittest trianer_runner 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    317a9fd View commit details
    Browse the repository at this point in the history

  3. wip 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    cbc9b02 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into trainer-runner-scaling-config

    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    869717e View commit details
    Browse the repository at this point in the history

  5. test_trainer_runner updated to support all variations of scaling config 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    defa5f1 View commit details
    Browse the repository at this point in the history

  6. removed test trainer runner local and moved it to test_trainer_runner.py 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    e0a0bcf View commit details
    Browse the repository at this point in the history

  7. fixed the test failures 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    cf4041e View commit details
    Browse the repository at this point in the history

  8. 1. Removed tf due to flakiness from test_trainer_runner 

    2. Moved the logic of trainerScalingConfig into the object instead of
using a separate torch/tf specific dataclass.

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    d4cd654 View commit details
    Browse the repository at this point in the history

  9. removed backend class definitions 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    54eb315 View commit details
    Browse the repository at this point in the history

  10. Removed Hyperparams class 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    1c826db View commit details
    Browse the repository at this point in the history

  11. introed FrameworkHPs to differebntiate between tf/torch specific stuf… 

    …f vs. algorithm specific stuff

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    e8cf7e1 View commit details
    Browse the repository at this point in the history

  12. the unittests pass 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    2ea3a4d View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2023

  1. Merge branch 'trainer-runner-scaling-config' into ppo-torch-trainer 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    92dd832 View commit details
    Browse the repository at this point in the history

  2. addressed comments and fixed some introduced bug 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    fd84f7a View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into ppo-torch-trainer 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    4c38455 View commit details
    Browse the repository at this point in the history

  4. fix from_worker_or_trainer renaming issue 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    d40c48d View commit details
    Browse the repository at this point in the history

  5. fixed tests 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    b0fed29 View commit details
    Browse the repository at this point in the history

  6. fixed test_ppo_rl_trainer.py 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    b92eee9 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2023

  1. rerunning ci 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    1c3ccb5 View commit details
    Browse the repository at this point in the history

  2. lint 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    37c9fca View commit details
    Browse the repository at this point in the history

  3. added TODO 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    1687241 View commit details
    Browse the repository at this point in the history

  4. empty commit 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    d3ee81a View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    24abebd View commit details
    Browse the repository at this point in the history

  6. fixed weights to numpy 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    3ff0668 View commit details
    Browse the repository at this point in the history

  7. [release] minor fix to pytorch_pbt_failure test when using gpu. (ray-… 

    …project#32070)

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
    @xwjiang2010 @kouroshHakha
    xwjiang2010 authored and kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    d91eca5 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'master' into err-out-marl-env

    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    5b209be View commit details
    Browse the repository at this point in the history

  9. Merge branch 'err-out-marl-env' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    833e491 View commit details
    Browse the repository at this point in the history

  10. error out when no agent is passed in in the indepenent MARL case 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    e29021d View commit details
    Browse the repository at this point in the history

  11. Merge branch 'err-out-marl-env' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    d113d3a View commit details
    Browse the repository at this point in the history

  12. 1. set resources for trainable 2. convert_to_numpy weights on RLTrain… 

    …er get_weights()

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    f28a385 View commit details
    Browse the repository at this point in the history

  13. added examples as a unittest to BUILD kite 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    993932f View commit details
    Browse the repository at this point in the history

  14. fixed test name conflict 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    05c8297 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2023

  1. removed the wrong tag from docs 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Feb 1, 2023
    Configuration menu
    Copy the full SHA
    839ff90 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Feb 1, 2023
    Configuration menu
    Copy the full SHA
    320b116 View commit details
    Browse the repository at this point in the history

  3. fixed as test flag 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Feb 1, 2023
    Configuration menu
    Copy the full SHA
    8466fc8 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2023

  1. Merge branch 'master' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    4c4f7cc View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2023

  1. made the sync_weights equivalent to the implementation before this PR 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Feb 6, 2023
    Configuration menu
    Copy the full SHA
    d78f3d5 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2023

  1. addressed jun's comments, created a minibatchCycleIterator 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    7a68bb4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'ppo-torch-trainer' of github.com:kouroshHakha/ray into … 

    …ppo-torch-trainer
    @kouroshHakha
    kouroshHakha committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    0008833 View commit details
    Browse the repository at this point in the history

  3. added annotations 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    edbf081 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into ppo-torch-trainer

    @kouroshHakha
    kouroshHakha committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    2698973 View commit details
    Browse the repository at this point in the history

  5. empty

    @kouroshHakha
    kouroshHakha committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    4c8ce18 View commit details
    Browse the repository at this point in the history

  6. empty 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    9f29038 View commit details
    Browse the repository at this point in the history

  7. empty 

    Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
    @kouroshHakha
    kouroshHakha committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    b1d3f63 View commit details
    Browse the repository at this point in the history