[FIX] prioritize cli options over yaml #5495

maryamhonari · 2021-08-06T17:21:07Z

Proposed change(s)

steps to reproduce the bug:

run exp1 for 1M steps
initialize exp2 from last one with yaml["checkpoint_settings"]["initialize_from"] and continue running for eg. 200k
resume exp2 from cmd and the step count continues from 1M instead of 200k

solution: checkpointSetting class now prioritize the command line arguments (resume or initialize_from for now) over conflicting yaml options. yaml option is nuke with a warning

Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)

Types of change(s)

Checklist

Added tests that prove my fix is effective or that my feature works
Updated the changelog (if applicable)
Updated the documentation (if applicable)
Updated the migration guide (if applicable)

Other comments

ml-agents/mlagents/trainers/settings.py

ervteng · 2021-08-11T22:06:23Z

ml-agents/mlagents/trainers/tests/test_learn.py

+    # ignore init if resume set
+    full_args.append("--resume")
+    opt = parse_command_line(full_args)
+    assert opt.checkpoint_settings.initialize_from is None  # ignore init if resume set
    assert opt.checkpoint_settings.resume is True



Do we need a test to check if they were both set in the YAML but not the CLI? I think we can do this in the test_settings.py.

ervteng

Forgot in my last approval - add this bug fix to the CHANGELOG.md file and reference the PR before merging.

maryamhonari added 3 commits August 6, 2021 10:06

priorotize cli over yaml in checkpointSettings

7ab60fb

priorotize resume if both set in one place

dfc903a

clean up

03c3dad

maryamhonari requested a review from ervteng August 9, 2021 22:36

maryamhonari added 2 commits August 9, 2021 17:59

fixed test_commandline_args

38a89c6

formating

ddee367

ervteng reviewed Aug 10, 2021

View reviewed changes