ML-Agents Beta 0.11.0

Major Changes

• The BroadcastHub has been deprecated. If there is a training Python process, all LearningBrains in the scene will automatically be trained. If there is no Python process, inference will be used.
• The Brain ScriptableObjects have been removed. The Brain Parameters are now on the Agent and are referred to as Behavior Parameters. Make sure the Behavior Parameters is attached to the Agent GameObject.
• The setup for visual observations has changed significantly.
  • Camera resolutions are no longer stored in the Brain Parameters.
  • AgentParameters no longer stores lists of Cameras and RenderTextures
  • To add visual observations to an Agent, you must now attach a CameraSensorComponent or RenderTextureComponent to the agent. The corresponding Camera or RenderTexture can be added to these in the editor, and the resolution and color/grayscale is configured on the component itself.
• The definition of the gRPC service has changed.
• The online BC training feature has been removed.

For instructions on how to migrate from 0.10.x to 0.11.0 see the Migration Guide.

See the updated Getting Started and Basic Guide for more information.

Minor Fixes and Improvements

• Fixed an exception being thrown when dragging between two screens with different DPIs (#2782)
• Improved hyperparameters for the GridWorld environment using SAC (#2776)
• Fixed crash when SAC is used with Curiosity and Continuous Actions (#2740)
• Sped up processing large vector observations (#2717)
• Fixed bug causing excessive memory usage during inference (#2722)
• Added support for multiple training areas in the GridWorld environment (#2721)
• RayPerception.Perceive() was made abstract and RayPerception2D now correctly overrides it. (#2788)

ML-Agents Beta 0.10.1

Fixes & Performance Improvements

• Fix crash when exporting a behavioral cloning model with LSTM (#2679)
• Fix issue when using an SAC model with LSTM in the Unity Inference Engine (#2698)
• Fixed bug with missing cameras in VisualPyramids and VisualPushBlock scenes (#2672)
• Fixed an issue when batch size is a non-multiple of sequence length when using LSTM (#2661)

ML-Agents Beta 0.10.0

New Features

• Soft Actor-Critic (SAC) is added as a new trainer option, complementing Proximal Policy Optimization (PPO). SAC, an off-policy algorithm, is more sample-efficient (i.e., requires fewer environment steps). For environments that take a long time to execute a step (about >0.1 second or greater) this can lead to dramatic training speedups of around 3-5 times versus PPO. In addition to sample-efficiency, SAC has been shown to be robust to small variations in the environment and effective at exploring the environment to find optimal behaviors. See the SAC documentation for more details.
• Example environments have been updated to a new dark-theme visual style and colors have been standardized across all environments.
• Unity environment command line arguments can be passed through mlagents-learn. See the documentation on how to use this feature.

Fixes and Improvements

• ML-Agents is now compatible with Python v3.7 and newer versions of Tensorflow up to 1.14.
• Fixed an issue when using recurrent networks and agents are destroyed. (#2549)
• Fixed a memory leak during inference. (#2541)
• The UnitySDK.log is no longer logged out, which fixes an issue with 2019 versions of the Unity Editor (#2580).
• The Academy class no longer has a Done() method. All Done calls should be handled in the Agent (#2519). See Migrating for more information.
• C# code was updated to follow Unity coding conventions.
• Fixed a crash that happens when enabling VAIL with a GAIL reward signal (#2598)
• Other minor documentation enhancements and bug fixes.

Acknowledgements

• Thanks to @tomatenbrei and everyone at Unity who contributed to v0.10.0.

ML-Agents Beta 0.9.3

Fixes and Improvements

• Fixed a bug in BCTrainer that would cause it to raise an exception when starting training (#2505)
• Fixed a bug in the BananaIL scene that made it unplayable, due to missing reset parameters (#2512)

ML-Agents Beta 0.9.2

New Features

• Added multi-gpu implementation for PPO. You can turn this on by using --multi-gpu option in mlagents-learn. In our own experiments we've observed a 38% boost on the update speed of Resnet using the Obstacle Tower Environment. (#2288)

Fixes and Improvements

• Improved the training performance by 20-30% when curiosity is used. We achieved this by updating the reward signal in parallel with the policy. (#2362)
• Changed the Crawler example environment observation space to include relative observations. Increases performance in CrawlerDynamic scene. (18f661)
• Fixed a bug with online Behavior Cloning. (#2384)
• Fixed a compile error on DemonstrationImporter.cs. (b617de)

Acknowledgements

Thanks to everyone at Unity who contributed to v0.9.2, as well as: @RunSwimFlyRich, @niskander.

ML-Agents Beta 0.9.1

Fixes and Improvements

• Fixed issue where BC Trainer would not start.
• Fixed issue when using Demonstration Recorder with a LearningBrain.
• Fixed issue where Samplers would not use a consistent seed even when one is specified.
• Fix crash when using GAIL, Curiosity, and visual observations together.

ML-Agents Beta 0.9.0

New Features

• Added Generative Adversarial Imitation Learning (GAIL), a new way to do imitation learning. (#2118)
  • Unlike Behavioral Cloning, which requires demonstrations that exhaustively cover all the scenarios that an agent could encounter, GAIL enables imitation learning from as few as 5-10 demonstrations. This makes GAIL more applicable to more problems than Behavioral Cloning, and less work (in recording demonstrations) to set up.
  • GAIL can also be used with reinforcement learning to guide the behavior of the agent to be similar to the demonstrations. In environments where the reward is sparse, providing demonstrations can speed up training by several times. See imitation learning for more information on how to use GAIL, and a comparison of training times on one of our example environments.
• Enabled pre-training for the PPO trainer. (#2118)
  • Pre-training can be used to bootstrap an agent's behavior using human-provided demonstrations, and helps the agent explore in the right direction during training. It can be used in conjunction with GAIL for further training speedup, especially in environments where the agent rarely sees a reward, or gets "stuck" in certain parts. See imitation learning for more information on how to use pre-training.
• Introduced training generalized reinforcement learning agents. (#2232)
  • Agents trained in the same environment throughout the training process can learn to be really good at solving that particular problem. However, when introduced to variations in the environment (e.g., the terrain changes, the agent's physics changes slightly) these agents will fail.
  • This release enables varying the environment during training, so that the trained agent is robust to environment variations. In addition, we've added changeable parameters to our example environments that enable them to train and test these generalized agents. See Training Generalized Agents to learn more about using this feature.
• Changed stepping of the environments to be done asynchronously when running multiple Unity environments. (#2265)
  • Prior to this change, ML-Agents waited for all the parallel environments to complete a step. For environments where some steps (e.g. reset) could take much longer than others, this slows the step collection time to the slowest step. Note that this changes the definition of "step" reported in TensorBoard when using multiple environments.
• Added options for Nature and ResNet CNN architectures when using visual observations. These larger networks may help with more complex visual observations. (#2289)
• Added basic profiling in Python (#2180).

Fixes & Improvements

• Upgraded the Unity Inference Engine to 0.2.4, significantly reducing memory usage during inference (#2308).
• Unified definition of reward sources in trainer_config.yaml across Curiosity, Extrinsic, and GAIL. (#2144)
• Added support for gym wrapper and multiple visual observations. (#2192)
• Added Korean documentation and localization (#2219, #2356)
• Fixed custom reset parameters for SubprocessEnvManager (#2242)
• Fixed spawning bug in VisualBanana example environment. (#2277)
• Fixed memory leak when using visual observations in a Docker container (#2274)
• Added ability to pass in Unity executable command line parameters while instantiating a UnityEnvironment. (#2243)
• Included other minor bug and doc fixes

Workflow Changes

• Enabling Curiosity (as well as GAIL) is done under a new reward_signals parameter in the trainer configuration YAML file.
• When running training using multiple environments, the number of steps reported to TensorBoard now correspond to the number of steps taken per environment, not one per all environment.

See Migrating for more details on workflow changes.

Known Issues

In some rare cases, the model may not be saved when quitting with Ctrl+C on Windows. If this occurs, reload the model by running mlagents-learn using the --load parameter, and attempt saving again.

Acknowledgements

Thanks to everyone at Unity who contributed to v0.9.0, as well as: @BlueFisher, @shakenes, @JangHyeonJun, @Kyushik, @erikfrey, @gregnz, @AcelisWeaven, @rsfutch77, @quevedin, and @LuoYouRen.

ML-Agents Beta 0.8.2

New Features

• The Unity Inference Engine upgraded to version 0.2.2 with fixes for issues on discrete actions and visual observations.
• Video recorder.

Fixes & Improvements

• Added a script to automate the check for metafiles.
• Added no graphics option to gym wrapper.
• Ability to switch to a certain scene within a build before training starts.
• Added a script to turn on the control mode for all the brains in the example scenes.
• Various improvements to the documentation.
• Fixed the issue that the demo filenames might be too long and corrupt the demo files.
• Fixed the issue where the last action recorded by demonstration recorder in an episode was always 0.
• Fixed the null reference bug on demonstration recorder.
• Fixed a bug on parallel environment.
• External PRs require agreement to our Contributors License Agreement (both for individuals and on behalf of corporations)

Acknowledgements

Thanks to everyone at Unity who contributed to v0.8.2, as well as: @TashaSkyUp, @rsfutch77 , @laurentopia, @LeSphax, @GProulx, @automata

ML-Agents Beta 0.8.1

Fixes & Performance Improvements

• Fix issues with pypi packaging of mlagents and mlagents_envs.

ML-Agents Beta 0.8.0

New Features

• Enable training with multiple concurrent Unity simulations.
• Ability to exchange custom protobuf messages.
• Support for 2D ray casting.
• Enable tracking and logging training metrics.
• Splitting ML-Agents package into mlagents.trainers and mlagents.envs.
• Ability to also use RenderTexture for Visual Observations.

Fixes & Performance Improvements

• Various improvements to the documentation.
• Various other bug fixes.

Known Issues

• Inference does not work with multi-discrete action when training with LSTM.

Acknowledgements

Thanks to everyone at Unity who contributed to v0.8.0, as well as: @LeSphax, @pyjamads, @tkggwatson, @malmaud, @supercurious and @rafvasq. Special thanks also go out to our partners Jam City who helped us evaluate the performance gains of this release on a real game, Snoopy Pop.