Make SubprocessEnvManager take asynchronous steps #2265
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chriselion
reviewed
Jul 15, 2019
chriselion
reviewed
Jul 15, 2019
ervteng
reviewed
Jul 15, 2019
ervteng
approved these changes
Jul 17, 2019
chriselion
reviewed
Jul 17, 2019
| from multiprocessing.connection import Connection | ||
| from queue import Empty as EmptyQueue |
There was a problem hiding this comment.
nit: maybe call this EmptyQueueException? bad naming in the stdlib :/
chriselion
reviewed
Jul 17, 2019
chriselion
approved these changes
Jul 17, 2019
added 4 commits
July 19, 2019 14:19
SubprocessEnvManager takes steps synchronously to reproduce old behavior, meaning all parallel environments will need to wait for the slowest environment to take a step. If some steps take much longer than others, this can lead to a substantial overall slowdown in practice. We've seen extreme cases where we see almost a 2x speedup from using asynchronous stepping, with no downside for our faster environments. This PR changes the SubprocessEnvManager to use async stepping. This means on the "step" call the environment manager will enqueue step requests to workers, and then only wait until at least one step has been completed before returning.
harperj
force-pushed
the
develop-async-env-manager
branch
from
503249b
to
855eaff
Compare
mantasp
added a commit
that referenced
this pull request
Jul 22, 2019
* develop: (69 commits) Add different types of visual encoder (nature cnn/resnet) Make SubprocessEnvManager take asynchronous steps (#2265) update mypy version one more unused remove unused variables Fix respawn part of BananaLogic (#2277) fix whitespace and line breaks remove codacy (#2287) Ported documentation from other branch tennis reset parameter implementation ported over Fixed the default value to match the value in the docs two soccer reset parameter implementation ported over 3D ball reset parameter implementation ported over 3D ball reset parameter implementation ported over Relax the cloudpickle version restriction (#2279) Fix get_value_estimate and buffer append (#2276) fix lint checks Add Unity command line arguments Swap 0 set and reward buffer append (#2273) GAIL and Pretraining (#2118) ...
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SubprocessEnvManager takes steps synchronously to reproduce old
behavior, meaning all parallel environments will need to wait for
the slowest environment to take a step. If some steps take much
longer than others, this can lead to a substantial overall slowdown
in practice. We've seen extreme cases where we see almost a 2x
speedup from using asynchronous stepping, with no downside for our
faster environments.
This PR changes the SubprocessEnvManager to use async stepping.
This means on the "step" call the environment manager will enqueue
step requests to workers, and then only wait until at least one
step has been completed before returning.
Testing
Reacher environment on GCP with 8 parallel environments. 15000 steps. 16.5% improvement in overall time for Async EnvManager, comparable reward.
Walker environment on GCP with 8 parallel environments. 30000 steps. 14.2% improvement in overall time for Async EnvManager, comparable reward.