[RLlib; Off-policy] Add episode sampling to EpisodeReplayBuffer.

[simonsays1980]

Why are these changes needed?

The EpisodeReplayBuffer is still sampling batches, i.e. dicts. This PR proposes a way to add episode sampling to the buffer such that the buffer can be used in the same form as the other episode buffers. This is also a preliminary for using buffers in the OfflinePreLearner.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

Awesome unification PR! This opens the door to a DQfD style mixing of offline and off-policy RL!

Dumb question: Would this already work even if the PrioritizedEpisodeReplayBuffer doesn't support this flag yet?

[simonsays1980]

Awesome unification PR! This opens the door to a DQfD style mixing of offline and off-policy RL!

Dumb question: Would this already work even if the PrioritizedEpisodeReplayBuffer doesn't support this flag yet?

Thanks @sven1977! I guess that mixing would be possible even before? Wouldn't it? We just use DQN and add a dataset, then sample from the buffer and let the OfflinePreLearner return episodes instead of MultiAgentBatch to be able to mix the two episode lists. Then pass the lists to the trainer. We cannot make use of the offline pipeline because we need to add continuously episodes to the off-policy buffer and this one should exist over the full course of training.

The PrioritizedEpisodeReplayBuffer did already support the episode sampling and could be used in the OfflinePreLearner. Imo this, however, would somehow bias offline learning because we would filter data between offline data and learner. But, if a user wants to have a prioritized sampling from her offline data, we could support this out-of-the-box :) Will add a buffer definition to the AlgorithmConfig of offline().

[sven1977]

LGTM now. Thanks a ton @simonsays1980 !!