Configurable ReplayBuffer

[pbontrager]

Added custom Sample and Eviction policies to Replay buffer. Our existing replay buffer did not allow for custom handling of these two properties which are important for tuning async training runs. To allow these functions to be generic and user defined, I added two callable inputs, sample_policy and eviction_policy, they take in the current buffer, some buffer parameters (max_policy_age, max_resample_count) and return a list of indices to either evict or sample. An list of indices is returned instead of directly modifying the buffer so we can control how the buffer is updated ourself instead of the user.

Changelog:

• BufferEntry type created to track episode sample_counts. This allows for user control over how often data can be resampled
• default_evict and default_sample are default policies that match the existing behavior of the ReplayBuffer
• max_buffer_size added with a first in first out policy for removing old data (the assumption is that stale data will have already been removed by evict
• max_policy_age, max_resample_count, and max_buffer_size all support a None option now to remove this restriction
• buffer changed from list to deque. This allows for efficient enforcement of max_buffer_size without constant reallocation of the entire buffer.
• _collect method added to get the buffer entries given a list of indices. Needs special handling to do this efficiently with a deque object.
• calling sample no longer removes sampled data from the buffer, is just increments the sample count for that data. It's up to the eviction policy to remove data based on sample count.
• removed buffer_size parameter from sample to simplify api
• updated unit_test for api changes and slight behavior change in when sampled data is removed.

[joecummings]

Can we put some type hints?

[joecummings]

Can you also add a test specifically for _collect?

[joecummings]

Rename this to the actual things they do lol.

default_sample -> random_sample

[joecummings]

default_evict -> evict_if_too_old

[joecummings]

Please contain all sampling logic in one piece.

[joecummings]

typehints?