Track histogram of environment reward

[awjuliani]

Proposed change(s)

Tracks the episodic reward as a histogram in addition to a scalar in tensorboard. This allows the multi-modality of rewards to be more apparent to end-users.

Below is an example of such a graph generated from a PushBlock training run.

Here is the learning curve from FoodCollector:

Not sure that my current solution is the most optimal given how the StatsSummary and StatsWriters are current designed. Open to feedback.

Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

Other comments

[chriselion]

StatsSummary.empty().num will return 1 now.

[awjuliani]

My first thought is to use an empty list for full_dist when calling empty(). That means the other stats will return NaN which is technically more correct than returning 0, thought may break something else. Do you have a different preference?

[chriselion]

Only other thought is to do e.g. np.mean(self.full_dist) if self.full_dist) else 0.0 which is basically the old behavior, and same for np.std (np.sum is safe)

[chriselion]

But I think this is fine too.

[chriselion]

Can you make it so that user stats can use this too? I think it's pretty small, you'd just need to

• Add a new StatsAggregationMethod enum values (say, HISTOGRAM) on python and C#
• Make sure Environment/Cumulative Reward uses the new value
• Check StatsSummary.aggregation_method in TensorBoard writer instead of hard-coding the key

[awjuliani]

Can you make it so that user stats can use this too? I think it's pretty small, you'd just need to

• Add a new StatsAggregationMethod enum values (say, HISTOGRAM) on python and C#
• Make sure Environment/Cumulative Reward uses the new value
• Check StatsSummary.aggregation_method in TensorBoard writer instead of hard-coding the key

Made these changes.

[ervteng]

Nice, this is actually super powerful for other things besides the reward too 🚢 🇮🇹