[change] Move hyperparameter printing entirely into StatsWriters

[ervteng]

Proposed change(s)

Hyperparameter printing was originally handled by the TrainerController and Trainer. This PR unifies the Tensorboard Hyperparameter save with the console one, and moves both into their respective ConsoleWriter and TensorboardWriter classes.

To do this, we've removed the write_text method from StatsWriter, which was only used to write hyperparameters (and in a TB-specific way). We now have a add_property method that takes in a key. This is a general way to pass data to StatsWriters, which some (or all) may choose to ignore. For instance, the hyperparameters property is a Dict and is only used by TensorboardWriter and ConsoleWriter.

There are small changes to the output in the console:

OLD

2020-03-12 15:15:15 INFO [trainer_controller.py:178] Hyperparameters for the PPOTrainer of brain 3DBall: 
	trainer:	ppo
	batch_size:	64
	beta:	0.001
	buffer_size:	12000
	epsilon:	0.2
	hidden_units:	128
	lambd:	0.99
	learning_rate:	0.0003
	learning_rate_schedule:	linear
	max_steps:	5.0e5
	memory_size:	128
	normalize:	True
	num_epoch:	3
	num_layers:	2
	time_horizon:	1000
	sequence_length:	64
	summary_freq:	12000
	use_recurrent:	False
	vis_encode_type:	simple
	reward_signals:	
	  extrinsic:	
	    strength:	1.0
	    gamma:	0.99
	summary_path:	testsummarywrite_3DBall
	model_path:	./models/testsummarywrite/3DBall
	keep_checkpoints:	5

NEW

2020-03-12 15:09:37 INFO [stats.py:109] Hyperparameters for behavior name testsummarywrite_3DBall: 
	trainer:	ppo
	batch_size:	64
	beta:	0.001
	buffer_size:	12000
	epsilon:	0.2
	hidden_units:	128
	lambd:	0.99
	learning_rate:	0.0003
	learning_rate_schedule:	linear
	max_steps:	5.0e5
	memory_size:	128
	normalize:	True
	num_epoch:	3
	num_layers:	2
	time_horizon:	1000
	sequence_length:	64
	summary_freq:	12000
	use_recurrent:	False
	vis_encode_type:	simple
	reward_signals:	
	  extrinsic:	
	    strength:	1.0
	    gamma:	0.99
	summary_path:	testsummarywrite_3DBall
	model_path:	./models/testsummarywrite/3DBall
	keep_checkpoints:	5

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

[chriselion]

Should key be an enum instead?

[chriselion]

(maybe StatsPropertyType)

[ervteng]

This makes sense - less magic strings.

[chriselion]

Suggested change

	hyperparameters = "hyperparameters"
	HYPERPARAMETERS = "hyperparameters"

I think generally python enums use SCREAMING_SNAKE_CASE.

[chriselion]

Do you anticipate other parameter types being added in the future? Right now it seems a little weird to have a general proprerty interface, but only use it for hyperparameters.

[ervteng]

Do you anticipate other parameter types being added in the future? Right now it seems a little weird to have a general proprerty interface, but only use it for hyperparameters.

I actually took this idea from the progress bar branch, since there we need to write max_step to the StatsWriters as well (but only certain writers will need it). I'd also like to use it to differentiate between a multi-agent trainer and a regular trainer (and have the ConsoleWriter output ELO instead of reward). I'd imagine there are other one-time parameters that we'll need eventually.

[chriselion]

Cool, as long as they're going to get more use in the future, sounds good.