How to stop training after a fixed number of steps?

[CloudyDory]

The default OnPolicyTrainer has an option to stop training after a fixed number of episodes. However, sometimes it is desirable to stop after a fixed number of total environment steps that the agent has experienced with. How to implement it in Tianshou?

[Trinkle23897]

tianshou/tianshou/trainer/onpolicy.py

Lines 28 to 35 in 4ac407c

	:param int step_per_collect: the number of transitions the collector would
	collect before the network update, i.e., trainer will collect
	"step_per_collect" transitions and do some policy network update repeatedly
	in each epoch.
	:param int episode_per_collect: the number of episodes the collector would
	collect before the network update, i.e., trainer will collect
	"episode_per_collect" episodes and do some policy network update repeatedly
	in each epoch.

You can specify either step_per_collect or episode_per_collect

[CloudyDory]

Hi, I do not mean to specify the number of steps before a network update. I hope to specify the total number of environment steps in an entire training session. For example, stop training after 3.0e7 steps.

[CloudyDory]

I hope to do so because when training Atari games, we usually plot the epsiodic return as a function of total number of environment steps that the agent has experienced. And I would like to specify the maximum number of environment steps, not the maximum number of episodes.

[Trinkle23897]

https://github.com/thu-ml/tianshou/blob/master/examples/atari/atari_ppo.py

For example, if you want to set total steps 3e7, you can set epoch=100 with --step-per-epoch=3e5: total_steps == epoch * step_per_epoch; you can change number of epoch to control the evaluation interval

[CloudyDory]

Thank you! Another small question: what will happen if the game does not end when reaching the maximum number of steps in an epoch? Will the game stop immediately, or will it stop only after the current episode ends (so the actual steps will be slightly larger than the maximum steps)?

[Trinkle23897]

it will stop stepping env, switch to test env, do one evaluation, then continue stepping the partially stepped training env