Questions about the imitation learning

[Tortes]

• I have marked all applicable categories:
  • exception-raising bug
  • RL algorithm bug
  • documentation request (i.e. "X is missing from the documentation.")
  • new feature request
• I have visited the source website
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and environment, where applicable:

  import tianshou, torch, sys
  print(tianshou.__version__, torch.__version__, sys.version, sys.platform)

Now I am training a new network with no pre-trained weights, while I meet with the action asturation problem(the agent only take one action). Therefore, I prepare to train the network with supervised data(generate from another optimization algorithm). So how could I send the supervised data to tianshou, or I have to write another script for training. Thanks for any help!
By the way, I currently use the PPO with onpolicy trainer, and is there any examples to #188 if it could solve my problem ?

[Trinkle23897]

You can refer to the imitation learning script, as provided in test/discrete/test_a2c_with_il.py or test/continuous/test_sac_with_il.py. Another way is to format your data as a ReplayBuffer (ReplayBuffer can use pickle.save and pickle.load to store/load data) as the initialized data, therefore you can collect data offline and train your agent with any data you want.

[zhujl1991]

You can refer to the imitation learning script, as provided in test/discrete/test_a2c_with_il.py or test/continuous/test_sac_with_il.py. Another way is to format your data as a ReplayBuffer (ReplayBuffer can use pickle.save and pickle.load to store/load data) as the initialized data, therefore you can collect data offline and train your agent with any data you want.

If I understand correctly, test/discrete/test_a2c_with_il.py or test/continuous/test_sac_with_il.py actually use the data collected by a2c/sac to train imitation learning, right? How can I disable the online collecting logic so that I can only use the offline collected data to train imitation model?

I think it looks confusing to train any offline algorithms with onpolicy/offpolicy trainer. IMO, there should be a separate offline trainer.

[Trinkle23897]

You can use the buffer with collected data, together with pickle.load and pickle.save.

[zhujl1991]

pickle.load

But when training, it still has to interact with env via collector, which is unnecessary for offline algorithms, right?
Do you think it makes sense to have a separate offline trainer for all offline algorithms, e.g., imitation, BCQ?

[Trinkle23897]

That's a good point. At first I didn't implement this kind of trainer because we always know Dagger is better than BC.