support for google cloud TPU

[QingyuanWang]

• I have marked all applicable categories:
  • exception-raising bug
  • RL algorithm bug
  • documentation request (i.e. "X is missing from the documentation.")
  • new feature request
• I have visited the [source website], and in particular read the [known issues]
• I have searched through the [issue tracker] for duplicates
• I have mentioned version numbers, operating system and environment, where applicable:

Hi,
I see distributed training in your todo list, does that include support for google cloud TPU?

[Trinkle23897]

For me, I don't use TPU. But @fengredrum said he would like to do this work in the future.

[fengredrum]

Hi, @QingyuanWang
Google just announces an offline RL paradigm, which is suitable for training on a hardware like cloud TPU.

They've released the corresponding dataset and code, you can check the detail from this url: https://ai.googleblog.com/2020/04/an-optimistic-perspective-on-offline.html

Maybe some day I'll write some code about it, just maybe :)

[QingyuanWang]

Thank you for your information, I will check that.

[DrJimFan]

I think you can always use tianshou to rollout env and collect samples, and then train the network on pytorch compiled with TPU support: https://medium.com/pytorch/get-started-with-pytorch-cloud-tpus-and-colab-a24757b8f7fc
It seems quite straightforward and doesn't require change to tianshou itself.

[QingyuanWang]

I think you can always use tianshou to rollout env and collect samples, and then train the network on pytorch compiled with TPU support: https://medium.com/pytorch/get-started-with-pytorch-cloud-tpus-and-colab-a24757b8f7fc
It seems quite straightforward and doesn't require change to tianshou itself.

Hi, it is not difficult to use pytorch on a single TPU core. However, a TPU has 8 cores with 8gb vram each and that is one important feature where TPU outperforms GPU. What I hope is tianshou will support the distributed training with all 8 cores. Not very sure whether if that will be different from distributed training on GPU.

[duburcqa]

@Trinkle23897 I don't think keeping this issue open is relevant.