Supporting RWKV (a RNN that can match transformer LM & zero-shot performance at 1B+ params)

[BlinkDL]

Hi guys. I am working on RWKV, which might be the only RNN (no attention!) that can match transformer LM & zero-shot performance at 1B+ params:
https://www.reddit.com/r/MachineLearning/comments/vzr6ie/r_rwkv3_scaling_rnn_to_15b_and_reach_transformer/

I am using some CUDA in my project too. Probably we can collaborate to promote RNN and scale it to 100B+ params :)

The RWKV discord: https://discord.gg/bDSBUMeFpc

Github: https://github.com/BlinkDL/RWKV-LM

CUDA stuff: https://github.com/BlinkDL/RWKV-CUDA