More complex reward mechanism

[davide221]

I'm experimenting with chain of thoughts and finetuning, is there a way to use this library to assign different reward for each steps towards the problem resolution of the LLM? Like the new technique proposed in the step by step paper from OpenAI.

Would be amazing to experiment with it

[lvwerra]

Currently only a reward can be given to the last token and it discounted towards the beginning. The KL-div however is added per token and you could adapt the compute_rewards method to accept lists of scores for each sample.

[vwxyzjn]

This could be related to #424 and #429, which allow users to inject rewards mid-trajectory.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.