Exposing GenerationConfig in the GRPO Trainer

[Superskyyy]

Feature request

Often people need to customize the generation config, now it's embedded in the training loop. Should be easy to extract it out.

Motivation

Customization

Your contribution

I can help to contribute.

[qgallouedec]

More control on the generation does make sense. A reasonable way to allow for more control is probably to add more generation args in the GRPOConfig.
Are you willing to contribute?

[Superskyyy]

More control on the generation does make sense. A reasonable way to allow for more control is probably to add more generation args in the GRPOConfig. Are you willing to contribute?

Yes I will contribute this feature.

[Benjoyo]

More control on the generation does make sense. A reasonable way to allow for more control is probably to add more generation args in the GRPOConfig. Are you willing to contribute?

Yes I will contribute this feature.

Please add stop_strings or stopping criteria :)

Although I don’t see why not exposing the full generation config to avoid the next issue of this type in a few weeks.

[Superskyyy]

More control on the generation does make sense. A reasonable way to allow for more control is probably to add more generation args in the GRPOConfig. Are you willing to contribute?

Yes I will contribute this feature.

Please add stop_strings or stopping criteria :)

Although I don’t see why not exposing the full generation config to avoid the next issue of this type in a few weeks.

@qgallouedec wdyt? Either we should just directly expose the entire generation config because there are all kinds of tricks that people might want to tune there.

[xzuyn]

Would love to see this in Online DPO as well. Currently it's hard-coded to do_sample=True, top_k=50, top_p=1.

[Superskyyy]

Since we are doing vLLM now, this might need some extra design in it. Either we expose one of them (HF preferrablly) and convert automatically into vLLM config. Or we allow two mutually exclusive configs being supplied. But I guess explicit is better than implicit.