Tool fine-tuning support DPO

[August-murr]

What does this PR do?

adding tool support for function calling models in the DPOTrainer

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

@qgallouedec

[August-murr]

Before adding it to all the trainers, what do you think of the overall structure? Is it okay to include the tools in each trainer configuration?

[qgallouedec]

Thanks for this addition!

Let's keep things as separate as possible, and keep this PR for DPO only.

The code as is looks good to me. The only question is: can this type (Optional[list[Union[dict, Callable]]]) being parsed. I'll try.

[qgallouedec]

That's why I thought:

from trl import DPOConfig, TrlParser

parser = TrlParser((DPOConfig,))
parser.parse_args_and_config()

$ python 2479.py --output_dir out --tools "{'type': 'function', 'function': {'name': 'multiply', 'description': 'A function that multiplies two numbers', 'parameters': {'type': 'object', 'properties': {'a': {'type': 'number', 'description': 'The first number to multiply'}, 'b': {'type': 'number', 'description': 'The second number to multiply'}}, 'required': ['a', 'b']}}}"
[...]
2479.py: error: argument --tools: invalid Union value: "{'type': 'function', 'function': {'name': 'multiply', 'description': 'A function that multiplies two numbers', 'parameters': {'type': 'object', 'properties': {'a': {'type': 'number', 'description': 'The first number to multiply'}, 'b': {'type': 'number', 'description': 'The second number to multiply'}}, 'required': ['a', 'b']}}}"

I'm not sure what the best way to handle it right now, I'll sleep on it.

[August-murr]

Let's keep things as separate as possible, and keep this PR for DPO only.

a different PR for each trainer then?

can this type (Optional[list[Union[dict, Callable]]]) being parsed.

Adding tools to the CLI would be quite complicated. It wouldn't be practical to add all the tools into the CLI. My best guess is to read the functions from another source, like another script, if there’s a request for it later.

[August-murr]

does this need anything else? test or docs?

[August-murr]

I also wanted to add it to SFTTrainer but it doesn't use maybe_apply_chat_template