🃏 Model card for TRL

[qgallouedec]

What does this PR do?

Having our own model card.

Demo

from datasets import load_dataset
from trl import DPOConfig, DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
dataset_name = "trl-lib/tldr-preference"
train_dataset = load_dataset(dataset_name, split="train").select(range(50))
args = DPOConfig(output_dir="dpo-qwen2", logging_steps=10, push_to_hub=True, save_strategy="epoch", num_train_epochs=1)
trainer = DPOTrainer(model=model, args=args, tokenizer=tokenizer, train_dataset=train_dataset)
trainer.train()

if args.push_to_hub:
    trainer.push_to_hub(dataset_name=dataset_name)

result: https://huggingface.co/qgallouedec/dpo-qwen2

It adds

Link to the paper

Link to the dataset

TRL own model card

Other

• Citations
• Links to wandb run

TODO

• AlignPropTrainer
• BCOTrainer
• CPOTrainer
• DPOTrainer
• GKDTrainer
• IterativeSFTTrainer
• KTOTrainer
• NashMDTrainer
• OnlineDPOTrainer
• ORPOTrainer
• PPOv2Trainer
• RewardTrainer
• RLOOTrainer
• SFTTrainer
• XPOTrainer

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lewtun]

Amazing work on the model cards @qgallouedec - now they're really packed with useful information 🔥 !!

LGTM with a tweak to the example inference code

[lewtun]

Note to self: we should move these datasets that aren't strictly used for tests to trl-lib

[qgallouedec]

Note that the code demo for diffusion models and VLM will be wrong but we can probably keep it like that for now

[valayDave]

When will this commit be released ? Will it be a part of v0.12 ?