diff --git a/docs/source/paper_index.md b/docs/source/paper_index.md index 662c2f034f..39f60de40b 100644 --- a/docs/source/paper_index.md +++ b/docs/source/paper_index.md @@ -479,6 +479,33 @@ training_args = DPOConfig( These parameters only appear in the [published version](https://aclanthology.org/2025.tacl-1.22.pdf) +## Kahneman–Tversky Optimization + +Papers relating to the [`KTOTrainer`] + +### KTO: Model Alignment as Prospect Theoretic Optimization + +**📜 Paper**: https://huggingface.co/papers/2402.01306 + +KTO derives an alignment objective from prospect theory and learns directly from **binary** human feedback (liked/disliked), matching or surpassing DPO-style methods while handling imbalanced/noisy signals well. +To reproduce the paper's setting, you can use the default configuration of [`KTOTrainer`]: + +```python +from trl import KTOConfig, KTOTrainer +from transformers import AutoModelForCausalLM, AutoTokenizer + +model = AutoModelForCausalLM.from_pretrained(model_id) +tokenizer = AutoTokenizer.from_pretrained(model_id) + +trainer = KTOTrainer( + model=model, + processing_class=tokenizer, + args=KTOConfig(), + train_dataset=..., +) +trainer.train() +``` + ## Supervised Fine-Tuning Papers relating to the [`SFTTrainer`]