Skip to content

Commit 468b9d4

Browse files
docs: add KTO (2402.01306) to Paper Index + link ref to KTOTrainer (#4440)
Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
1 parent 9bc6206 commit 468b9d4

File tree

1 file changed

+27
-0
lines changed

1 file changed

+27
-0
lines changed

docs/source/paper_index.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -479,6 +479,33 @@ training_args = DPOConfig(
479479

480480
These parameters only appear in the [published version](https://aclanthology.org/2025.tacl-1.22.pdf)
481481

482+
## Kahneman–Tversky Optimization
483+
484+
Papers relating to the [`KTOTrainer`]
485+
486+
### KTO: Model Alignment as Prospect Theoretic Optimization
487+
488+
**📜 Paper**: https://huggingface.co/papers/2402.01306
489+
490+
KTO derives an alignment objective from prospect theory and learns directly from **binary** human feedback (liked/disliked), matching or surpassing DPO-style methods while handling imbalanced/noisy signals well.
491+
To reproduce the paper's setting, you can use the default configuration of [`KTOTrainer`]:
492+
493+
```python
494+
from trl import KTOConfig, KTOTrainer
495+
from transformers import AutoModelForCausalLM, AutoTokenizer
496+
497+
model = AutoModelForCausalLM.from_pretrained(model_id)
498+
tokenizer = AutoTokenizer.from_pretrained(model_id)
499+
500+
trainer = KTOTrainer(
501+
model=model,
502+
processing_class=tokenizer,
503+
args=KTOConfig(),
504+
train_dataset=...,
505+
)
506+
trainer.train()
507+
```
508+
482509
## Supervised Fine-Tuning
483510

484511
Papers relating to the [`SFTTrainer`]

0 commit comments

Comments
 (0)