Roadmap - trl 0.2

[younesbelkada]

A list of cool things that we can aim for trl 0.2! :

API:

• general decoder models support (OPT, BLOOM, GPT-J, etc): xxxForCausalLM support #53
• accelerate integration for training in mixed precision, DP (multi-GPU), using DeepSpeed: accelerate integration #58
• shared backbones for lighter training: add create_reference_model #61
• Once ^ is merged (or inside the PR) - make the reference model optional and create both models inside the trainer if no reference model is passed | [PPOTrainer] make the reference model optional #67
• soften requirements : Relax requirements #66
• Make quality & make style: Improve Makefile and code quality #62
• use accelerator logger instead of wandb to remove hard dependency (for users that do not want to use wandb) | [core] remove wandb dependency #92
• Refactor Trainer (push to hub etc. cc @lewtun) Add push to Hub for PPOTrainer #68
• general encoder-decoder models support | Encoder-Decoder models support #93
• Support for Adam 8bit (for lighter training)? | [PPOTrainer] Support generic optimizers #78
• clean step (sanity checks) | [core] refactor step method #76
• dataset attribute should be optional ? | [API] Make dataset attribute optional #85
• Add learning rate scheduler support | [API] LR scheduler support #96
• Handle: Add push to Hub for PPOTrainer #68 (comment) | [core] Push v_head when using AutoModelForCausalLMWithValueHead #86

Documentation

• doc build: add docbuilder skeleton #59
• document API (classes, functions) | [Doc] Improve docs #91
• document tips and good practices when doing RLHF (hyper-parameters etc.) / https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
• clean up README (examples don't work anymore) Improvements 1a #70

Improvements

• add different example than fine-tuning gpt2 on sentiment analysis
• add an example with a custom reward model (SetFit?)
• convert the notebook 05 Convert notebook 05 #80
• What is the largest model we can train on?
• What is the coolest dataset we can use to train a model (e.g. anthropic AI dataset: https://huggingface.co/datasets/Anthropic/hh-rlhf ?
• Make all internal imports relative (e.g. from trl.trainer -> from .trainer) Improvements 1a #70
• Clean up setup.py and remove settings.ini (legacy from nbdev) Improvements 1a #70
• Create & add a logo | add logo #95

Any suggestion very welcome!

[LouisCastricato]

BTW, I can confirm that SetFit does make for a really good zero shot RM. There are some issues with using contrastive models as RMs though. It often requires very careful data cleaning and identifying what kinds of clusters work as RMs is a dark art to the point where we decided that it wasn't worth seriously pursing further after CARP CoOp. Rerank models are much better.

[TristanThrush]

I think that the "coolest" dataset we can use to train a model could be https://huggingface.co/datasets/openai/webgpt_comparisons, but it is hard to evaluate this sort of model after we train it. I might start by adding a summarization example, and then some decent ways by which it can be evaluated. Then the webgpt comparisons example

[AlexWortega]

https://colab.research.google.com/drive/1hkPBFtMP5xBAjNYMjWH7NqYn118kRLOJ?usp=sharing
I am trying to implement own gpt + trl with QA retrival reward, but i think something is wrong with reward/or generation

[natolambert]

@AlexWortega can you open a separate issue / PR for this? Looks interesting, but may get loss in this big 1.0 roadmap thread.

[lvwerra]

We ended up calling this release 0.2 (not 1.0). I am closing the issue and will move the open tasks to a new issue.

[AlexWortega]

Hi @lvwerra i opened PR #149 with this feature(?) idea