🪜 Stepwise supervision dataset type

[qgallouedec]

This PR aims to create a standard for stepwise data

Related: #2110

Chosen name: Stepwise supervision
Columns: "prompt", "completions", "labels"
Structure:

standard_example = {
    "prompt": "Beautiful is better than",
    "completions": [", let me think...", " ugly."],
    "labels": [False, True],
}

TODO

• Update dataset format doc
• Update internal testing dataset (zen)
• Create a script for having our getting started script (from PRM800K?)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lewtun]

@qgallouedec I think this PR is mixing two concepts together:

• Stepwise / process supervision: here one has a single prompt and N steps, where each step is assigned a score (not a preference label).
• Multi-step preferences: here one has a prompt with a tree of (typically) paired preferences, similar to the MCTS DPO paper.

For stepwise supervision, I think the dataset structure can be quite simple:

{
    "prompt": "What number is larger, 9.8 or 9.11?",
    "completions": ["Step 1 ...", "Step 2 ...", ...],
    "labels": [0.6, 0.9, ...], # Can also be bools
}

WDYT about splitting this PR so that we target stepwise supervision first and include multistep preferences only if we end up extending the DPO trainer in that direction?

I'm happy to open a separate PR for stepwise supervision if you prefer

[qgallouedec]

The main intention was to implement stepwise process supervision. The doc isn't probably clear enough. Feel free to recommend/directly change anything that made you think it was multi-step preferences.
Related: #2148 (comment)

[lewtun]

Thanks for working on a standard format for stepwise supervision @qgallouedec ! The main comment I have is whether this should be grouped with preference learning or not and whether we should bother with the conversational format for now since I've never seen this used in practice (everything is single-turn thus far)

[lewtun]

As mentioned above, I don't think this format applies to process supervision

[qgallouedec]

removed in fe98b5b

[lewtun]

If you agree with my comment about not treating this as preference learning, then we might want to move this to it's own row for stepwise supervision

[qgallouedec]

Done in fe98b5b

[lewtun]

Thanks for iterating @qgallouedec - this looks great now and the PRM dataset is a very useful starting point for the PRM PR @gaetanlop