Synthetic preference dataset

[vwxyzjn]

Useful to reuse our existing rejection sampling infra and build on top of it to create synthetic preference dataset

# 1. first sample a bunch of completions given prompts
# Here is an example created dataset: https://huggingface.co/datasets/vwxyzjn/generation_1725567768
python open_instruct/rejection_sampling/generation.py \
    --dataset_name HuggingFaceH4/no_robots \
    --model_name_or_path allenai/llama-3-tulu-2-8b \
    --num_completions 3 \
    --save_filename output/completions.jsonl \
    --sanity_check \
    --push_to_hub

Create preference pairs

# 2.1 do LLM as a judge to create synthetic preference dataset
# Here is an example created dataset: https://huggingface.co/datasets/vwxyzjn/synthetic_preference_dataset_1725567862
python open_instruct/rejection_sampling/synthetic_preference_dataset.py \
    --input_filename output/completions.jsonl \
    --model gpt-4o-2024-08-06 \
    --save_filename output/synthetic_preferences.jsonl \
    --num_completions 3 \
    --push_to_hub \

Multiple prompt pairs as well.

[ValentinaPy]

Very cool! I can run it over the WildChat prompts!

[nouhadziri]

This is pretty cool Costa! thanks for adding this. LGTM