Add Storm-7B to AlpacaEval

[yifan123]

🤗 Card model: https://huggingface.co/jieliu/Storm-7B

[yifan123]

We also added Storm-7B (num_beams=10).

[YannDubs]

this is the right name, but it seems that you generated the outputs with a different name?

Both the leaderboard and the outputs were generated with a config named "Storm-7B (num_beams=10)" please change those manually or regenerate with the current config.

[yifan123]

I apologize for the typos. In the num_beams experiment, we use beam search, which slows down the generation process. To address this, we use our own repository that supports multi-GPU generation and then copy the results into this repository.

We have updated the commit for Storm-7B (num_beams=10).

Thanks for your review.

[YannDubs]

Why do you have GPT4 in the prompt?

[yifan123]

Thanks for your detailed review. Our model is finetuned from openchat-3.5-0106 which uses the chat template: GPT4 Correct User: {instruction}<|end_of_turn|>GPT4 Correct Assistant:. For example, if the question is How are you today?, we input GPT4 Correct User: How are you today?<|end_of_turn|>GPT4 Correct Assistant: into our model. It's just a template, similar to other chat templates. You can see the same chat template in another model in this repo.

[YannDubs]

Very impressive results @yifan123 !

I must say that I'm a shocked that a 7B gets that performance. What data was it trained on? What's the magic sauce to get such performance?

[yifan123]

We follow the same setup as in Starling-7B, using the same base model (openchat-3.5-0106) and fine-tuning on the same offline preference dataset (Nectar), without adding any additional datasets.

Our LM only differs in the optimization algorithms: apply DPO iteratively on newly generated responses ranked by the reward model (Starling-RM-34B). Note that Starling-RM-34B is also trained on Nectar by the K-wise maximum likelihood estimator, which means no extra data is used.

Our improvement stems mainly from the iterative training. The figure below shows that using only the first iteration's dataset in the second iteration results in minimal improvement.

We need to highlight the limitations of our study:

Storm-7B is a quick demonstration that a language model, fine-tuned with AI feedback, can easily surpass or match state-of-the-art models, as assessed by the same AI feedback. However, this improvement on the automatic leaderboard may not necessarily indicate better alignment with human intentions. Our model therefore represents a critical, preliminary reevaluation of the RLAIF paradigm, questioning how much learning from and being evaluated by AI feedback aligns with actual human preferences.