FuseChat-3.0: Preference Optimization for Implicit Model Fusion

| 📑 Paper | 🐱 GitHub Repo | 🤗 HuggingFace Repo | 🌐 Website |

Ziyi Yang*, Fanqi Wan*, Longguang Zhong*, Canbin Huang, Guosheng Liang, Xiaojun Quan†

Sun Yat-sen University

*Main Contributors †Corresponding Author

We present FuseChat-3.0, a series of models crafted to enhance performance by integrating the strengths of multiple source LLMs into more compact target LLMs. To achieve this fusion, we utilized four powerful source LLMs: Gemma-2-27B-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For the target LLMs, we employed three widely-used smaller models—Llama-3.1-8B-Instruct, Gemma-2-9B-It, and Qwen-2.5-7B-Instruct—along with two even more compact models—Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. The implicit model fusion process involves a two-stage training pipeline comprising Supervised Fine-Tuning (SFT) to mitigate distribution discrepancies between target and source LLMs, and Direct Preference Optimization (DPO) for learning preferences from multiple source LLMs. The resulting FuseChat-3.0 models demonstrated substantial improvements in tasks related to general conversation, instruction following, mathematics, and coding. Notably, when Llama-3.1-8B-Instruct served as the target LLM, our fusion approach achieved an average improvement of 6.8 points across 14 benchmarks. Moreover, it showed significant improvements of 37.1 and 30.1 points on instruction-following test sets AlpacaEval-2 and Arena-Hard respectively. We have released the FuseChat-3.0 models on Huggingface, stay tuned for the forthcoming dataset and code.

Overview

Combining the strengths of multiple large language models (LLMs) represents a promising approach to enhance individual model capabilities. Model fusion is a technique that integrates the strengths of robust source LLMs into a target LLM.

Previous iterations of the FuseChat series employed probabilistic distribution matrices generated by source models to transfer knowledge to target models. We refer to this method as explicit model fusion (EMF) because it involves a well-defined knowledge transfer process. While applicable to models with varying architectures and sizes, and without increasing memory overhead during inference, this approach presents notable challenges such as vocabulary alignment and the merging of distribution matrices from different LLMs. These issues complicate model fusion, reduce its efficiency, and may introduce noise and errors and affect the fusion results.

FuseChat-3.0, however, takes a different approach by enhancing a single LLM through implicit learning from robust open-source LLMs, a process we term implicit model fusion (IMF). The concept of IMF has been widely utilized to improve the performance of weaker models. For instance, a weak model can be boosted through fine-tuning with outputs from stronger LLMs. Moreover, a reward model can be trained using outputs from various LLMs, enabling it to learn and capture the differences in capabilities between the LLMs. Zephyr further collects responses from multiple LLMs and ranks them with GPT-4 to obtain preference data for training the policy. Inspired by recent alignment techniques, we propose an IMF method to transfer the capabilities of source LLMs to a target LLM through preference optimization.

Our IMF method follows a three-stage process aimed at effectively transferring capabilities from source LLMs to a target LLM. First, during dataset construction, we sample N responses from each of the source LLMs and annotate these responses using an external reward model. Second, in the supervised fine-tuning (SFT) stage, we fine-tune the target model using the best responses, which not only enhances the target model's capabilities but also helps mitigate the distributional gap between the source and target models. Finally, in the direct preference optimization (DPO) stage, we optimize the target model by using the best and worst responses from the source models as preference pairs, further enhancing the target model's performance. The complete pipeline will be detailed in the following paragraph.

Dataset

Prompt Selection

Our datasets were designed to enhance model's instruction following, general conversation, mathematics, coding, and Chinese-language capabilities. We selected data from open-source community datasets, applying targeted filtering and preprocessing. Key datasets and filtering criteria included:

• Instruction Following & General Conversation: Sourced from UltraFeedback, Magpie-Pro-DPO-100K-v0.1, and HelpSteer2, excluding code and math data.
• Mathematics: Selected from OpenMathInstruct-2, with nearly 60,000 unique samples.
• Coding: Curated from leetcode and self-oss-instruct-sc2-exec-filter-50k, retaining prompts with test cases.
• Chinese Language: Integrated alpaca_gpt4_zh and Magpie-Qwen2-Pro-200K-Chinese, filtering out code and math prompts to retain approximately 10,000 high-quality samples.

Response Sampling

For each dataset's prompts, we synthesized responses mainly from four different series of source models, specifically Gemma-2-27b-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. Compared to the original approach in WRPO using up to ten source LLMs, this streamlined selection ensures both representativeness and accessibility of the experiments.

• Instruction Following & General Conversation: We sampled each prompt five times from all the source models.
• Mathematics: We retained the responses generated by Llama-3.1-405B-Instruct from the original dataset (OpenMathInstruct-2) and additionally sampled responses using Qwen-2.5-Math-72B-Instruct.
• Coding: We sampled each prompt eight times for all source models.
• Chinese Language: We included single response sampled exclusively from Qwen-2.5-72B-Instruct.

The sampling parameters for different models are detailed in Table below.

Source LLMs	Sampling Params
Gemma-2-27b-It	Temp 0.8 Top-p 0.95
Mistral-Large-Instruct-2407	Temp 0.8 Top-p 0.95
Qwen-2.5-(Math)-72B-Instruct	Temp 0.7 Top-p 0.8 Repetition penalty 1.05
Llama-3.1-70B-Instruct	Temp 0.8 Top-p 0.95

Data Construction

Unlike the original approach in WRPO, which constructs preference pairs from target model responses and treats source model responses as additional positive samples, our research in mathematics and coding domains revealed that sampling from multiple source models yields more and higher-quality preference pair data. Based on this insight, FuseChat-3.0 leverages the best and worst response pairs generated by source models as preference pairs to optimize the target model. This refined approach not only preserves the core advantages of implicit model fusion but also results in a more streamlined and practical implementation, making it particularly well-suited for real-world applications within the open-source community.

• Instruction Following: To assign RM scores to the five responses generated by each source model, we employed ArmoRM for annotation. We then divided the annotated data into SFT and DPO datasets using a 4:6 ratio. For the SFT phase, we selected the responses with the highest RM scores. During the DPO phase, we paired responses from the same source model, designating those with the highest RM scores as positive samples and those with the lowest RM scores as negative samples. We ensured that the RM score difference between the positive and negative samples in each pair ranged from 0.01 to 0.1.
• Mathematics: We initially annotated the responses from all source models for correctness by comparing them with the gold labels and evaluating them using the RM scores provided by ArmoRM. We then strategically divided the dataset into SFT phase and DPO phase. In the SFT phase, we incorporated responses that were correct and had the highest RM scores. This selection ensured that the fine-tuning process was based on high-quality responses that aligned closely with the desired outcomes. For the DPO phase, we constructed paired samples from the same source model. The positive samples consisted of correct answers with the highest RM scores, while the negative samples were incorrect answers with the lowest RM scores. To ensure meaningful comparisons during optimization, we maintained an RM score differential between positive and negative pairs within the range of 0.01 to 0.1.
• Coding: We employed a dual-scoring system comprising correctness scores and RM scores for coding evaluation. The correctness scores assessed whether the code passed both static analysis and test cases, ensuring functional accuracy. The RM scores were used for preference evaluation, gauging the quality of responses based on predefined criteria. During the SFT phase, we included responses that not only passed all test cases but also achieved the highest RM scores. This selection ensured that the model was fine-tuned on exemplary code that met both correctness and preference standards. In the DPO phase, we contrasted positive samples—high-scoring responses that passed the tests—with negative samples—low-scoring responses that failed the tests. This comparison aimed to optimize the model's ability to prefer higher-quality code during training. We excluded any instances where all model responses failed to meet the testing criteria. This exclusion was necessary to maintain the integrity of the evaluation process, as such cases did not provide meaningful data for assessing and improving the model's performance.
• Chinese: We exclusively utilized responses sampled from Qwen-2.5-72B-Instruct during the SFT phase, due to its strong performance in the Chinese language.

Our final dataset comprised 158,784 total entries, with 94,539 entries for the SFT phase and 64,245 preference pairs for the DPO phase. The overall composition of the datasets is shown below.

Dataset	Total Count	SFT Count	DPO Count	Category
UltraFeedback	51098	20439	30659	Instruction following
Magpie-Pro-DPO	20374	8149	12225	Instruction following
HelpSteer2	9435	3774	5661	Instruction following
OpenMathInstruct-2	58546	40188	11615	Mathematics
leetcode	3113	1877	1236	Coding
self-oss-instruct-sc2	13696	10160	2849	Coding
alpaca_gpt4_zh	2471	2471	0	Chinese Language
Magpie-Qwen2-Pro	7481	7481	0	Chinese Language
Total	158784	94539	64245	All

Training

The implicit model fusion process involves a two-stage training pipeline comprising Supervised Fine-Tuning (SFT) to mitigate distribution discrepancies between target and source LLMs, and Direct Preference Optimization (DPO) for learning preferences from multiple source LLMs.

SFT

We used Llama-Factory as our fine-tuning library. For all target models, we fine-tuned for 3 epochs, with a batch size of 128 and a maximum sequence length of 2048 tokens. A cosine learning rate schedule with a warmup ratio of 0.1 is employed. Different models' learning rates are shown in the table below.

Target Models	Learning rate
Llama-3.1-8B-Instruct	5e-6
Qwen-2.5-7B-Instruct	2e-6
Gemma-2-9B-It	2e-6
Llama-3.2-(1/3)B-Instruct	5e-6

DPO

We used alignment-handbook as our DPO training library. For all Target SFT models, we trained for 1 epoch with a batch size of 128, set maximum sequence length to 2048, used cosine learning rate with a warmup ratio of 0.1. We saved checkpoints every 100 steps and selected the best from the last two checkpoints. For Llama-3.1 and Llama-3.2 series models, we introduced length normalization in DPO training, as shown in the formula below.

Different models' hyperparameters are shown in the table below.

Target SFT Models	Learning rate	β	Length normalize
FuseChat-Llama-3.1-8B-SFT	8e-7	10	Yes
FuseChat-Qwen-2.5-7B-SFT	3e-7	0.01	No
FuseChat-Gemma-2-9B-SFT	5e-7	0.01	No
FuseChat-Llama-3.2-(1/3)B-SFT	1e-6	10	Yes

Evaluation

The evaluation of instruction-tuned models mainly focuses on the model performance of instruction following, natural language understanding, general question answering, reasoning, mathematics, coding, etc. For the evaluation of FuseChat-3.0, we include 14 benchmarks and organize them into four categories:

• Instruction Following Tasks: AlpacaEval-2, Arena-Hard, MTbench, AlignBench v1.1 (Chinese).
• General Tasks: LiveBench-0831, MMLU-Pro, MMLU-redux, GPQA-Diamond.
• Mathematics Tasks: GSM8K, MATH, AMC 23.
• Coding Tasks: HumanEval, MBPP, LiveCodeBench 2408-2411.

We include more details and release our evaluation code at FuseEval.

The evaluation results of five series fused models are as follows, showing that our FuseChat-3.0 models achieved varying degrees of improvement across different target models. When selecting Llama-3.1-8B-Instruct as the target model, our fusion model FuseChat-Llama-3.1-8B-Instruct achieved an average performance improvement of 6.8 points across 14 benchmarks. Notably, it showed significant improvements of 37.1 and 30.1 points on instruction-following test sets AlpacaEval-2 and Arena-Hard respectively. Additionally, FuseChat-Llama-3.1-8B-Instruct outperformed AllenAI's recently released Llama-3.1-Tulu-3-8B model on all benchmarks except GSM8K and GPQA-Diamond. All these results demonstrate the effectiveness and success of FuseChat-3.0.

FuseChat-Llama-3.1-8B-Instruct Performance

Benchmarks	Llama-3.1-8B-Instruct	Llama-3.1-Tulu-3-8B	FuseChat-Llama-3.1-8B-SFT	FuseChat-Llama-3.1-8B-Instruct
AlpacaEval-2 (LC %)	28.3	33.4	41.3	65.4
Arena-Hard (WR %)	28.1	45.6	38.7	58.2
MT-Bench	8.4	8.3	8.5	9.0
AlignBench v1.1	4.6	6.2	6.3	6.7
GSM8K	85.9	88.6	87.0	88.0
MATH	50.7	47.5	54.7	55.2
AMC 23	25.0	25.0	30.0	37.5
LiveBench 0831	27.6	30.1	30.2	32.0
MMLU-Pro	50.0	42.9	47.8	49.2
MMLU-redux	67.2	66.3	68.4	69.2
GPQA-Diamond	33.8	35.9	37.9	34.9
HumanEval	69.5	66.5	69.5	71.3
MBPP	75.4	56.3	71.4	72.0
LiveCodeBench 2408-2411	12.3	10.6	12.6	13.1
Average	40.5	40.2	43.2	47.3

FuseChat-Qwen-2.5-7B-Instruct Performance

Datasets	Qwen-2.5-7B-Instruct	FuseChat-Qwen-2.5-7B-SFT	FuseChat-Qwen-2.5-7B-Instruct
AlpacaEval-2 (LC %)	33.2	34.2	63.6
Arena-Hard (WR %)	50.7	45.2	61.4
MT-Bench	8.4	8.5	9.0
AlignBench v1.1	7.5	7.4	7.6
GSM8K	91.7	92.3	91.7
MATH	75.0	72.7	73.6
AMC 23	52.5	45.0	57.5
LiveBench-0831	35.4	33.7	33.2
MMLU-Pro	54.1	51.7	53.0
MMLU-redux	75.1	72.7	74.4
GPQA-Diamond	34.9	38.4	33.8
HumanEval	85.4	81.7	79.9
MBPP	80.2	84.1	83.1
LiveCodeBench 2408-2411	15.8	17.3	18.9
Average	50.0	48.9	52.9

FuseChat-Gemma-2-9B-Instruct Performance

Datasets	Gemma-2-9B-It	FuseChat-Gemma-2-9B-SFT	FuseChat-Gemma-2-9B-Instruct
AlpacaEval-2 (LC %)	51.1	49.8	70.2
Arena-Hard (WR %)	40.8	44.5	63.4
MT-Bench	8.5	8.7	8.6
AlignBench v1.1	7.0	7.1	7.4
GSM8K	88.5	90.5	91.0
MATH	49.6	58.0	57.8
AMC 23	20.0	27.5	35.0
LiveBench-0831	31.6	33.3	33.2
MMLU-Pro	50.5	52.5	52.9
MMLU-redux	72.8	72.8	73.7
GPQA-Diamond	39.4	33.3	35.4
HumanEval	67.1	65.9	64.0
MBPP	75.1	70.6	71.7
LiveCodeBench 2408-2411	11.9	11.0	10.1
Average	43.9	44.7	48.2

FuseChat-Llama-3.2-3B-Instruct Performance

Benchmarks	Llama-3.2-3B-Instruct	FuseChat-Llama-3.2-3B-SFT	FuseChat-Llama-3.2-3B-Instruct
AlpacaEval-2 (LC %)	21.4	31.1	54.0
Arena-Hard (WR %)	16.6	21.3	30.2
MT-Bench	6.9	7.3	7.7
AlignBench v1.1	3.8	5.5	5.9
GSM8K	82.0	82.8	82.0
MATH	51.4	52.9	53.1
AMC23	22.5	20.0	35.0
LiveBench 0831	23.4	24.5	24.9
MMLU-Pro	39.3	40.3	40.3
MMLU-redux	58.5	58.2	59.0
GPQA-Diamond	29.8	33.3	33.8
HumanEval	61.0	62.8	60.4
MBPP	68.5	67.5	67.5
LiveCodeBench 2408-2411	8.3	7.1	9.0
Average	35.2	36.8	40.2

FuseChat-Llama-3.2-1B-Instruct Performance

Benchmarks	Llama-3.2-1B-Instruct	FuseChat-Llama-3.2-1B-SFT	FuseChat-Llama-3.2-1B-Instruct
AlpacaEval-2 (LC %)	9.7	14.0	25.3
Arena-Hard (WR %)	5.1	6.0	8.6
MT-Bench	4.7	5.2	5.7
AlignBench v1.1	2.9	3.9	4.3
GSM8K	46.3	55.6	54.5
MATH	32.7	34.7	33.6
AMC23	17.5	15.0	20.0
LiveBench 0831	14.0	13.9	15.8
MMLU-Pro	22.3	21.5	21.3
MMLU	45.8	45.0	44.8
GPQA-Diamond	21.2	25.3	24.2
HumanEval	39.6	36.6	40.2
MBPP	49.5	42.1	46.6
Average	24.0	24.5	26.5

BibTeX

@article{yang2024wrpo,
  title={Weighted-Reward Preference Optimization for Implicit Model Fusion},
  author={Ziyi Yang and Fanqi Wan and Longguang Zhong and Tianyuan Shi and Xiaojun Quan},
  journal={arXiv preprint arXiv:2412.03187},
  year={2024}
}