Inquiry About SFT Warm-Up Data Ratio and Its Impact on RL Performance

[starHu123]

Dear Authors,

First, thank you for your excellent work! In the paper, you mentioned using a small amount of data for supervised fine-tuning (SFT) warm-up before the RL phase. Could you kindly clarify:

1. Data Scale: What is the specific data size used for the SFT warm-up phase?
2. Proportion to RL Data: What percentage does this warm-up data account for relative to the total data used in the RL training stage?
3. Impact Analysis: How does the ratio of SFT-to-RL data (whether higher or lower) affect the final RL evaluation metrics? Have you observed any notable patterns or thresholds in your experiments?

This clarification would greatly help readers understand the relationship between warm-up strategies and RL optimization.

Thank you for your time and insights!

[rb93dett]

Hi, thank you for your interest in AlphaDrive, to answer your questions,

1. As mentioned in paper (Sec 3.1), we use 30k SFT data for warm-up, and 80k data for RL (without planning reasoning annotation), the total training data size is 110k.
2. The proportion of SFT data to the total training data is 27.28% (30k / 110k).
3. Even a small amount of warm-up data (~10k) can effectively enhance planning accuracy after RL. With the current total amount of data, allocating around 20% to warm-up data is an effective ratio, and further increases bring only marginal gains. We believe the model's potential has not yet been fully realized, and that incorporating more high-quality warm-up and RL data will lead to further improvements.

[jiangyigithub]

“The dataset is divided into 110k clips for training and 10k clips for validation. As for reasoning, we sample 30k data from the training dataset to generate the planning reasoning process. All reported results are obtained by training on the training set and evaluating on the validation set.

As mentioned in paper (Sec 3.1), we use 30k SFT data for warm-up, and 80k data for RL (without planning reasoning annotation), the total training data size is 110k.”
您好，问下这是意思是带think标注的数据有30k 用来做SFT， SFT训好的模型再用 80K没有think标签的数据来做GRPO训练吗？所以提供的example数据是用来做SFT的吗？可以提供一个做GRPO的example数据吗？