qwen2.5 增量预训练，上下文长度
#1000
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
-
qwen2.5上下文长度是128k，看官网技术报告，达到这个上下文长度是通过三个步骤实现的，先是在8k上下文训练，然后通过扩大上下文窗口和调ROPE base参数，最后通过yarn和一些train-free技术扩大到128k。那么，如果我要做增量预训练的话，上下文长度怎么设置，是否需要也按照这三个步骤训练
Beta Was this translation helpful? Give feedback.
All reactions