[dygraph hybrid pp for interleave] The interleave scheduler for pipeline parallel

[FeixLiu]

PR types

Others

PR changes

Others

Describe

Support interleave scheduler.
Main changes:

1. Interleave scheduler for both pipeline training and evaluation.
2. Under eager dygraph mode, use async comm methods (isend/irecv) instead of sync methods (send/recv).
3. Add two new methods _get_p2p_next_rank(), _get_p2p_prev_rank() in class HybridCommunicateGroup to get the global rank of prev pp stage and next pp stage.
4. In p2p communication, use the global rank to represent the prev pp stage and next pp stage instead of only use 0 or 1 to represent.

Note that: the interleave scheduler only supports eager dygraph mode.

All works for supporting interleave scheduler for pipeline:

• Support virtual pipeline layer spit ([dygraph hybrid pp for interleave] Virtual pp stage layer split #45402)
• Support virtual pipeline layer forward function (compatible with recompute) ([dygraph hybrid pp for interleave] Virtual pipeline layer forward function #45444)
• Implement interleave scheduler in pipeline_parallel ([dygraph hybrid pp for interleave] The interleave scheduler for pipeline parallel #45497)
• NCCLGroup start and NCCLGroup end. Implement save/load function for pipeline_parallel with virtual stage
• All necessary UT relating with recompute, AMP and etc.
• Some other functions but not sure yet

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[gongweibao]

I think we may split 1F1B pipeline and interleave pipe into two class.

[gongweibao]

Some comments.

[gongweibao]

LGTM

[sneaxiy]

LGTM

[sneaxiy]

Not quite important: self.output_tensor_grads is not necessary when forward_only = True.

[XieYunshen]

下个PR删除RUN_SERIAL 1参数