python pipeline support a new feature, only receiving the frist arrival data from pre-ops #1380
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
python pipeline 新增特性：
channel支持从前置多个op中只拿到最快到达数据后，交给后置OP，其他数据丢弃。如： op C的前置op 是A和B，请求分别达到A,B后，由于A处理快，B处理慢，因此A先将结果发给op C的channel中。此时，如果config.yml中有如下设置
dag:
# 只接收最快到达的前置op数据，其他数据丢弃，默认为False
channel_recv_frist_arrive: True
使用场景在分布式MP节点上使用，因为MP节点之间有broadcast保证数据结果一样。