Windowing Support for the Dask Runner #23913
Conversation
- CoGroupByKey is broken due to how tags are used with GroupByKey
- GroupByKey should output `[('0', None), ('1', 1)]`, however it actually outputs: [(None, ('1', 1)), (None, ('0', None))]
- Once that is fixed, we may have test pipelines work on Dask.
…initial tests pass.
This reverts commit 09365f6.
…n infinite loop." This reverts commit 676d752.
|
Reminder, please take a look at this pr: @AnandInguva |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @pabloem for label python. Available commands:
|
|
Reminder, please take a look at this pr: @pabloem |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @damccorm for label python. Available commands:
|
|
stop reviewer notifications |
|
(pablo is already reviewing, this one should probably be sticky) |
|
Stopping reviewer notifications for this pull request: requested by reviewer |
|
@alxmrs how do we deal with this one? : ) |
|
Hey Pablo! I'm happy to take a look at these dask related issues in a week or two. I imagine that these changes have diverged from the upstream main branch since they were written. For a sooner timeline, do you have any capacity @cisaacstern or maybe @TomAugspurger? Charles, can you help me understand timelines for when Pangeo would like Dask support in Beam? |
|
Thanks for the ping, @alxmrs, and glad to see this progressing. I do not have extra bandwidth to look at this, unfortunately. Regarding Pangeo Forge timelines, for the next 6 months or so at least, and maybe a fair bit longer, our needs are covered by the Dataflow and Flink runners. |
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
This is currently as work-in-progress. To server near-term goals, the Dask runner needs to support side-inputs. In order to do this, Windowing needs to be supported. This CL adds basic Windowing support to this runner, including a few tests for side inputs.
Reviewers: @pabloem
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.