[#24789][prism] Handlers for combine, ParDo, GBK, Flatten #25558
Conversation
Codecov Report
@@ Coverage Diff @@
## master #25558 +/- ##
==========================================
- Coverage 72.64% 72.33% -0.32%
==========================================
Files 763 766 +3
Lines 101060 101494 +434
==========================================
Hits 73413 73413
- Misses 26226 26660 +434
Partials 1421 1421
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
This is where things get interesting! This converts/prepares the Pipeline Proto representations to fully expanded alternatives and similar. You'll probably find the comments on how SDFs and Combine composites are expanded interesting. GroupByKeys and Flattens are pretty straight foward so far and do the appropriate in memory work. That would change depending on how data services evolve (eg. Stop being strictly in memory). But as it ism this is an OK abstraction to start. The code living in an internal package gives us the freedom to evolve the abstractions we're using as needed. |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
lostluck
changed the title
|
LGTM - but I've only been skimming the code here [ and https://github.com//pull/25556 and https://github.com//pull/25557 ... and your other related/previous PRs ]. Fantastic contribution, so glad you've gotten it this far already! |
| SDKGBK bool // Sets whether the GBK should be handled by the SDK, if possible by the SDK. | ||
| } | ||
|
|
||
| func Runner(config any) *runner { |
There was a problem hiding this comment.
The Combine(), ParDo() and Runner() functions are exported with an unexported return type, but you might have an intention with this?
There was a problem hiding this comment.
In a public API, returning unexported types is annoying for users, but it's less so under the isolation of the internal folder. It's likely the handlers will end up in their own package, and get exported out that way, with the types changed as appropriate.
While I've got the interfaces (preparer, executer) they're there to reduce the chance the abstraction will be broken. They'll be refined as other features are added to Prism, like Fusion, and handling the TestStream, and PubSubm URNs (and more).
…he#25558) * [prism] add in handlers * [prism] executor interface * delint - rm processor * [prism] comments
…he#25558) * [prism] add in handlers * [prism] executor interface * delint - rm processor * [prism] comments
Add in the handlers for Combine, ParDo, GBK, Flatten
Adds in an executor interface.
No unit tests largely because these are largely rote Proto transformations, and are best tested in the pipeline context. They'll fully covered in the completed internal package.
See #24789 for more information.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.