-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[prism] Triggers and Panes and Windowing Strategies #31438
Comments
lostluck
changed the title
[prism] Triggers and Panes
[prism] Triggers and Panes and Windowing Strategies
May 29, 2024
This was referenced May 30, 2024
Open
Merged
3 tasks
lostluck
added a commit
that referenced
this issue
Jan 9, 2025
3 tasks
3 tasks
lostluck
added a commit
that referenced
this issue
Feb 4, 2025
* [prism] Never mix Data and timers in the same bundle. No execution order guarantees. * Trigger State machine - no processing time - no merging. * comments * fix aftereach Handling to proceed only on finished * Add default + fix OrFinally * Split the isReady into onElement, shouldFire, and onFire methods. * Add string prints, adjust OnWindowExp behavior. * Add tests for unset AfterEndOfWindow subtriggers & fixes. --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Triggers are the "last" complex bit of modeling necessary for Prism's Table stakes for being incorporated for use by other SDKs.
Triggers are a way to refine how and when aggregations occur, but require sophisticated accounting Runner side to implement. https://beam.apache.org/documentation/programming-guide/#triggers has the user side details.
Triggers (and WindowingStrategy in general) contribute to the "fun" bit of the batch and stream unification that the Beam model provides. To help manage this, Panes are metadata passed along with the elements, and indicate the upstream aggregation conditions that produced the values.
https://docs.google.com/document/d/12r7frmxNickxB5tbpuEh_n35_IJeVZn1peOrBrhhP6Y/edit describes how Lateness and late data is handled WRT panes.
Ultimately, in prism, this will probably be handled via the state and timer system instead of an additional layer of separate logic, which has the cost of less efficient behavior for "only watermark" triggered aggregations. Robust State and Timers allow for all these.
The text was updated successfully, but these errors were encountered: