Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#32115] Fix timer support, support timer clears. #32119

Merged
merged 7 commits into from
Aug 9, 2024

Conversation

lostluck
Copy link
Contributor

@lostluck lostluck commented Aug 8, 2024

  • Prism wasn't decoding all timers returned in the logical timer stream, dropping later set timers.
    • Moved timer decoding to be iterator compatible for the future, instead of trying to reprocess inplace, or returning the remaining byte buffer.
  • From there supporting clearing timers was simple, by additionally returning the window set, and deleting existing entries to the pending timer list.
  • Needed to re-write + override the timer clearing test, since for some reason the test DoFn would not execute when there were multiple timer parameters to the DoFn.

Allows a few of the Python timer based tests to pass.

TODO for a different test: Support interval and custom window functions for timers.

Fixes #32115


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@lostluck
Copy link
Contributor Author

lostluck commented Aug 8, 2024

R: @damondouglas

Copy link
Contributor

github-actions bot commented Aug 8, 2024

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

Copy link

codecov bot commented Aug 8, 2024

Codecov Report

Attention: Patch coverage is 70.49180% with 18 lines in your changes missing coverage. Please review.

Project coverage is 58.68%. Comparing base (adc3b2b) to head (33946a3).
Report is 6 commits behind head on master.

Files Patch % Lines
...o/pkg/beam/runners/prism/internal/engine/timers.go 71.42% 12 Missing and 2 partials ⚠️
...am/runners/prism/internal/engine/elementmanager.go 66.66% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             master   #32119   +/-   ##
=========================================
  Coverage     58.68%   58.68%           
  Complexity     3025     3025           
=========================================
  Files          1122     1122           
  Lines        172906   172923   +17     
  Branches       3285     3285           
=========================================
+ Hits         101463   101476   +13     
- Misses        68131    68134    +3     
- Partials       3312     3313    +1     
Flag Coverage Δ
go 34.63% <70.49%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@damondouglas damondouglas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I admit I didn't understand what a lot of this code was doing. From a reviewer perspective, I'm approving to unblock given there is at least a testing coverage somewhere.

sdks/go/pkg/beam/runners/prism/internal/engine/timers.go Outdated Show resolved Hide resolved
@lostluck lostluck merged commit 17298b5 into apache:master Aug 9, 2024
89 of 90 checks passed
@lostluck lostluck deleted the beam32115 branch August 9, 2024 21:21
lostluck added a commit to lostluck/beam that referenced this pull request Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[prism] Timer Clearing
2 participants