Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Periodically reload instance properties in state store committer #4173

Open
patchwork01 opened this issue Jan 30, 2025 · 0 comments
Open

Periodically reload instance properties in state store committer #4173

patchwork01 opened this issue Jan 30, 2025 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@patchwork01
Copy link
Collaborator

patchwork01 commented Jan 30, 2025

Analysis

S3PropertiesReloader already applies some configuration for whether to reload properties. We can add an instance property to reload periodically and default that to e.g. every 5 minutes.

We'll need to add PropertiesReloader into StateStoreCommitter, and we could avoid letting you force reload the properties there. We could have a separate static constructor for S3PropertiesReloader for the committer, similar to in StateStoreFactory.forCommitterProcess.

We could also consider how to apply changes to the table properties, but there's a problem as it is in that the state store objects hold the state of the Sleeper tables in memory and don't retain a reference to the table properties. The table properties are looked up once when the state store is created, and then not checked again. We could deal with this as a separate issue.

@patchwork01 patchwork01 added the enhancement New feature or request label Jan 30, 2025
@patchwork01 patchwork01 added this to the 0.28.0 milestone Jan 30, 2025
@patchwork01 patchwork01 self-assigned this Jan 30, 2025
@patchwork01 patchwork01 modified the milestones: 0.28.0, 0.29.0 Jan 30, 2025
@patchwork01 patchwork01 removed their assignment Jan 30, 2025
@gaffer01 gaffer01 modified the milestones: 0.29.0, 0.30.0 Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants