Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider whether we can we push metadata and s3 data in parallel. #295

Open
purplefox opened this issue Oct 3, 2024 · 0 comments
Open
Milestone

Comments

@purplefox
Copy link
Contributor

Can we register and push data to s3 in parallel in order to reduce overall produce latency?

Consider failure modes:

If the reg fails and the push succeeds we just retry the reg until it works

If the push fails and the reg succeeds…. we will retry the push but if agent crashes we will end up with a table entry in the LSM but no corresponding table. when a query occurs against the LSM and selects that table entry, the iterator will try and get the table and fail - we can just ignore it.

So. possibly we can register and push to S3 in parallel. Needs more investigation.

@purplefox purplefox added this to the 1.0 milestone Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant