You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Can we register and push data to s3 in parallel in order to reduce overall produce latency?
Consider failure modes:
If the reg fails and the push succeeds we just retry the reg until it works
If the push fails and the reg succeeds…. we will retry the push but if agent crashes we will end up with a table entry in the LSM but no corresponding table. when a query occurs against the LSM and selects that table entry, the iterator will try and get the table and fail - we can just ignore it.
So. possibly we can register and push to S3 in parallel. Needs more investigation.
The text was updated successfully, but these errors were encountered:
Can we register and push data to s3 in parallel in order to reduce overall produce latency?
Consider failure modes:
If the reg fails and the push succeeds we just retry the reg until it works
If the push fails and the reg succeeds…. we will retry the push but if agent crashes we will end up with a table entry in the LSM but no corresponding table. when a query occurs against the LSM and selects that table entry, the iterator will try and get the table and fail - we can just ignore it.
So. possibly we can register and push to S3 in parallel. Needs more investigation.
The text was updated successfully, but these errors were encountered: