Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel request policy for DeltaManager / move to streaming API #5523

Closed
vladsud opened this issue Mar 14, 2021 · 1 comment
Closed

Parallel request policy for DeltaManager / move to streaming API #5523

vladsud opened this issue Mar 14, 2021 · 1 comment
Assignees
@vladsud
Labels
area: driver Driver related issues perf
Milestone
April 2021

Comments

@vladsud
Copy link
Contributor

vladsud commented Mar 14, 2021

Issue #5211 was looking into the issue of op catch up performance.
It has a lot of data, but as result of it, I've started implemented concurrent op retrieval

#5393 issue was tracking first part of it - implementation of capability, but also a policy of 4 concurrent requests in fetch-tool, with no change in DeltaManager.

This issue tracks future impartments in this space, including

  • moving code into driver-utils for drivers to be able to leverage it
  • changing driver interface to be streaming interface and not using batches at that API boundary
  • coming up with actual policy for DeltaManager / OPSP driver.
@vladsud vladsud added the bug Something isn't working label Mar 14, 2021
@vladsud vladsud added this to the April 2021 milestone Mar 14, 2021
@vladsud vladsud self-assigned this Mar 14, 2021
@ghost ghost added the triage label Mar 14, 2021
@vladsud vladsud mentioned this issue Mar 14, 2021
Closed
vladsud added a commit that referenced this issue Mar 14, 2021
@vladsud
Implement parallel ops requesting (#5407)
c9cfcd1 
Implements and closes #5393

Implement ability to to do multiple parallel requests to storage to speed up retrieval of ops.
This change implements ability to do concurrent fetches. It does not change behaviour of DeltaManager for now, and only enables this new capability in fetch-tool with 4 parallel requests by 20K ops.
Future work in this area is tracked by issue #5523.

This new capability will help in following scenarios:
- Boot from cached (stale) snapshot (in future)
- Being offline for a day - week (in future)
- fetching a lot of ops using fetch-tool (in this PR)

IN future, I'll refactor this approach into an adapter layer that drivers could use to implement parallel requests (or not), as well as their batching strategy.
An API that will be exposed from driver will be an IReadPipe API and full (including infinite) request, where DeltaManager no longer would control batching or parallelism.
It's easy to start with no parallelism and add concurrency (workers) later. We need to explore that such that we start with one request and add more parallel requests only when we know we are far behind. That will reduce pressure on storage (extra empty requests for nothing).
@curtisman curtisman added perf area: driver Driver related issues and removed bug Something isn't working triage labels Mar 15, 2021
@vladsud
Copy link
Contributor Author

vladsud commented Apr 8, 2021

Resolving as fixed via #5703.
All of the work is done with exception of actual parallel fetch logic - that will happen a bit later.

@vladsud vladsud closed this as completed Apr 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vladsud vladsud

Labels
area: driver Driver related issues perf
Projects
None yet
Milestone
April 2021
Development

No branches or pull requests
2 participants
@curtisman @vladsud