Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: piecereader: Allow parallel access #10895

Closed
wants to merge 4 commits into from

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented May 19, 2023

Related Issues

Closes #10532

Proposed Changes

  • Allow pieceReader reader-getters to define exact read size
  • Rewrite pieceReader.ReadAt to be called in parallel, while still caching small amounts ahead for parallel sequential reads
    • This is necessary, because carv1 reads byte-by-byte for the header/cid varints, saves ~9 requests per block read
  • Kept the old more aggressive prefetching mechanism for ReadSeeker, which is only used by the indexing scan, and makes it wayy faster

Additional Info

  • Make sure tests pass
  • Test on a real setup, best in booster-bitswap
    • With a race detector

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@magik6k magik6k requested a review from a team as a code owner May 19, 2023 16:52
@magik6k magik6k changed the title Feat/palallel piecereader feat: pierereader: Allow parallel access May 19, 2023
@magik6k magik6k changed the title feat: pierereader: Allow parallel access feat: piecereader: Allow parallel access May 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Re-tune piece-reader to allow for random parallel access of data
1 participant