Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client chunking default params #173

Closed
alanshaw opened this issue Jul 26, 2021 · 0 comments · Fixed by #170
Closed

Client chunking default params #173

alanshaw opened this issue Jul 26, 2021 · 0 comments · Fixed by #170
Assignees
Labels
kind/enhancement A net-new feature or improvement to an existing feature

Comments

@alanshaw
Copy link
Member

alanshaw commented Jul 26, 2021

We're considering a change to the default chunking parameters that will more closely reflect the defaults chosen by Lotus:

Current IPFS defaults:

  • Chunk size: 262144
  • Max links: 174

Lotus defaults:

  • Chunk size: 1048576
  • Max links: 1024

Why?

  • 🟢 Flatter DAGs
  • 🟢 Fewer things to announce to the DHT
  • 🟡 Same CIDs for the same content vs Lotus different hash functions (see below)
  • 🟡 Different CIDs for the same content vs IPFS (although IPFS still stuck in v0 so we have that problem anyway since we use v1)
  • 🔴 Less de-duping

UPDATE:

Thanks to @vasco we know that blake2js is way slower than sha256. It is not surprising because sha256 uses native webcrypto in the browser and Node.js crypto module. There is an npm module with native bindings for Node.js and a module with a WASM version of blake as well. The work to integrate and test this would be huge and there's no benchmark data for either module so it might not even be faster than the native sha256 versions. Unless there are objections I vote we stick with sha256 hashing for now but switch chunk size and max children.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement A net-new feature or improvement to an existing feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants