Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: erasure encoder #4429

Merged
merged 55 commits into from
Dec 4, 2023
Merged

feat: erasure encoder #4429

merged 55 commits into from
Dec 4, 2023

Conversation

nugaon
Copy link
Member

@nugaon nugaon commented Oct 26, 2023

  • NewHashTrieWriter is responsible for writing chunk tree -> we should add parity chunks for intermediate levels too
    • change children reference serialization according to rsParity
    • cache effective chunks and add RS parities.
    • encode used redundancy level into chunk span
  • created redundancy package in file for adding erasure coded data to files

Missing parts:

  • replace RS Parity parameter in requests with high level abstraction (4 levels),
  • span encoding intermediate level
  • add parity chunks to intermediate levels,
  • handle RS encoding when it is not a full chunk,
  • edge-case handling at encrypted chunks,
  • table
  • updated openAPI docs
  • add tests.

Checklist

  • I have read the coding guide.
  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have filled out the description and linked the related issues.

Description

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

@nugaon nugaon changed the base branch from master to feat/redundancy November 8, 2023 11:32
@nugaon nugaon force-pushed the feat/erasure-encoder branch from 6a4b79c to cb39e6b Compare November 8, 2023 11:42
@nugaon nugaon marked this pull request as ready for review November 8, 2023 11:42
@nugaon nugaon force-pushed the feat/erasure-encoder branch from cb39e6b to 5bb42e0 Compare November 8, 2023 11:47
@nugaon nugaon merged commit 31d0862 into feat/redundancy Dec 4, 2023
13 checks passed
@nugaon nugaon deleted the feat/erasure-encoder branch December 4, 2023 14:40
@nugaon nugaon mentioned this pull request Dec 4, 2023
7 tasks
nugaon added a commit that referenced this pull request Dec 4, 2023
nugaon added a commit that referenced this pull request Dec 18, 2023
nugaon added a commit that referenced this pull request Dec 20, 2023
nugaon added a commit that referenced this pull request Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant