Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: erasure encoder #4429

Merged
merged 55 commits into from
Dec 4, 2023
Merged
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
048fa87
build: add klauspost/reedsolomon library
nugaon Oct 26, 2023
709d3d8
feat: initial reed solomon encoding
nugaon Oct 26, 2023
6bc4930
feat: initial api for rsParity
nugaon Oct 26, 2023
700878e
test: change test calls for pipeline inits
nugaon Oct 26, 2023
6a52615
refactor: parity chunks does not need encryption
nugaon Oct 26, 2023
de5216d
fix: inc iterator
nugaon Oct 27, 2023
709ef06
feat: init redundancy
nugaon Oct 31, 2023
ca59060
feat: init rParams
nugaon Oct 31, 2023
1d56efa
fix: no need to encode rsParity to the span of data chunk
nugaon Oct 31, 2023
3d81f02
feat: add rLevel to hashTrieWriter
nugaon Oct 31, 2023
e20bd45
test: update
nugaon Oct 31, 2023
17eae4a
feat(api): add rlevel headers and params
nugaon Oct 31, 2023
0ca4994
refactor: level none and buffersize opt
nugaon Oct 31, 2023
cbf2c5a
refactor: erasure buffer
nugaon Oct 31, 2023
c807c14
fix: pass exact size buffer
nugaon Oct 31, 2023
c759ccb
fix: buffer size
nugaon Nov 1, 2023
10123a2
refactor: (style) separate encode logic
nugaon Nov 1, 2023
5b02576
fix: rparams chainwrite after writing the chunk to above level
nugaon Nov 1, 2023
bc55479
refactor: effective chunk metrics
nugaon Nov 2, 2023
e0ea985
feat: wrap up rightmost chunks and carrier chunk handling
nugaon Nov 3, 2023
fa9f6da
feat: encode parity
nugaon Nov 3, 2023
351d163
refactor: split package
nugaon Nov 3, 2023
41f0a61
feat: erasure table
nugaon Nov 3, 2023
975b504
chore: table is finally finished
nugaon Nov 3, 2023
2c14818
chore: remove unnecessary breakline
nugaon Nov 3, 2023
0899f77
fix: remove unnecessary encode
nugaon Nov 3, 2023
6a6f898
refactor: remove chunksize and branching from hashTrie params
nugaon Nov 6, 2023
15df1f5
refactor: mock rParams
nugaon Nov 6, 2023
4da4079
refactor: move span related functions
nugaon Nov 7, 2023
077eb9d
refactor: make erasuretable unexported
nugaon Nov 7, 2023
1c6abc4
refactor: make erasure encoder swappable
nugaon Nov 7, 2023
9460df7
fix: names of level function calls changed to unexported
nugaon Nov 7, 2023
7059bc7
test: redundancy unit test init
nugaon Nov 7, 2023
251d9da
refactor: move redundancy folder
nugaon Nov 7, 2023
5c245fb
fix: redundancy paths changed
nugaon Nov 8, 2023
7663797
test: range through all redundancy levels and shards
nugaon Nov 8, 2023
cddec37
test: iterate through encryption or not
nugaon Nov 8, 2023
a2eeec5
fix: changed redundancy paths
nugaon Nov 8, 2023
aee5d3a
feat: encode parity into intermediate chunks
nugaon Nov 8, 2023
5bb42e0
docs: update openapi specs
nugaon Nov 8, 2023
73cdcf9
fix: linting issues
nugaon Nov 8, 2023
acc4750
fix: hopefully the last lint error
nugaon Nov 8, 2023
2fc5adb
feat: add isParityEncoded
nugaon Nov 8, 2023
efa4988
build: readd klauspost reedsolomon package
nugaon Nov 10, 2023
5c47f55
fix: full chunk write and carrier chunk index handling
nugaon Nov 10, 2023
c54ea26
fix(enc): define min function
nugaon Nov 10, 2023
07f69b6
refactor: export DecryptChunkData for decoding parity in need
nugaon Nov 10, 2023
70d5ca6
fix: zero padding for chunkwrite
nugaon Nov 12, 2023
bd572d4
fix: parameter handling of chainwrite in hashtrie
nugaon Nov 12, 2023
f673586
fix: decodespan return value calculated wrong somehow
nugaon Nov 12, 2023
93625fb
test: integration
nugaon Nov 12, 2023
25ab573
chore: go mod tidy
nugaon Dec 1, 2023
70a487d
fix: elevate last chunk
nugaon Nov 29, 2023
7c2915e
feat: referenceCount
nugaon Dec 3, 2023
9c3d7e0
feat: encode redundancy level instead of parity
nugaon Dec 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: changed redundancy paths
nugaon committed Nov 8, 2023
commit a2eeec5250e3d2a42b99ece8da30dbbec09153e2
2 changes: 1 addition & 1 deletion pkg/api/api.go
Original file line number Diff line number Diff line change
@@ -32,7 +32,7 @@ import (
"github.com/ethersphere/bee/pkg/feeds"
"github.com/ethersphere/bee/pkg/file/pipeline"
"github.com/ethersphere/bee/pkg/file/pipeline/builder"
"github.com/ethersphere/bee/pkg/file/pipeline/redundancy"
"github.com/ethersphere/bee/pkg/file/redundancy"
"github.com/ethersphere/bee/pkg/jsonhttp"
"github.com/ethersphere/bee/pkg/log"
"github.com/ethersphere/bee/pkg/p2p"
2 changes: 1 addition & 1 deletion pkg/api/api_test.go
Original file line number Diff line number Diff line change
@@ -31,7 +31,7 @@ import (
"github.com/ethersphere/bee/pkg/feeds"
"github.com/ethersphere/bee/pkg/file/pipeline"
"github.com/ethersphere/bee/pkg/file/pipeline/builder"
"github.com/ethersphere/bee/pkg/file/pipeline/redundancy"
"github.com/ethersphere/bee/pkg/file/redundancy"
"github.com/ethersphere/bee/pkg/jsonhttp/jsonhttptest"
"github.com/ethersphere/bee/pkg/log"
p2pmock "github.com/ethersphere/bee/pkg/p2p/mock"
2 changes: 1 addition & 1 deletion pkg/api/bytes.go
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ import (
"strconv"

"github.com/ethersphere/bee/pkg/cac"
"github.com/ethersphere/bee/pkg/file/pipeline/redundancy"
"github.com/ethersphere/bee/pkg/file/redundancy"
"github.com/ethersphere/bee/pkg/jsonhttp"
"github.com/ethersphere/bee/pkg/postage"
storage "github.com/ethersphere/bee/pkg/storage"
2 changes: 1 addition & 1 deletion pkg/api/bzz.go
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ import (
"github.com/ethersphere/bee/pkg/feeds"
"github.com/ethersphere/bee/pkg/file/joiner"
"github.com/ethersphere/bee/pkg/file/loadsave"
"github.com/ethersphere/bee/pkg/file/pipeline/redundancy"
"github.com/ethersphere/bee/pkg/file/redundancy"
"github.com/ethersphere/bee/pkg/jsonhttp"
"github.com/ethersphere/bee/pkg/log"
"github.com/ethersphere/bee/pkg/manifest"
2 changes: 1 addition & 1 deletion pkg/api/dirs.go
Original file line number Diff line number Diff line change
@@ -19,7 +19,7 @@ import (
"strings"

"github.com/ethersphere/bee/pkg/file/loadsave"
"github.com/ethersphere/bee/pkg/file/pipeline/redundancy"
"github.com/ethersphere/bee/pkg/file/redundancy"
"github.com/ethersphere/bee/pkg/jsonhttp"
"github.com/ethersphere/bee/pkg/log"
"github.com/ethersphere/bee/pkg/manifest"