Shwap workstream tracking issue

[Wondertan]

Tracking issue for the storage and sampling part of the new Shwap protocol. The list is extracted and adapted from draft PoC TODO list.

Storage(dirty)

• ODSFile
  • Storage Format with Header
  • Support Row and Col axis reads from ODSFile
  • Support for single share reads with proof from ODSFile
  • Support getting namespace data per row with proofs
  • Measure size diff between ODSFile with previous CAR files.
  • Lazy-cache implementations for proofs and recomputed/extended axises.
  • InMemory implementation for testing
  • Benchmarks for reconstructSome, full row encode (cauchy), full encode Leopard
  • reconstructSome compatability by using direct reedsolomon codec
  • extend reconstructSome on parity (klauspost repo)
  • reuse large object by memory pool
  • cache ods after first read
  • Leopard based reconstructSome (blocked)
  • thread safety
• q1 + q4 File
  • Base implementation
  • Benchmarks for worst case scenario vs ODSFile
• Filestore(integration stage)
  • ODSFile streaming (Write/Read)
  • Create new Store
  • Add index by height to Store
  • update caching to work over new File
  • Add caching to Store
  • Add index by Height to Cache
  • Blockstore over Store
  • Thread-safety for store
  • Lock Store folder while in use
  • Post-integration code cleanup of old storage.
  • Prunning support
  • Metrics
• Filestore integration
  • Update ShrEx/Nd protocol, client/server implementations
  • Update ShrEx/Eds
  • Update Store getter
  • update availability
• BEFP integration
  • extract shareWithProof from byzantine package
  • rework befp construction to work over array of shares (new retriever integration)

Sampling

• Protocol Messages
  • SampleID -> Sample - requests samples by share indecies
  • AxisID -> Axis - requests halfs of axises
  • DataID -> Data - requests namespace data per row
  • Codecs and Multihash codes with ID types conversion to CIDs
  • Stateless Hashers
  • Protobuf definitions
  • Use heights instead of DataHash to identify block
  • Hash requested Axis roots to cut namespaces and shorten requests
  • Versioning and backwards compatibility with old protocol
  • Server should decide which sample proof to return
  • Remove Colomn requests (AxisID -> RowID)
  • Remove AxisHash from AxisID
• Integration
  • IPLDv2 row based getter
    • Getter should support requesting batch of shares by idx at once
    • Store only samples by default
    • Bitswap Sessions
  • Retriever over v2 (IPLDv2 reconstruction getter) (assigned to @walldiss)
  • Multiplexer for different shwap protocols (@walldiss))
  • Metrics
  • IPLDv2 Server over LN storage (blockservice) Will work as is
  • Migrate actual sampling logic over GetShares share/light: LightAvailability to use GetShares #2984

Testing

• test shrex with new storage
• test shwap
• test on large storage
• design load tests
• perform load tests

Stretch Goals

• Protocol message for requesting range of shares
  • By extending SampleID with additional uint16 that defines the end idx of range
• Blob requests by commitements
  • BlobID{DataID, Comm} -> Blob{BlobShares, ProofsToSubroot}
• Requesting proofs only for some request types
• Reconsider DHT discovery.

Storage(clean)

TBC

[cristaloleg]

Related #3513

[Wondertan]

Closing this one as completed. The TODO completion is actual and needed points were migrated to hardening