[Tech Spike] Edge-UR integration

[jcace]

• Edge - horizontally scalable upload gateway
• auto aggregation and commP computation from bucket -> carfile, preconfigured size threshold

2 new APIs
-> GET cars

• piece cid/commp, sizes, CID/CAR mapping

-> DELETE cars

• remove from blockstore

-> GET stats

• disk usage, cpu usage, storage % etc

Authentication
when edge node spin up, have a token as an env var. required in order to call the delete/stats API

With this architecture, DDM can maintain a registry list of Edges (authenticated with their tokens), and call them continuously to get CARfile metadata, push out deals to SPs as they are claimed

Store the mapping of CID/CAR on both Edge and DDM database, so that an edge can die and we can reconstruct the CIDs from scratch by retrieving the carfiles from the network.

[jcace]

[jcace]

@alvin-reyes to implement on Edge side (Edge-URI / Upload, Retrieve, Import)

[jcace]

Validated the import from Edge-URID to DDM -> simple bash script for testing: https://github.com/application-research/delta-dm/blob/spike.edgeImport/scripts/edge-import.sh

End-to-end flow is blocked on the importer side as we still need a way to automate downloads of the carfiles on the SP side