Import deal / DDM integration - Edge URID

[jcace]

TBD

application-research/delta-dm#78

[alvin-reyes]

Edge-urid

Functional / features

• Content creator
• File aggregator
  • Aggregates small files (> BUCKET_AGG_SIZE) to bucket
  • The last file to be added can exceed the BUCKET_AGG_SIZE
  • Creates a CAR file for the bucket
• File splitter
  • Splits a large file into (SPLIT_SIZE configuration)
  • Applies to files that are larger than the BUCKET_AGG_SIZE.
  • The large file will be associated to a new bucket with each split being added as a content
  • Creates a CAR file for the bucket
• Bucket-creator
  • If the content has a miner, a bucket will be created for the miner and for the content. This bucket will need to be filled based on BUCKET_AGG_SIZE.
• Deal-checker
  • If bucket is already replicated more than (MAX_REP) times, it can be deleted by the ADMIN
• Gateway for serving content

System objects

• Content
• Content_Deals
• Buckets

Endpoints

To serve file

• gw/

To add a file

• add
  • File, bucket_uuid (optional), miner (optional)
• gw
  • File, bucket_uuid (optional), miner (optional)
• fetch-from-url
  • File, bucket_uuid (optional), miner (optional)

To get buckets (pull model from SP)

• get-available-open-buckets
  • Return list of CAR files and COMMP and deal request metadata
• Get-open-bucket
  • Return the CAR file and COMMP and deal request metadata
• get-private-bucket (private)
  • Return list of CAR files and COMMP of each for a specific bucket

To manage buckets

• create-bucket (admin or anyone)
  • create a bucket with specific meta/keys
  • bucket_uuid
  • miner
• delete-buckets

To get status

• status/content
• status/bucket
• status/cid

To get stats

• stats
  • Number of content
  • Number of buckets
  • Number of deals attempted
  • Number of deals made

[jcace]

hey @alvin-reyes great work! Just a few questions/points:

I'm envisioning a layer in your diagram between "REST" and "Storage Providers" that is the Deal Orchestration Layer (essentially, where DDM will grow into) - while the SPs could query edge for the car list directly, they'll likely get it from the layer in between (DDM,spade,etc) . They will still pull the carfile contents down from the gateway, though.

Deal-checker
If bucket is already replicated more than (MAX_REP) times, it can be deleted by the ADMIN

I think conceptually, this should not be part of Edge-URID. This can be handled solely by calling the API, and can be modularized for the specific use case / criteria for deletion.

• can you confirm/explicitly state that bucket -> car file is always a 1:1 mapping?
• delete-buckets is called (requiring the admin key) to delete carfiles after they've been served up, correct?
• create-bucket what's the use case for this api? is it just for future expansion/compatibility if we wanted to produce a bucket with arbitrary contents? Should it be an admin-only api?
• get-private-bucket (private) What's this for?

stats

Number of deals attempted
Number of deals made

I don't think those deal stats belong on Edge, it's not goin to be the one making the deals - leave it to Delta or DDM or the other downstream tool

[alvin-reyes]

can you confirm/explicitly state that bucket -> car file is always a 1:1 mapping? - Yes

delete-buckets is called (requiring the admin key) to delete carfiles after they've been served up, correct? - Yes

create-bucket what's the use case for this api? is it just for future expansion/compatibility if we wanted to produce a bucket with arbitrary contents? Should it be an admin-only api? this is if someone wants to have a separate CAR files for all their files (files not to be shared with other files). So if you have a bucket, you can upload a file and specify the bucket to assign the file to that bucket. I'm also adding an "API key" authentication for each bucket so that the owner can be the only one that can upload and retrieve the bucket metadata using /get-private-bucket.