Spark v1.5: limited support for DDO deals

[bajtos]

There is an increasing urgency for Spark to support DDO deals. The Spark v2 architecture requires changes in miner SW; it will take months until the network meets Spark v2 requirements.

Spark v1.5 is a short-term stop-gap solution which retrofits DDO deals onto the existing Spark v1 database of eligible deals in a backwards-compatible way that preserves current RSR for f05 deals.

Consult the following documents for more details:

• Spark & DDO deals
• Spark DDO hack

Tasks:

• Enhance fil-deal-ingester to store PieceCID and PieceSize alongside Label CID in the database of eligible deals. Also, add a new column to the database of eligible deals to distinguish between f05 and DDO deals.
  → Extend eligible deals schema to enable adding DDO deals fil-deal-ingester#29
• Build deal-observer for DDO as planned for Spark v2. This will produce a list of (minerId, clientId, PieceCID, PieceSize) rows in an SQL table.
  → Deal Observer research: how to handle deal expiration/extension/slashing #171
  → Deal Observer: initial implementation (deal activation events) #172
• Add a PayloadCID column to deal-observer DB and set it to NULL for newly ingested deals.
• Periodically scan the table produced by spark-observer and for each deal that does not have PayloadCID set, ask piece-indexer for a sample payload block CID and store the CID in our table if found. If not, then we need to retry the query, because it can take some time until IPNI advertisements are processed for the new deal. We can stop retrying after some configured time, e.g. 3 days.
  → feat: fetch payload CIDs from piece indexer spark-deal-observer#31
  UPDATE (2024-01-31): In feat: fetch payload CIDs from piece indexer spark-deal-observer#31 (comment), we agreed to leave the retrying part out of the initial implementation. The follow-up issue is Piece Indexer Missing CIDs spark-deal-observer#30
• Periodically cross-check the list of f05 deals created by fil-deal-ingester with the list of deals created by deal-observer. Find deals that are in the spark-observer database only and which have PayloadCID value filled in. Add these deals to the “master” table of deals eligible for retrieval checking, and flag these deals as coming from spark-observer.
  Note: There should be a lot of overlap between the list produced by fil-deal-ingester and deal-observer, because f05 deals will be ingested by both services.
  → feat: POST /eligible-deals-batch spark-api#502
  → Periodically add new deals to spark-api's database of eligible deals spark-deal-observer#26
• Check the data produced by the new system and fix any issues discovered.
  Things to look for:
  • Is deal-observer capturing the correct set of deals per epoch?
  • Is the PieceIndexer-based process for adding PayloadCID working as expected?
  • What is the Spark RSR score for these new deals?
  • Did this new system cause any major changes in the Spark RSR score (especially for miners who used to have a good score)?

Additional tasks on top of the above - please add new issues as we discover them:

• active_deals contain more deals than shown by Beryx explorer spark-deal-observer#67

Follow-up tasks (moved out of the scope of this issue):

• Document requirements for SPs who want their DDO deals to be tested by Spark #232

[bajtos]

Shipped! 🚀