CID sampling v0.2

[bajtos]

eta: 2024-01-31

The next step on our journey to #24, a follow-up for CheckerNetwork/roadmap#43

1. Build IPNI Context ID from FIL Deal proposal, so that we can filter IPNI records to pick only the advertisement from the SP handling the deal
   → Moved to Link retrievals to miners roadmap#65

2. Rework SPARK tasking to push IPNI queries to spark-checkers

   • Change the tasks from (CID, address, protocol) to (miner, contextId, CID)
   • Change spark-checkers to query IPNI to get the address & protocol, include miner and contextId in the measurements
   • Change spark-api to ingest new measurement fields
   • Change spark-evaluate - add a fraud-detection step to validate that all members of the committee used the same address & protocol

[bajtos]

A Rust library we can use to parse JSON in a streaming way: https://crates.io/crates/json-event-parser

[bajtos]

Build IPNI Context ID from FIL Deal proposal, so that we can filter IPNI records to pick only the advertisement from the SP handling the deal

See the discussion in
https://filecoinproject.slack.com/archives/C048DLT4LAF/p1699956597137929

Example deal:

{
    "Proposal": {
      "PieceCID": {
        "/": "baga6ea4seaqa3onc6kbej6ekjydnbgjnqjzlnlmcafedswhoyn5ubrfmb34uida"
      },
      "PieceSize": 34359738368,
      "VerifiedDeal": true,
      "Client": "f01985718",
      "Provider": "f02008883",
      "Label": "bafybeieosqoxjt6xvwby33ts4gdcykd3fbj6ffj2olhl67lbpefccdnihe",
      "StartEpoch": 2652977,
      "EndEpoch": 4173617,
      "StoragePricePerEpoch": "0",
      "ProviderCollateral": "12325970877125150",
      "ClientCollateral": "0"
    },
    "State": {
      "SectorStartEpoch": 2632102,
      "LastUpdatedEpoch": 3128814,
      "SlashEpoch": -1,
      "VerifiedClaim": 8526408
    }
}

CID Contant entry:

[
  {
    ContextID: 'AXESID6uXFBtGR4r+3yxTK9DqevBRwfAhtlnbqVqusVYErka',
    Metadata: 'kBKjaFBpZWNlQ0lE2CpYKAABgeIDkiAgDbmi8oJE+IpOBtCZLYJytq2CAUg5WO7De0DErA75RAxsVmVyaWZpZWREZWFs9W1GYXN0UmV0cmlldmFs9Q==',
    Provider: {
      ID: '12D3KooWBCPM3hAFfCgaRRfZ55HmRHECTjsZsSpSQXGXFaNQhy7m',
      Addrs: [ '/ip4/113.23.169.21/tcp/30005', [length]: 1 ]
    }
  },
  [length]: 1
]

https://github.com/filecoin-project/boost/blob/main/indexprovider/wrapper.go#L168-L172
https://github.com/filecoin-project/boost/blob/main/indexprovider/wrapper.go#L195
bytes of CID of dag-cbor encoded DealProposal

[bajtos]

I hit a block while working on "IPNI Context ID from FIL Deal proposal", so I decided to implement the second part (push IPNI queries to Spark checkers) first.

Design doc: https://www.notion.so/pl-strflt/Moving-IPNI-queries-to-Spark-checkers-c58175c0f1c841e6bb82ac44a1472a98