Retrieval Discovery and Payload CIDs

[hannahhoward]

The existing retrieval spec defines an interface for discovering retrieval miners with the following interface:

type RetrievalPeerResolver struct {
    GetPeers(PieceCID piece.PieceCID) [RetrievalPeer]
}

Because deals are on chain, we can query chain history to identify which miners have a given piece CID. However, soon a piece CID will simply be CommP, while most likely people are going to be interested in a given payload CID. CommP / PieceCID cannot be calculated without underlying data, so how does a client find a miner who can serve a given payload CID? Some options to consider:

1. Defer this question to after main net and address in a future version of the retrieval protocol
2. Defer this question to outside the spec -- a third party service could establish itself to maintain a registry of PayloadCID -> pieceCID
3. Maintain some sort of light DHT or other decentralized mechanism to map PayloadCID -> PieceCid (doesn't even need to track peers who have pieceCID -- cause the chain can be queried to retriev that -- though it might be a useful optimization)

Option 1 or 2 may limit the ability to use filecoin as a store for apps seeking faster retrievals. Though alternatively an app could keep a piece CID onhand to query as needed.

[anorth]

Thanks. I think this is ok for now, and we're pursuing option 1 or 2 (both of which mean no immediate work in this spec). cc @jbenet

[hannahhoward]

closing for now then.

[hannahhoward]

Hi, I'm going to reopen this issue because @pooja has raised it as a priority.

[mishmosh]

Synced live with @pooja, and clarified that she's interested in the ability to retrieve by payloadCID (#853) rather than blind discovery. Closing this again as a nice-to-have.