Add a lotus-miner retrieval add-file command

[whyrusleeping]

In order to allow the retrieval market to bloom beyond the storage market, retrieval miners need to be able to serve arbitrary content.

We should add a command/api that allows retrieval miners to make given data available to their clients, registering it in their local piecestore, and making it available for serving to others in the network. The data should be backed by a blockstore, either a local one or an ipfs backed one.

[mikeal]

If you’re using an IPFS backed store you could make it lotus-miner retrieval {CID} and just pull the data out of the network into the local store. This would allow for a lot more than just individual files to be added.

[whyrusleeping]

@mikeal Yeah, I think that approach makes sense. I'm just being initially somewhat hesitant, and opting for an 'allowlist' of content as the default.

[aschmahmann]

I'm not very familiar with how the IPFS backed datastore works with Lotus, but I'm a little confused as to how/why a retrieval miner would use go-ipfs.

go-ipfs does not currently make it very easy to restrict the flow of information in your blockstore and instead makes it really easy for anyone one the network to get data from you. Since retrieval miner's presumably want to get paid for sending data instead of just sending it for free they almost certainly are not going to want to store their data in go-ipfs.

If you end up implementing some access control blockstore I'm sure go-ipfs would take a look at it though 😄

[mikeal]

I’m thinking about this from a user perspective, which isn’t entirely congruent with the implementation.

There’s already a command that allows you to store any CID from the IPFS network. I’m imagining a command that, to the user, is very similar. The actual implementation is probably going to be quite different, since the client doesn’t need to permanently store and make available in IPFS the data it pulls in for a storage deal in the way this retrieval miner would need to do. This is all to say that this command would not necessary leverage any existing pattern of IPFS usage in Lotus if it decides to take on this feature.

[rjan90]

Closing this ticket as the Legacy Lotus/Lotus-Miner Markets sub-system reached EOL at the end of the 31st January 2023, and is being completely removed from the code here:

• cleanup: Lotus client: remove markets and deal-making from Lotus Client #11999
• cleanup: miner: remove markets and deal-making from Lotus Miner #12005