feat: retrieval server & client setup #654
Conversation
th7nder
changed the title
There was a problem hiding this comment.
I lack completeness in this PR, there is a lot of todos, the functionality is unclear, lots of scenarios are unhandled.
This feels like a prototype for trying out how to transfer a 1 packet via beetswap and libp2p, not something that we can merge into 'production code.'
I'd need to know the strategy on how are we going to implement this whole thing before I agree to merge it.
E.g. the server.rs looks like it'll be gone anyways, or not present in the production code -> because that's something that should be implemented on the Storage Provider Server.
client.rs looks like a candidate to be a separate CLI, cause I suppose we're going to run it like:
polka-fetch <CID> <STORAGE_PROVIDER_NODE_ADDR>
I assume if we'd like to start writing production code and merging it, we should:
- setup CLI
polka-fetch packageand define it's funcionalities. - add libP2P server to storage provider node, so we can
dial it - dial from
polka-fetchtostorage provider node - Add beetswap protocol support to Server and client, prove that they can connect and download a single file (hardcoded data always returned)
- Add ability to storage provider server to return a CAR archive identified by CID (I guess use local indexer).
- Call polka-fetch with those multiple CIDs and verify those archives are downloaded properly.
(I just thought on the fly about it, nothing binding)
So very similar to the list of tasks that you wrote in #634.
| tracing = { workspace = true } | ||
|
|
||
| [dev-dependencies] | ||
| # multihash = "0.19.3" |
There was a problem hiding this comment.
| # multihash = "0.19.3" |
| @@ -136,6 +142,7 @@ pallet-market = { path = "pallets/market", default-features = false } | |||
| pallet-proofs = { path = "pallets/proofs", default-features = false } | |||
| pallet-randomness = { path = "pallets/randomness", default-features = false } | |||
| pallet-storage-provider = { path = "pallets/storage-provider", default-features = false } | |||
| polka-index = { path = "storage/polka-index" } | |||
storage-retrieval/README.md
Outdated
| use std::{str::FromStr, sync::Arc, time::Duration}; | ||
|
|
||
| use anyhow::Result; | ||
| use blockstore::{ |
There was a problem hiding this comment.
nice, didn't know eiger had this library
|
|
||
| // Setup server | ||
| let server = Server::new(blockstore)?; | ||
| let listener: Multiaddr = format!("/ip4/127.0.0.1/tcp/8989").parse()?; |
There was a problem hiding this comment.
| let listener: Multiaddr = format!("/ip4/127.0.0.1/tcp/8989").parse()?; | |
| let server_address: Multiaddr = format!("/ip4/127.0.0.1/tcp/8989").parse()?; |
| .with_behaviour(|_| Behaviour { | ||
| bitswap: beetswap::Behaviour::new(blockstore), | ||
| }) | ||
| .expect("infallible") |
| bitswap: beetswap::Behaviour::new(blockstore), | ||
| }) | ||
| .expect("infallible") | ||
| .with_swarm_config(|c| c.with_idle_connection_timeout(Duration::from_secs(60))) |
| self.queries.insert(query_id, payload_cid); | ||
|
|
||
| // Pin cancellation future | ||
| pin_mut!(cancellation); |
There was a problem hiding this comment.
Why? What about CancellationTokens?
Description
Setup the retrieval server and client. They encapsulate the logic needed to successfully transfer files between them. The current blockstore used is in memory and doesn't have a concept of car format. The idea is to swap the server's and client's blockstore to the implementation specific for them.
The example initializes server and client and transfers a single block between them.
Checklist