Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpc: add full support reconnecting rpc client #1505

Merged
merged 78 commits into from
May 8, 2024
Merged
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
476afd9
add simple reconnecting rpc client
niklasad1 Mar 21, 2024
0bf7299
Merge remote-tracking branch 'origin/master' into na-reconnecting-rpc…
niklasad1 Mar 26, 2024
9f6d38b
initial retryable calls
niklasad1 Mar 27, 2024
c107e14
add reconnecting backend
niklasad1 Mar 28, 2024
4208599
add reconnecting example for unstable backend
niklasad1 Mar 28, 2024
1ecc4a0
add todo what isn't working
niklasad1 Mar 28, 2024
ffabc04
FollowStream: restart on reconn
niklasad1 Apr 9, 2024
4ba4d75
naive fix: fetch sub_id in stream_headers
niklasad1 Apr 10, 2024
76cbd71
cleanup
niklasad1 Apr 11, 2024
e71a583
remove resubscribe APIs
niklasad1 Apr 11, 2024
2457d3a
cleanup and remove many wrapper streams
niklasad1 Apr 12, 2024
25fa3b1
remove retry backend
niklasad1 Apr 15, 2024
39dcea4
legacy rpc: make it retryable
niklasad1 Apr 15, 2024
a5ae924
unstable rpc: make it retryable
niklasad1 Apr 16, 2024
f27a7f4
Merge remote-tracking branch 'origin/master' into na-reconnecting-rpc…
niklasad1 Apr 16, 2024
5822c40
fix nits
niklasad1 Apr 16, 2024
adc35b9
support wasm as well
niklasad1 Apr 17, 2024
643af0b
remove deadcode
niklasad1 Apr 17, 2024
825e3b3
address grumbles
niklasad1 Apr 18, 2024
6027072
revert rpc methods
niklasad1 Apr 18, 2024
7b01f1f
don't create a subscription per block
niklasad1 Apr 18, 2024
2bfede2
get rid off retry logic in subxt rpc
niklasad1 Apr 19, 2024
b14f144
Update subxt/Cargo.toml
niklasad1 Apr 19, 2024
284abdb
Update subxt/src/backend/legacy/mod.rs
niklasad1 Apr 19, 2024
77bb9fd
Update subxt/src/backend/legacy/mod.rs
niklasad1 Apr 19, 2024
43c4cf1
remove outdated comments
niklasad1 Apr 19, 2024
0dfaeea
Merge remote-tracking branch 'origin/na-reconnecting-rpc-client-v2' i…
niklasad1 Apr 19, 2024
be93cdd
Merge branch 'master' into na-reconnecting-rpc-client-v2
niklasad1 Apr 22, 2024
85285c7
fix bad merge
niklasad1 Apr 22, 2024
eb84437
Merge remote-tracking branch 'origin/master' into na-reconnecting-rpc…
niklasad1 Apr 22, 2024
d645aee
Fix reconnecting RPC client and update dependencies
niklasad1 Apr 22, 2024
3cee6e9
add back retry logic and remove `finito`
niklasad1 Apr 22, 2024
2dcac78
fix nits
niklasad1 Apr 22, 2024
91b2804
cleanup
niklasad1 Apr 22, 2024
d673203
add hack for race when reconnecting
niklasad1 Apr 23, 2024
802f7ee
backend: emit Stop event DisconnectWillRecoonect
niklasad1 Apr 23, 2024
2794ac1
merge reconnecting client examples
niklasad1 Apr 23, 2024
80c2396
Merge remote-tracking branch 'origin/master' into na-reconnecting-rpc…
niklasad1 Apr 23, 2024
7b4a9fe
add fn retry_stream
niklasad1 Apr 23, 2024
44b15ff
cleanup
niklasad1 Apr 24, 2024
a369f26
add all features from reconnecting-rpc-client
niklasad1 Apr 24, 2024
6261cb8
fix build
niklasad1 Apr 24, 2024
3dc77c2
Merge remote-tracking branch 'origin/master' into na-reconnecting-rpc…
niklasad1 Apr 25, 2024
7d6428f
remove needless retry for fetch_storage
niklasad1 Apr 25, 2024
628ee57
StorageFetchDescendantKeysStream handle disconnect err
niklasad1 Apr 25, 2024
0a837a4
dont retry transactions
niklasad1 Apr 25, 2024
55f68d5
fetch subscription ID from FollowStreamMsg
niklasad1 Apr 25, 2024
ada46e7
fix nits
niklasad1 Apr 25, 2024
3b4963a
Update subxt/src/backend/legacy/mod.rs
niklasad1 Apr 25, 2024
a1f3c0e
Update subxt/src/backend/legacy/mod.rs
niklasad1 Apr 25, 2024
97f9480
add reconn to StorageItems stream
niklasad1 Apr 25, 2024
a7ae56f
Merge remote-tracking branch 'origin/master' into na-reconnecting-rpc…
niklasad1 Apr 30, 2024
1bae24b
StorageFetchDescendantKeysStreamchore: retry storage call
niklasad1 Apr 30, 2024
7747382
RetryStream: emit DisconnectWillReconnect msg
niklasad1 Apr 30, 2024
425c7fa
runtime subscriptions ignore DisconnectWillReconn
niklasad1 Apr 30, 2024
ab9a6e3
Update subxt/examples/setup_reconnecting_rpc_client.rs
niklasad1 Apr 30, 2024
0d45b1b
Update subxt/src/client/online_client.rs
niklasad1 Apr 30, 2024
e9d171c
Update subxt/src/client/online_client.rs
niklasad1 Apr 30, 2024
1ba9814
Add custom stream wrapper for finalized blocks
niklasad1 May 2, 2024
5ebdfe2
add missing retry block
niklasad1 May 2, 2024
8deaba2
clippy
niklasad1 May 2, 2024
523904e
clippy again
niklasad1 May 2, 2024
9da32c9
cleanup
niklasad1 May 3, 2024
46d9879
remove duplicate logic
niklasad1 May 3, 2024
546425f
fix more grumbles
niklasad1 May 3, 2024
22e13f1
Update subxt/examples/setup_reconnecting_rpc_client.rs
niklasad1 May 3, 2024
1a986e5
simplify the example
niklasad1 May 4, 2024
abb7cf0
remove pin-project dep
niklasad1 May 4, 2024
8cce412
Merge remote-tracking branch 'origin/master' into na-reconnecting-rpc…
niklasad1 May 6, 2024
775d3fd
remove duplicate retry logic
niklasad1 May 7, 2024
d3faa21
remove extra code
niklasad1 May 7, 2024
59401b8
specify trait bounds for retry api
niklasad1 May 7, 2024
3b3e8e4
simplify the example
niklasad1 May 7, 2024
57881fa
fix weird Poll::Pending return
niklasad1 May 7, 2024
7bd55e7
fix nit in poll impl
niklasad1 May 8, 2024
c74d514
remove needless paths
niklasad1 May 8, 2024
27a2ea4
make retry_stream pub and add doc examples
niklasad1 May 8, 2024
64d3aae
Update subxt/src/backend/utils.rs
niklasad1 May 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 7 additions & 5 deletions subxt/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ native = [
"jsonrpsee?/async-client",
"jsonrpsee?/client-ws-transport-native-tls",
"subxt-lightclient?/native",
"tokio-util"
"tokio-util",
"reconnecting-jsonrpsee-ws-client?/native",
]

# Enable this for web/wasm builds.
Expand All @@ -39,7 +40,8 @@ web = [
"getrandom/js",
"subxt-lightclient?/web",
"subxt-macro/web",
"instant/wasm-bindgen"
"instant/wasm-bindgen",
"reconnecting-jsonrpsee-ws-client?/web",
]

# Enable this to use the reconnecting rpc client
Expand Down Expand Up @@ -99,7 +101,7 @@ subxt-metadata = { workspace = true, features = ["std"] }
subxt-lightclient = { workspace = true, optional = true, default-features = false }

# Reconnecting jsonrpc ws client
reconnecting-jsonrpsee-ws-client = { version = "0.4", optional = true }
reconnecting-jsonrpsee-ws-client = { version = "0.4", optional = true, default-features = false }

# For parsing urls to disallow insecure schemes
url = { workspace = true }
Expand Down Expand Up @@ -137,8 +139,8 @@ path = "examples/light_client_local_node.rs"
required-features = ["unstable-light-client", "jsonrpsee", "native"]

[[example]]
name = "reconnecting_rpc_client"
path = "examples/reconnecting_rpc_client.rs"
name = "setup_reconnecting_rpc_client"
path = "examples/setup_reconnecting_rpc_client.rs"
required-features = ["unstable-reconnecting-rpc-client"]

[package.metadata.docs.rs]
Expand Down
73 changes: 0 additions & 73 deletions subxt/examples/reconnecting_rpc_client.rs
niklasad1 marked this conversation as resolved.
Outdated
Show resolved Hide resolved

This file was deleted.

107 changes: 107 additions & 0 deletions subxt/examples/setup_reconnecting_rpc_client.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
//! Example to utilize the `reconnecting rpc client` in subxt
//! which hidden behind behind `--feature unstable-reconnecting-rpc-client`
//!
//! To utilize full logs from the RPC client use:
//! `RUST_LOG="jsonrpsee=trace,reconnecting_jsonrpsee_ws_client=trace"`

#![allow(missing_docs)]

use std::{sync::Arc, time::Duration};

use futures::StreamExt;
use subxt::backend::legacy::LegacyBackend;
use subxt::backend::rpc::reconnecting_rpc_client::{Client, ExponentialBackoff};
use subxt::backend::rpc::RpcClient;
use subxt::{OnlineClient, PolkadotConfig};

// Generate an interface that we can use from the node's metadata.
#[subxt::subxt(runtime_metadata_path = "../artifacts/polkadot_metadata_small.scale")]
pub mod polkadot {}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
tracing_subscriber::fmt::init();

// Create a new client with with a reconnecting RPC client.
let rpc = Client::builder()
// Reconnect with exponential backoff
//
// This API is "iterator-like" and we use `take` to limit the number of retries.
.retry_policy(
ExponentialBackoff::from_millis(100)
.max_delay(Duration::from_secs(10))
.take(3),
)
// There are other configurations as well that can be found at [`reconnecting_rpc_client::ClientBuilder`].
.build("ws://localhost:9944".to_string())
.await?;

// If you want to use the unstable backend with the reconnecting RPC client, you can do so like this:
//
// ```
// use subxt::backend::unstable::UnstableBackend;
// use subxt::OnlineClient;
//
// let (backend, mut driver) = UnstableBackend::builder().build(RpcClient::new(rpc.clone()));
// tokio::spawn(async move {
// while let Some(val) = driver.next().await {
// if let Err(e) = val {
// eprintln!("Error driving unstable backend: {e}; terminating client");
// }
// }
// });
// let api: OnlineClient<PolkadotConfig> = OnlineClient::from_backend(Arc::new(backend)).await?;
// ```

let backend: LegacyBackend<PolkadotConfig> =
LegacyBackend::builder().build(RpcClient::new(rpc.clone()));

let api: OnlineClient<PolkadotConfig> = OnlineClient::from_backend(Arc::new(backend)).await?;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as a note: in the "default" case, one can also just to something like OnlineClient::from_rpc(rpc_client), which might be an easier thing to point them at in an example.

(And perhaps they can then figure out how to use custom backends in a separate example)


// Optionally print if the RPC client reconnects.
let rpc2 = rpc.clone();
tokio::spawn(async move {
loop {
// The connection was lost and the client is trying to reconnect.
let reconnected = rpc2.reconnect_initiated().await;
let now = std::time::Instant::now();
// The connection was re-established.
reconnected.await;
println!(
"RPC client reconnection took `{}s`",
now.elapsed().as_secs()
);
}
});

// Run for at most 100 blocks and print a bunch of information about it.
//
// The subscription is automatically re-started when the RPC client has reconnected.
// You can test that by stopping the polkadot node and restarting it.
let mut blocks_sub = api.blocks().subscribe_finalized().await?.take(100);

while let Some(block) = blocks_sub.next().await {
let block = match block {
Ok(b) => b,
Err(e) => {
// This can only happen on the legacy backend and the unstable backend
// will handle this internally.
if e.is_disconnected_will_reconnect() {
println!("The RPC connection was lost and we may have missed a few blocks");
continue;
}

return Err(e.into());
}
};

let block_number = block.number();
let block_hash = block.hash();

println!("Block #{block_number} ({block_hash})");
}

println!("RPC client reconnected `{}` times", rpc.reconnect_count());

Ok(())
}
Loading
Loading