-
Notifications
You must be signed in to change notification settings - Fork 689
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chainHead: Allow methods to be called from within a single connection context and limit connections #3481
Conversation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
let _ = tx.send(result); | ||
}; | ||
|
||
executor.spawn_blocking("substrate-rpc-subscription", Some("rpc"), blocking_fut.boxed()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah right, spawn_blocking
doesn't return a handle for the future that's why you need the extra oneshot channel
/// Limit the RPC functionality to a single connection. | ||
#[derive(Default, Clone)] | ||
pub struct RpcConnections { | ||
data: Arc<Mutex<HashMap<ConnectionId, HashSet<String>>>>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could also be Arc<Mutex<HashSet<(ConnectionId, SubscriptionId)>>>
I guess but perhaps you have plan for it to more generic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mmm since ConnectionId
is just a small u64, I'd just combine to Arc<Mutex<HashSet<(ConnectionId, SubscriptionId)>>>
and that would be more efficient & slightly simpler code below, and probably small or no real extra space cost :)
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@@ -110,6 +110,8 @@ pub struct ChainHead<BE: Backend<Block>, Block: BlockT, Client> { | |||
/// The maximum number of items reported by the `chainHead_storage` before | |||
/// pagination is required. | |||
operation_max_storage_items: usize, | |||
/// Limit the RPC functionality to a single connection. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would re-phrase this, I think you are trying to say that each connection has its own state and not that the RPC functionality is limited to a single connection?
This connection state is then used to determine whether a certain call is permitted on a particular connection such as "unstable_storage calls" or something like that...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've rephrased the documentation a bit, let me know if it makes more sense now 🙏
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
substrate/client/rpc-spec-v2/src/chain_head/subscription/mod.rs
Outdated
Show resolved
Hide resolved
/// For transactionBroadcast, this represents the operation ID. | ||
/// For transaction, this is empty and the number of active calls is tracked by | ||
/// [`Self::num_identifiers`]. | ||
identifiers: HashSet<String>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Am I right to assume that we are not worries about subscription IDs and operation IDs colliding at all in our implementation here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, IIRC, we'll either use this for chainHead with subscriptions ID only; or for the tx with operation ID only
let mut data = self.data.lock(); | ||
|
||
let entry = data.entry(connection_id).or_insert_with(ConnectionData::default); | ||
entry.num_identifiers = entry.num_identifiers.saturating_sub(1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we ever clear out entries completely when the number goes to 0? Else we might eventually leak memory
connection_data.num_identifiers = connection_data.num_identifiers.saturating_sub(1); | ||
|
||
if connection_data.num_identifiers == 0 { | ||
data.remove(&connection_id); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I guess this is whewre we clean up :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me; only very small comments in the end :)
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
… context and limit connections (#3481) This PR ensures that the chainHead RPC class can be called only from within the same connection context. The chainHead methods are now registered as raw methods. - paritytech/jsonrpsee#1297 The concept of raw methods is introduced in jsonrpsee, which is an async method that exposes the connection ID: The raw method doesn't have the concept of a blocking method. Previously blocking methods are now spawning a blocking task to handle their blocking (ie DB) access. We spawn the same number of tasks as before, however we do that explicitly. Another approach would be implementing a RPC middleware that captures and decodes the method parameters: - #3343 However, that approach is prone to errors since the methods are hardcoded by name. Performace is affected by the double deserialization that needs to happen to extract the subscription ID we'd like to limit. Once from the middleware, and once from the methods itself. This PR paves the way to implement the chainHead connection limiter: - #1505 Registering tokens (subscription ID / operation ID) on the `RpcConnections` could be extended to return an error when the maximum number of operations is reached. While at it, have added an integration-test to ensure that chainHead methods can be called from within the same connection context. Before this is merged, a new JsonRPC release should be made to expose the `raw-methods`: - [x] Use jsonrpsee from crates io (blocked by: paritytech/jsonrpsee#1297) Closes: #3207 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
… context and limit connections (paritytech#3481) This PR ensures that the chainHead RPC class can be called only from within the same connection context. The chainHead methods are now registered as raw methods. - paritytech/jsonrpsee#1297 The concept of raw methods is introduced in jsonrpsee, which is an async method that exposes the connection ID: The raw method doesn't have the concept of a blocking method. Previously blocking methods are now spawning a blocking task to handle their blocking (ie DB) access. We spawn the same number of tasks as before, however we do that explicitly. Another approach would be implementing a RPC middleware that captures and decodes the method parameters: - paritytech#3343 However, that approach is prone to errors since the methods are hardcoded by name. Performace is affected by the double deserialization that needs to happen to extract the subscription ID we'd like to limit. Once from the middleware, and once from the methods itself. This PR paves the way to implement the chainHead connection limiter: - paritytech#1505 Registering tokens (subscription ID / operation ID) on the `RpcConnections` could be extended to return an error when the maximum number of operations is reached. While at it, have added an integration-test to ensure that chainHead methods can be called from within the same connection context. Before this is merged, a new JsonRPC release should be made to expose the `raw-methods`: - [x] Use jsonrpsee from crates io (blocked by: paritytech/jsonrpsee#1297) Closes: paritytech#3207 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
This PR ensures that the chainHead RPC class can be called only from within the same connection context.
The chainHead methods are now registered as raw methods.
The concept of raw methods is introduced in jsonrpsee, which is an async method that exposes the connection ID:
The raw method doesn't have the concept of a blocking method. Previously blocking methods are now spawning a blocking task to handle their blocking (ie DB) access. We spawn the same number of tasks as before, however we do that explicitly.
Another approach would be implementing a RPC middleware that captures and decodes the method parameters:
However, that approach is prone to errors since the methods are hardcoded by name. Performace is affected by the double deserialization that needs to happen to extract the subscription ID we'd like to limit. Once from the middleware, and once from the methods itself.
This PR paves the way to implement the chainHead connection limiter:
Registering tokens (subscription ID / operation ID) on the
RpcConnections
could be extended to return an error when the maximum number of operations is reached.While at it, have added an integration-test to ensure that chainHead methods can be called from within the same connection context.
Before this is merged, a new JsonRPC release should be made to expose the
raw-methods
:Closes: #3207
cc @paritytech/subxt-team