Implement pallet view function queries #4722
Conversation
It is a good point. Client libraries will probably use metadata to generate the view function accessors. When using indices they will have to generate one set of functions per chain. When using hashes they can just generate functions from all the metadata files they have access to and merge them while removing duplicates. I think thats much better. Assuming that pallet names are unique of course. And even when using generated code people will refer to the function by name and signature. Changing the name will break peoples code when they upgrade the lib as it will have generated the function under a different name. So we might as well encode it into the id to make sure we never ever change it. So yes I think hashing based on name + signatures makes it easier for client lib devs. I think it outweighs the benefits of having everything done the same way. What do you think @kianenigma ? |
I don't really get what you are saying here? Why do they need multiple functions? For what? Pallet names are not unique across different chains, you can name your pallet whatever you like in |
@bkchr: They can be, because you might forget to put the right piece of code in
@athei: Second. It is hard to foresee things without getting the hands dirty. This PR is stable enough to be built on top of. Any of us can and should go and build a few pallet-level view fns on top of this PR, and then see how it will look like if we want to amalgamate them. The view function forum post has some ideas in it both for pallet level, and runtime level view functions.
Sadly both pallet name and index are variable per-runtime, so the north star of removing complexity from client libs is not feasible. In all future scenarios, if a client lib wants to provide an abstraction over the same view function in 5 different chains, it would inevitably need to go through some effort and "speculate/hardcode" a few things. I don't think there is any solution for it. Given this, I am also still leaning towards enum based dispatch. It is more unified, and we can pursue someday moving all dispatches to hashes, but not have one foot in each. With the time bomb of reaching the 255 limit, we will have to do it someday :) We can then reserve a special index in both the |
|
Further comment on stability: 💡 in fact the only thing that is not variable across runtimes is the "original" (prior to cargo package renaming) Of course, you would still have the complexity of pallet instancin, but I think working on the basis of crate name is the only reasonable step forward here. All in all, the ordering seems to be, from "unstable" to "stable": Pallet index << Pallet Name << Pallet Crate name |
I think that identifying a pallet by the pallets name is still the way to go for client libs. This allows a crate to be reused with different config for more than one pallet without clients caring (which was done not too long ago for some pallet I can't remember off the top of my head). I'm also happy to stick with enum index over hash; we've had to solve this problem already for eg RuntimeCalls on the client side (which was can do by looking up the correct index based on the name of the pallet / call that we want) so offhand I don't mind the same being true for making these view function calls, and don't feel like it will add a lot of complexity on the client side. The 256 limit scares me a bit, but I expect we can migrate things over to hash based lookup by the time we get close to that (which may come more quickly once each top level call gets its own variant). |
|
@bkchr has said the solution to the 256 limit problem actually is to use compact integers for the enum length, whereas we use a single byte right now. But this is a big breaking change to SCALE and our ecosystem. That would support enums of arbitrary length. That being said, using a small hash probably brings to our ecosystem a lot of familiarity and devex to Polkadot, as Ethereum uses this pattern in smart contracts, and it allows ways to uniquely identify certain kinds of calls and functions across pallets and implementations. That being said, in the Rust ecosystem, we do have traits, and it seems that this would solve the same problem if we actually used them right. |
|
I was assuming we are using the create name as pallet name when writing my text. Given that it should be stable across chains. Keep in mind we are also hashing the signature in. This means the hash is the same across chains as changing the view functions semantic is not allowed. I see really no benefit in using indices except not introducing a new way of dispatching. |
|
Just asking couple of possibly DQs as I try to understand this area/direction better :):
|
It is some infrastructure built on top of runtime APIs to allow for easy definition and consumption of queries. Assuming just the pallet level functions as defined in this PR: It allows for the definition inside the pallet without the cumbersome declaration and later implementation at runtime level. It is just shipped with the pallet. The other feature is immutability: They are never ever allowed to be removed or altered. This is not the case for runtime APIs in general.
Settle the debate on hashes vs index. I don't think we need to do anything for subscribe ability in this PR. We can add an RPC later like
A lot. It is almost the whole point of it. So the will be stable. Period. We don't break downstream tools on purpose. So this is also why we suggest hashing the crate name and not the name within the runtime. |
|
Thanks for your reply @athei! Mayeb I'm way off base or derailing things and apologies if so, but just some thoughts and responses:
Does this mean that somebody will be able to define runtime APIs and runtime view functions and then access either via eg
Ah ok, good to know we can leave subscribing for another PR. I geuss the same applies when we impl runtime level view functions; subscribe-ability can be a later followup hopefully!
Yup 100%; having stable functions whose signature/meanings do not change makes sense for sure. I guess I am just questioning:
I suppose in the end, I feel like in order to have a stable interface, we'd need to define a written spec anyway for runtimes to adhere to. The spec could break the interface down into sections for eg staking, governance, assets or whatever, a bit like https://paritytech.github.io/json-rpc-interface-spec/ does, to allow runtimes to adhere to one or more sections but not necessarily everything. if we have such a spec, we could expose runtime-API-like functions that adhere to it, and have a test suite to help test for conformance on a runtime. Pallet level view functions would be called in this interface to assemble the information we want to hand back (and avoid direct storage requests as much as possible), but they themselves would be an implementation detail w.r.t the spec, so it would be totally pallet agnostic. We also wouldn't really need any special hashing of Maybe I'm going way off base here though and missing some crucial bits; what do you reckon? |
A view function is a runtime API. So yes, both are accessed via Regarding the stability stuff: Let's not over engineer here. There is a lot of code if you would change it, everything would fall apart. The code is the definition and we just not change the semantics. Just like for dispatchables. I mean what for do we have a runtime if we still need a spec? CI checks that detect breakage is welcome, though. |
My thinking is that having a spec comes in handy when we want multiple runtimes to implement the same sets of methods and semantics in order that the facade stuff can talk to them all. It's a single source of truth from which we can then also have eg CI tests or whatever to check for conformance. So then any runtime can choose to impl the spec or not (or sections of it) and we can have tests/tools to help check this, and help ensure that support isn't later broken by a runtime code change. If the higher level facade APIs are just thin wrappers around this spec (ignoring historical stuff), then we might also be able to define both the expected runtime APIs and then higher level facade APIs and semantics in one go. |
|
Multiple runtimes importing the same pallets will offer the same semantics. Thats the whole point of bundling them into modules (aka pallets) and not implementing them at runtime level like a generic runtime API. And yes runtime wide view functions will put a slight wrinkle into this plan. But we can also just package them into a crate that we distribute and that everybody imports and wires them up with their pallets. Maybe you can view that as some sort of spec. |
|
Regarding the hashing: no one is strongly backing it, we should set our aim to merge this PR with old school dispatch. This method also has not benefit per-se, but it is at least more consistent. Some final counterarguments:
Having all this conversation is a good encouragement to already start thinking about a migration plan into compact encoded indices for enums to eliminate this time bomb. |
This all makes sense, and yup this is the bit that I'm thinking about as a spec, and I agree that the spec can be implemented in the form of code and probably from this we can generate documentation to also describe it to people who want to then use the facade interface at a higher level (client facade implementations for instance) |
|
Just a general thought on the naming in this PR: How do people feel about calling these "Pallet APIs" rather than "Pallet view functions"? My thought is to be consistent with the term "Runtime APIs", which (from the client perspective at least) act very similar; read-only functions that can be called with some arguments and return some value back. |
@kianenigma I was thinking what the metadata would look like in order that clients know how to call pallet view functions. For the case of eg transactions, we provide an enum in the metadata for each pallet, and that works great because each variant name corresponds to the transaction name and each enum field correpsonds to one of the transaction arguments. For the case of Pallet View Functions (same as rutime APIs), an enum doesn't provide enough information (ie we also need to know the return type for each Pallet View Function. So I suppose the metadata for the ciew functions in a pallet using enum dispatch would look more like: view_functions: vec![
PalletViewFunction {
index: 1u8,
name: String,
inputs: vec![12, 34, 8], // u32 type IDs for each argument
output: 34, // u32 type ID for return type,
docs: Vec<String>
},
// ...
]The index in the metadata exists only to dispatch to the right function using the enum approach. If we did hash based dispatch, then we could omit the index and describe pallet view functions identically to Runtime APIs, ie like this: pub struct PalletApiMethodMetadata<T: Form = MetaForm> {
/// Method name.
pub name: T::String,
/// Method parameters.
pub inputs: Vec<RuntimeApiMethodParamMetadata<T>>,
/// Method output.
pub output: T::Type,
/// Method documentation.
pub docs: Vec<T::String>,
}(see https://github.com/paritytech/frame-metadata/blob/main/frame-metadata/src/v15.rs#L132) Does this change your opinion on enum versus hash based dispatch, or is it actually a non issue? |
I don't think we should use bad names for the sake of consistency. Remember that we call host functions "Runtime Interface". It is confusing to use those generic names. There will also be runtime wide view functions eventually. |
Throwing some thoughts out there:
I like the idea of things being unified, OTOH if we know we are going there, why not do it here first where the impact is low (nobody is using view functions yet). The cost of migrating in the future should be considered. |
This is a great idea, and a nice bonus: If you think we can already experiment with Enum based dispatch, with compact encoded indices, let's do it here. @ascjones
Indeed, the metadata for view functions would need to be more sophisticated than dispatch, so it cannot be purely enum. @jsdw
About naming, I would go with view functions for now. We can always rename it later and pre-launch as well, so it is okay. |
I was in fact referring to "we can pursue someday moving all dispatches to hashes", so presenting the case for using hash instead of enum based dispatch 🙈 |
|
Me and @kianenigma had a chat about this the other day and the consensus was that, while our opinions differ on enum dispatch versus not, we shouldn't let merging this PR get too held up on this point. If there are concerns about merging this and then later altering it, then perhaps we could mark it as unstable like we do with new metadata versions and call the attr Does that work for everybody here? Should we mark as "unstable" eg like above or can we review and merge and still tweak it later without this? |
|
yes |
| ) -> Result<(), QueryDispatchError>; | ||
| } | ||
|
|
||
| impl DispatchQuery for () { |
There was a problem hiding this comment.
dq: but why we need to implement it for unit type?
| let queries = view_fns.iter().map(|view_fn| { | ||
| let query_struct_ident = view_fn.query_struct_ident(); | ||
| let name = &view_fn.name; | ||
| let args: Vec<TokenStream> = vec![]; // todo |
|
|
||
| /// Metadata of a runtime method. | ||
| #[derive(Clone, PartialEq, Eq, Encode, Debug)] | ||
| pub struct QueryMetadataIR<T: Form = MetaForm> { |
There was a problem hiding this comment.
dq: but don't we want to have space for version/deprecation info on the query to allow iteration on already defined apis without explicitly removing them or redefining the type signature?
Or the intention is just o keep them all and fully rely on QueryId for the resolution?
Closes #216.
This PR allows pallets to define a
view_functionsimpl like so:QueryIdEach view function is uniquely identified by a
QueryId, which for this implementation is generated by:twox_128(pallet_name) ++ twox_128("fn_name(fnarg_types) -> return_ty")The prefix
twox_128(pallet_name)is the same as the storage prefix for pallets and take into account multiple instances of the same pallet.The suffix is generated from the fn type signature so is guaranteed to be unique for that pallet impl. For one of the view fns in the example above it would be
twox_128("get_value_with_arg(u32) -> Option<u32>"). It is a known limitation that only the type names themselves are taken into account: in the case of type aliases the signature may have the same underlying types but a different id; for generics the concrete types may be different but the signatures will remain the same.The existing Runtime
Calldispatchables are addressed by their concatenated indicespallet_index ++ call_index, and the dispatching is handled by the SCALE decoding of theRuntimeCallEnum::PalletVariant(PalletCallEnum::dispatchable_variant(payload)). Forview_functionsthe runtime/pallet generated enum structure is replaced by implementing theDispatchQuerytrait on the outer (runtime) scope, dispatching to a pallet based on the id prefix, and the inner (pallet) scope dispatching to the specific function based on the id suffix.Future implementations could also modify/extend this scheme and routing to pallet agnostic queries.
Executing externally
These view functions can be executed externally via the system runtime api:
XCQCurrently there is work going on by @xlc to implement
XCQwhich may eventually supersede this work.It may be that we still need the fixed function local query dispatching in addition to XCQ, in the same way that we have chain specific runtime dispatchables and XCM.
I have kept this in mind and the high level query API is agnostic to the underlying query dispatch and execution. I am just providing the implementation for the
view_functiondefinition.Metadata
Currently I am utilizing the
customsection of the frame metadata, to avoid modifying the official metadata format until this is standardized.vs
runtime_apiThere are similarities with
runtime_apis, some differences being:QueryIdwill change if the signature changes.Calling from contracts
Future work would be to add
weightannotations to the view function queries, and a host function topallet_contractsto allow executing these queries from contracts.TODO
runtime_api