Add more dynamic decoding support to subxt

[jsdw]

Currently subxt is optimised towards the compile-time oriented codegen usage; we generate an interface and then use that interface to query compatible nodes. This has many advantages (you benefit from compile-time errors, IDE support, performance and more). The main downside is that the compile-time generated interface is unable to work across different, differing nodes/chains in a general way (parts of it may continue to work, and others will not).

Having the ability to dynamically query nodes across different chains can be very useful in some cases, for example exploratory tooling which wants to interrogate the available interfaces, or tooling akin to polkagot.js which needs to be able to provide a dynamic interface to interact with arbitrary substrate based chains.

These two directions (compiletime, static codegen and dynamic, runtime interaction) are not mutually exclusive either. They would both benefit from the same client (JSONRPC transport/API) code for instance. Another overlap is that subxt is used to provide a static interface to a single pallet for the cargo-contracts tool, which means being able to dynamically decode Events emitted from other pallets that it does not know about (rather than erroring because it's unable to statically decode them).

So, my proposal is that we work towards something like the following:

• Three user facing library crates (names just placeholders); subxt-static (which itself uses subxt-macro and subxt-codegen), subxt-dynamic and subxt-client.
  • subxt-static is just subxt as it stands (with a little runtime decoding moved out of it).
  • subxt-dynamic provides similar functionality to the above, but at runtime. It contains the logic needed to dynamically encode and decode things.
  • subxt-client splits out the JSONRPC client API as it currently stands, so that it can be shared between the above.
  • (I wonder whether we would want to share the metadata wrapping in eg subxt-metadata, too; perhaps not initially)
• These crates should interop nicely; we can move back and forth between them where it makes sense. As an example, when we fetch the SCALE encoded Event blob, the cargo-contracts use case requires that we can dynamically decode (and throw away) any events we don't care about until we get to the one we're looking for (with eg find_event::<MyEvent>()).

[ascjones]

Related, I also have some similar code in my cargo-contract PR which is dynamically decoding events for display on the command line: https://github.com/paritytech/cargo-contract/blob/9bb545d77c13c3812f988635c90e5ce04acc95e1/src/cmd/extrinsics/transcode/decode.rs

[jsdw]

Also related: desub-current (https://github.com/paritytech/desub/tree/master/desub-current) has a bunch of dynamic decoding bits, including code quite similar to the subxt dynamic decoding. I'd hope that if we did this sort of split, desub-current would (if we care enough) migrate to relying on subxt-dynamic for some of its decoding stuff.

[jsdw]

Here's my shot at putting together a list of dynamic decoding/encoding features that I'd like to see supported in the subxt crates. All APIs are just thoughts/examples and may well change significantly. I've also stated the goals that I'm aiming towards with these.

Are there any other areas that you'd like me to think about when looking into this dynamic/runtime side of things? Feedback and use cases that you'd like supported but aren't currently are very welcome!

Goals

To be able to use subxt as the foundation on which something like a Rust equivalent to Polkadot.js could be built.
This means:

• Being able to connect and interact with any substrate based node (running V14 metadata).
• Being able to inspect the node "API" at runtime, eg find out which calls/storage exist on it.
• Being able to interact with the node at runtime, without any reliance on static codegen.

An example of a tool I'd like to be able to build on top of this is a CLI tool that will let you build and submit
a call, or inspect storage. The tool would guide you through the entire process so that you can learn what's possible
and build up to the correct transaciton/storage request starting from zero knowledge, and then submit it to the node
and see some representation of the result handed back.

Features

Add a runtime Value type to represent the data we can send/get back.

See desub-current for a Value type. This will come in handy for the below features.

Must haves:

• We want to be able to go from (Value, Metadata) -> SCALE bytes.
• We want to be able to go from (SCALE bytes, Metadata) -> Value.

(The desub-current code covers the (SCALE bytes, Metadata) -> Value case, but not the encode one).

Nice to haves:

• The Value type can optionally also implement Deserializer so that we can
  transform from Values to concrete types where they align.
• Also Deserialize so that we can transform from eg JSON to Value types where applicable.
• Also Serialize so that we can go from Value type to things like JSON.

(The desub-current Value implements these serde traits already.)

Metadata API: Ths ability to explore available storage/calls/events/constants.

To satisfy our goal, we want to be able to inspect, at runtime, the calls that we can make to a
node as well as the available storage and constants. This would allow us to show the available
calls/storage in a UI/CLI tool, and provide help as to what the shape of each parameter should be,
so that we know how to correctly encode things when we come to interacting with them.

• Which pallets exist?
• For a pallet, which transactions can I make, and which storage entries and such exist?
• For transactions; how many and what shape are the parameters?
• For storage entries, which names, parameters and types are required?
• What is the type of events that I can get back?
• What are the constants available and their values?

All of these basically involve providing a nice interface to the metadata.

The ability to dynamically build and submit a transaction, receiving back a dynamic result.

Once we can explore the available calls, we'd like to be able to actually interact with
them. Using the metadata to guide us, we should know at runtime what shape a transaction should
take, and we can use a dynamic Value type to allow us to construct this at runtime.

use subxt::dynamic::{ Value, tx };

let signer = PairSigner::new(AccountKeyring::Alice.pair());
let dest: MultiAddress<AccountId32> = AccountKeyring::Bob.to_account_id().into();

// `tx` is just a helper to provide Value::Variant types of the right shape.
let transaction = tx("Balances", "Transfer")
    // Anything that can be serialized to a Value type can be provided here, though
    // perhaps we can also accept anything that can be SCALE encoded and store that in some
    // opaque SCALE encoded blob type? (perhaps .scale_param and .value_param). Needs thought;
    // SCALE types would not be debuggable and inspectable in the same way, so I'd start without.
   .param(dest)
   .param(10_000u128)
   // to allow for things like UIs and CLI tools, a dynamic Value type (see desub-current)
   // would enable us to pass values without knowing the type. Encoded with the help of Metadata.
   .param(Value::variant("Foo", vec![Value::primitive(10u8)]))
   .build();

// transaction is essentially a Value::Variant type at this point, outer variant Balances,
// inner variant Transfer with whatever params provided.

let val = api
   .dynamic()
   .tx(transaction)
   // We can check whether the transaction is valid before trying to send it
   .check()?
   // Re-use the same logic for submitting and such as much as practical.
   .sign_and_submit_then_watch(&signer)
   .await?
   .wait_for_finalized_success()
   .await?;

// val would be a dynamic Value type. If we rely on features from desub-current's Valeu type,
// we can Deserialize this into any compatible static type, or just display it in whatever format
// we prefer.

Dyanmic storage queries

As with building dynamic transactions, let's expose an API to allow us to construct dynamic
storage requests; this will likely also lean on the Value type for the parameters, combined with
hasher types to define which hashing will take place.

use subxt::dynamic::{ Value, storage };

// Similarly to transacitons, perhaps we have some type
// which embeds the Value params along with the hashers
// and such needed to make a storage request
let storage_entry = storage("Staking", "Bonded")
    .param(something, Hasher::TwoX_128)
    .param(Value::primitive(123u8), Hasher::Blake2_256);

// We get back a Value representing whatever is in storage.
let val = api
   .dynamic()
   .storage(storage_entry)
   // As with transactions, some way to check this against current
   // metadata might be useful?
   .check()?
   .submit()
   .await?

Dynamic event decoding

We can already iterate over events dynamically to find ones of interest and ignore others, which requires
dynamically decoding them. Right now we away the results of doing that.

• It would be nice to be able to decode events to some dynamic Value type, enabling us to work with them
  or present them dynamically. (Can we have both, but avoid duplicating the decode logic?)
• As well as getting back all events as Value types, can we search events for their type name eg
  .filter_events_dynamically("Balances", "Withfraw"), returning Value types.
• All of this should be possible whether asking for events from a transaction, or subscribing to events.

[ascjones]

We want to be able to go from (Value, Metadata) -> SCALE bytes.
We want to be able to go from (SCALE bytes, Metadata) -> Value.

I'm particularly interested in this part. It would be good to replace the similar code I have in cargo-contract which I mentioned previously.