CLI Tool: Strip metadata to only include certain pallets

[jsdw]

A common use case of Subxt is to generate client logic to talk to specific pallets.

Currently the CLI tool downloads the entirety of the metadata from a node.

It'd be nice if we could instruct the client to strip metadata (either direct from a node or provided) to include only the information necessary for specific pallets to be called. This would allow for more compact codegen and smaller metadata files to be kept around among other things.

[jsdw]

Ah, I hit a snag having a go at stripping metadata to only include certain pallets. Well, two things:

1. PortableRegistry is pretty opaque and I can't see a way offhand to modify it, ie remove a bunch of types from it. I could leave it in, but I think it accounts for a large chunk of the bytes associated with a pallet.
2. Since the ID of a type is its location in a Vec, and I'd be shuffling things around in the vec to remove types from it, many of the IDs will end up changing. So, I'd also need to then go around the rest of the metadata swapping type IDs out for their new ones. Doable but a bit annoying and means the stripped metadata would deviate from the original a bunch more than hoped.

My options:

1. Only strip pallets data; leave all the type info in there even if it's no longer used.
2. PR to scale-info to expose some way to get at/alter the PortableRegistry types and tedious re-numbering of IDs.
3. I could probably do something awful involving serializing to JSON, stripping down and deserializing back to PortableRegistry..
4. Admit defeat and don't try to strip type info from metadata at all, taking away valuable learnings from the endeavour.

I'm tempted to just leave it be with 4 for now, since 1 is easy but has marginal gains and the others are rather fiddly and is all work that will need duplicating when V15 comes along. Perhaps this is something that should exist in Substrate, anyway?

[ascjones]

For 2., there is an open PR to allow dynamic construction of a PortableRegistry: paritytech/scale-info#164.

One way to do it that comes to mind would be to rebuild a new PortableRegistry with only those root types:

1. Strip pallets data, collect a list of all type ids used.
2. Implement a method to retain all but those root types e.g. impl PortableRegistry { pub fn retain(&self, ids: &[u32]) -> Result<(Self, Vec<u32>)> { todo!() } }
3. Inside retain:

• Create an empty PortableRegistry
• initialize a HashMap<u32, u32> id_map old_id => new_id
• Recursive function copy_type -> u32
  • if id_map already has an entry,
    • return new_id from mapping
  • else
    • traverse type def, recursively calling into copy_type
    • insert new type def into new PortableRegistry
    • return new id
• for each id in ids
  • call copy_type (returns new type id)
• return new PortableRegistry and new type ids

4. Replace all type ids in pallet metadata with new type ids, might be easy because all type ids are public e.g. https://github.com/paritytech/frame-metadata/blob/main/frame-metadata/src/v14.rs#L372

I think that is feasible to do, and even involves a fun programming problem. But obviously extra effort compared to "doing nothing" in 4.

Perhaps this is something that should exist in Substrate, anyway?

Possibly, e.g. supply a list of pallets to state_getMetadata which is passed into the runtime get_metadata, and then only register those pallets whose names are supplied. So maybe that is worth investigating. The tradeoff I suppose is just adding complexity to substrate. It would save on bandwidth overall so rpc nodes would only serve the minimum amount of metadata required.

[jsdw]

I like the Substrate approach personally. Another benefit of implementing stripping there, aside from the bandwidth savings, is that it only needs to be implemented once, rather than reimplemented on the periphery in various tools (like sidecar and subxt) that want to strip it.

This matters too when we update to V15 and V16 etc; feels nicer to keep the work done in one place if possible.

[jsdw]

paritytech/substrate#12370 Was a suggestion/PR to add this to Substrate, so that can be tracked there.

Failing that, The options I think are:

• Don't strip anything at all
• Strip at the codegen level; so full metadata bundle but only generate what we need.
• Strip in frame-metadata/scale-info (doing it here rather than Subxt allows others to benefit).

[lexnv]

I created a POC in subxt to gather some data regarding a single pallet metadata (here).

To achieve this, the code recursively examines all type IDs required to represent a pallet.

Pallet Type IDs Only

The first number indicates the number of type IDs necessary to describe the pallet, relative to the total number of type IDs in the PortableRegistry.
However, this calculation does not consider the type IDs needed to represent extrinsics or the metadata's runtime type.

Stripped   0 vs full 756 for pallet [Historical]
Stripped   3 vs full 756 for pallet [Authorship]
Stripped   3 vs full 756 for pallet [RootTesting]
Stripped   4 vs full 756 for pallet [Mmr]
Stripped   4 vs full 756 for pallet [Timestamp]
Stripped   5 vs full 756 for pallet [RandomnessCollectiveFlip]
Stripped   6 vs full 756 for pallet [AuthorityDiscovery]
Stripped   7 vs full 756 for pallet [AssetTxPayment]
Stripped   7 vs full 756 for pallet [TransactionPayment]
Stripped   8 vs full 756 for pallet [Remark]
Stripped   9 vs full 756 for pallet [Glutton]
Stripped   9 vs full 756 for pallet [Pov]
Stripped  13 vs full 756 for pallet [TechnicalMembership]
Stripped  14 vs full 756 for pallet [Indices]
Stripped  15 vs full 756 for pallet [TransactionStorage]
Stripped  16 vs full 756 for pallet [Preimage]
Stripped  16 vs full 756 for pallet [StateTrieMigration]
Stripped  16 vs full 756 for pallet [Vesting]
Stripped  17 vs full 756 for pallet [ChildBounties]
Stripped  17 vs full 756 for pallet [Offences]
Stripped  17 vs full 756 for pallet [VoterList]
Stripped  20 vs full 756 for pallet [Bounties]
Stripped  20 vs full 756 for pallet [MessageQueue]
Stripped  20 vs full 756 for pallet [RankedCollective]
Stripped  20 vs full 756 for pallet [Tips]
Stripped  20 vs full 756 for pallet [Treasury]
Stripped  22 vs full 756 for pallet [Elections]
Stripped  22 vs full 756 for pallet [FastUnstake]
Stripped  23 vs full 756 for pallet [Session]
Stripped  24 vs full 756 for pallet [Assets]
Stripped  24 vs full 756 for pallet [Balances]
Stripped  27 vs full 756 for pallet [Society]
Stripped  29 vs full 756 for pallet [ConvictionVoting]
Stripped  29 vs full 756 for pallet [Nis]
Stripped  31 vs full 756 for pallet [Grandpa]
Stripped  33 vs full 756 for pallet [Uniques]
Stripped  36 vs full 756 for pallet [Contracts]
Stripped  36 vs full 756 for pallet [ImOnline]
Stripped  41 vs full 756 for pallet [Babe]
Stripped  41 vs full 756 for pallet [NominationPools]
Stripped  56 vs full 756 for pallet [Staking]
Stripped  68 vs full 756 for pallet [Identity]
Stripped  78 vs full 756 for pallet [Nfts]
Stripped  91 vs full 756 for pallet [ElectionProviderMultiPhase]
Stripped 314 vs full 756 for pallet [Recovery]
Stripped 316 vs full 756 for pallet [Lottery]
Stripped 317 vs full 756 for pallet [Alliance]
Stripped 317 vs full 756 for pallet [Sudo]
Stripped 317 vs full 756 for pallet [Utility]
Stripped 320 vs full 756 for pallet [AllianceMotion]
Stripped 320 vs full 756 for pallet [Council]
Stripped 320 vs full 756 for pallet [Multisig]
Stripped 320 vs full 756 for pallet [TechnicalCommittee]
Stripped 321 vs full 756 for pallet [Whitelist]
Stripped 322 vs full 756 for pallet [Scheduler]
Stripped 325 vs full 756 for pallet [Proxy]
Stripped 327 vs full 756 for pallet [Democracy]
Stripped 328 vs full 756 for pallet [RankedPolls]
Stripped 329 vs full 756 for pallet [Referenda]
Stripped 417 vs full 756 for pallet [System]

Full Metadata for a Pallet Including Extrinsic and Runtime Type IDs

This complete metadata contains all the necessary type IDs for working with a single pallet and can be utilized in Subxt, among other things.

Stripped 321 vs full 756 for pallet [Authorship]
Stripped 321 vs full 756 for pallet [Historical]
Stripped 321 vs full 756 for pallet [Mmr]
Stripped 321 vs full 756 for pallet [RootTesting]
Stripped 321 vs full 756 for pallet [Timestamp]
Stripped 322 vs full 756 for pallet [AssetTxPayment]
Stripped 323 vs full 756 for pallet [AuthorityDiscovery]
Stripped 323 vs full 756 for pallet [RandomnessCollectiveFlip]
Stripped 323 vs full 756 for pallet [Remark]
Stripped 324 vs full 756 for pallet [Glutton]
Stripped 324 vs full 756 for pallet [Indices]
Stripped 324 vs full 756 for pallet [Pov]
Stripped 324 vs full 756 for pallet [StateTrieMigration]
Stripped 324 vs full 756 for pallet [TechnicalMembership]
Stripped 324 vs full 756 for pallet [Tips]
Stripped 324 vs full 756 for pallet [TransactionPayment]
Stripped 326 vs full 756 for pallet [ChildBounties]
Stripped 326 vs full 756 for pallet [Elections]
Stripped 326 vs full 756 for pallet [TransactionStorage]
Stripped 326 vs full 756 for pallet [Vesting]
Stripped 327 vs full 756 for pallet [Preimage]
Stripped 327 vs full 756 for pallet [Recovery]
Stripped 327 vs full 756 for pallet [Session]
Stripped 327 vs full 756 for pallet [Treasury]
Stripped 327 vs full 756 for pallet [VoterList]
Stripped 328 vs full 756 for pallet [Bounties]
Stripped 328 vs full 756 for pallet [Grandpa]
Stripped 329 vs full 756 for pallet [Lottery]
Stripped 329 vs full 756 for pallet [Offences]
Stripped 329 vs full 756 for pallet [Sudo]
Stripped 329 vs full 756 for pallet [Utility]
Stripped 330 vs full 756 for pallet [Alliance]
Stripped 330 vs full 756 for pallet [MessageQueue]
Stripped 330 vs full 756 for pallet [RankedCollective]
Stripped 331 vs full 756 for pallet [Assets]
Stripped 332 vs full 756 for pallet [AllianceMotion]
Stripped 332 vs full 756 for pallet [Balances]
Stripped 332 vs full 756 for pallet [Council]
Stripped 332 vs full 756 for pallet [FastUnstake]
Stripped 332 vs full 756 for pallet [Multisig]
Stripped 332 vs full 756 for pallet [TechnicalCommittee]
Stripped 332 vs full 756 for pallet [Uniques]
Stripped 333 vs full 756 for pallet [Identity]
Stripped 333 vs full 756 for pallet [Society]
Stripped 333 vs full 756 for pallet [Whitelist]
Stripped 334 vs full 756 for pallet [Nis]
Stripped 334 vs full 756 for pallet [Scheduler]
Stripped 335 vs full 756 for pallet [ConvictionVoting]
Stripped 336 vs full 756 for pallet [Contracts]
Stripped 336 vs full 756 for pallet [ElectionProviderMultiPhase]
Stripped 336 vs full 756 for pallet [ImOnline]
Stripped 337 vs full 756 for pallet [Babe]
Stripped 337 vs full 756 for pallet [Proxy]
Stripped 340 vs full 756 for pallet [Democracy]
Stripped 341 vs full 756 for pallet [NominationPools]
Stripped 341 vs full 756 for pallet [RankedPolls]
Stripped 342 vs full 756 for pallet [Referenda]
Stripped 347 vs full 756 for pallet [Nfts]
Stripped 349 vs full 756 for pallet [Staking]
Stripped 429 vs full 756 for pallet [System]

Conclusion

By removing unnecessary information from the metadata, we can create a smaller metadata binary that other teams can use specifically for working with one pallet. Based on the previous calculations, this approach may result in a metadata binary that is roughly half the size of the original. As such, it may be worthwhile to explore this option further.