Add tool for auto generating runtime code from metadata

[serban300]

Addresses: #1781 + TODOs / comments in the code like for example this one

Generating the wrapped runtime code for RialtoParachain using subxt-codegen instead of manually defining it.

Posting this only as a draft to gather feedback on the high-level idea.

I would say there are 2 main issues here:

1. subxt-codegen generates shadow structures for everything that is in the runtime (for example everything that's in sp_runtime, bp_millau, etc). There are 2 solutions to this:

   • using std::mem::transmute (see commit Use the generated runtime for rialto parachain )
   • defining type_substitutes for subxt-codegen (see commit Avoid using std::mem::transmute ) . It's possible that we'll need to do this for a lot of structures.

2. subxt-codegen generates a lot more code than a manually defined runtime wrapper would contain (and a lot of things we don't need). This doesn't seem to increase the compile time by much, just a couple of seconds. But other runtimes (Polkadot for examples) might be even bigger. Not sure. Also from what I understand subxt-codegen might add some features for generating less code in the future (specifying only a couple of Pallets for which to generate code, specifyong only certain sections i.e. Calls / Storage / Events ). So this might be partially addressed in the future.

However overall to me it seems like this might be usable.

@svyatonik @bkontur @acatangiu do you think it would be worth adding the possibility of generating wrapped runtimes with this approach instead of manually defining them ?

[svyatonik]

IMO the generated code looks good - we don't need to look at it anyway (most of time). But let's talk about when to use the codegen - are you planning to do that manually/with CI, right? Like when chain upgrade happens? Or do you actually want to do it during build?

Also - iiuc we don't need this code most of time, only when we prepare some transaction. So the rest of code will still use data types, defined in our code (types from bp_header_chain instead of api::runtime_types::bp_header_chain). That looks perfect.

Also - it looks like a good prototype of what we need to do when one of bridge hubs (and possibly relay chains) is upgraded. Like part of release docker publish needs to be this step that regenerates runtime types and checks compatibility.

[serban300]

@svyatonik thanks for the feedback !

But let's talk about when to use the codegen - are you planning to do that manually/with CI, right? Like when chain upgrade happens? Or do you actually want to do it during build?

Yes, I would say manually when needed. I would try to add a special build command/flag for this. Not sure if possible. I have to check what are the options.

Also - iiuc we don't need this code most of time, only when we prepare some transaction. So the rest of code will still use data types, defined in our code (types from bp_header_chain instead of api::runtime_types::bp_header_chain). That looks perfect.

Yes, exactly. We would only use this code to build calls mostly. And we can also access storage values with it. But apart from that we'll use the data types defined in our code.

Also - it looks like a good prototype of what we need to do when one of bridge hubs (and possibly relay chains) is upgraded. Like part of release docker publish needs to be this step that regenerates runtime types and checks compatibility.

Sounds good. I'll try to polish it a bit and see if we can regenerate the code only on some custom build command so that it could be used as a step in the upgrade process.

[bkontur]

@serban300
looks cool,
so now you run node locally (&"http://localhost:20433") and download metadata, which you use to generate runtime_codegen.rs,

I dont know if it is even possible now, but maybe it would be cool, if we could (re)generate this runtime_codegen.rs from the downloaded wasm from the cumulus releases: https://github.com/paritytech/cumulus/releases/tag/parachains-v9360
maybe with srtool - but I dont know if it is possible to extract metadata from the runtime wasm

generate_runtime_api_from_bytes
generate_runtime_api_from_path

[serban300]

@serban300 looks cool, so now you run node locally (&"http://localhost:20433") and download metadata, which you use to generate runtime_codegen.rs,

I dont know if it is even possible now, but maybe it would be cool, if we could (re)generate this runtime_codegen.rs from the downloaded wasm from the cumulus releases: https://github.com/paritytech/cumulus/releases/tag/parachains-v9360 maybe with srtool - but I dont know if it is possible to extract metadata from the runtime wasm

generate_runtime_api_from_bytes generate_runtime_api_from_path

Yes, I don't know if it's possible. But it would be nice. I'll try.

[bkontur]

@serban300 looks cool, so now you run node locally (&"http://localhost:20433") and download metadata, which you use to generate runtime_codegen.rs,
I dont know if it is even possible now, but maybe it would be cool, if we could (re)generate this runtime_codegen.rs from the downloaded wasm from the cumulus releases: https://github.com/paritytech/cumulus/releases/tag/parachains-v9360 maybe with srtool - but I dont know if it is possible to extract metadata from the runtime wasm
generate_runtime_api_from_bytes generate_runtime_api_from_path

Yes, I don't know if it's possible. But it would be nice. I'll try.

looks like, this should be possible with subxt-cli see here

[serban300]

@svyatonik @bkontur

I polished this PR and transformed it into a CLI tool for generating runtime code for metadata. Also added the posibility to generate the runtime code from the wasm binary as suggested by @bkontur .

PTAL when you have time.

[bkontur]

cool, that it is possible with cli for wasm file,
I could imagine, for example:

• Cumulus releases new runtimes
• we can add to Cumulus some trigger (auto or manual) that will ping our CI, that will run this cli tool and will create a PR with new codegen-runtime.rs
• so we can see directly from PR, that new codegen is relevant for redeploying relayer or not, if not we just close PR and dont care, if relevant, we will wait for version guard and then just merge PR and release :)
  • also we can have some test for encodings, if anything is changed

I think, if we want to go this way, I would try it for Rococo/Wococo/Kusama/Polkadot stuff asap,
the only disadvatage is the bit codegen-runtime.rs where we need just small stuff from this file

[svyatonik]

Looks very promising! Thanks!

[svyatonik]

Not an issue - just a question. I can see that the generated code has something like mod sp_runtime. That's why we need those substitutions, right? But why only for several types? Are they types that we use in relay?

[serban300]

Yes, exactly. I added type subsitutites only for the types that we use to build the indirect calls. In order to avoid using std::mem::transmute().

[serban300]

Trying to make spellcheck to ignore the codegen files

[serban300]

I added a small change to avoid autogenerated code files when running cargo spellcheck. Now spellcheck will check all the rust files except for the ones named codegen_runtime.rs. Not sure if before it was checking only the rust files, but it seemed so.

@svyatonik could you PTAL on this change ?

[svyatonik]

afaik it was only checking rs files, right. Everything is fine, thank you!

[svyatonik]

@serban300 Is that intended or just some accidental change? I mean - we now have many duplicate crates here (in lock file) - half of those are referencing Substrate master and the other half referencing this tag (ag=monthly-2022-07). I was going to update Substrate refs && found this :) Could fix in my PR if that was an accidental change

[serban300]

@svyatonik sorry, yes it was by mistake. Thanks !