chain-spec: getting ready for native-runtime-free world

[michalkucharczyk]

This PR prepares chains specs for native-runtime-free world.

This PR has following changes:

• substrate:

  • adds support for:
    • JSON based GenesisConfig to ChainSpec allowing interaction with runtime GenesisBuilder API.
    • interacting with arbitrary runtime wasm blob to chain-spec-builder command line util,
  • removes code from system_pallet
  • adds code to the ChainSpec
  • deprecates ChainSpec::from_genesis, but also changes the signature of this method extending it with code argument. ChainSpec::builder() should be used instead.

• polkadot:

  • all references to RuntimeGenesisConfig in node/service are removed,
  • all (kusama|polkadot|versi|rococo|wococo)_(staging|dev)_genesis_config functions now return the JSON patch for default runtime GenesisConfig,
  • ChainSpecBuilder is used, ChainSpec::from_genesis is removed,

• cumulus:

  • ChainSpecBuilder is used, ChainSpec::from_genesis is removed,
  • JSON patch configuration used instead of RuntimeGenesisConfig struct in all chain specs.

Note on testing: all old chain-spec generating functions were copied to legacy test mods in order to make sure that raw storage generated by new (json) and old (runtime genesis config struct) functions is exactly the same.

---

code moved into ChainSpec

In raw formats the code is one of the key-pair entry (0x3a636f6465)
Example:

{
  "name": "TestName",
  "id": "test_id",
  "chainType": "Local",
   ...
  "genesis": {
    "raw": {
      "top": {
        "0x3a636f6465": "0x010101"
      },
      "childrenDefault": {}
    }
  },
}

In plain chain specs using the patch or full JSON config, the code hex encoded blob shall be provided under the genesis::runtimeGenesis::code path, example:

{
  "name": "TestName",
  "id": "test_id",
  "chainType": "Local",
  "bootNodes": [],
  "telemetryEndpoints": null,
  "protocolId": null,
  "properties": null,
  "codeSubstitutes": {},
  "genesis": {
    "runtimeGenesis": {
      "patch": {
        "babe": {
          "epochConfig": {
            "allowed_slots": "PrimaryAndSecondaryPlainSlots",
            "c": [
              7,
              10
            ]
          }
        },
        "substrateTest": {
          "authorities": [
            "5CiPPseXPECbkjWCa6MnjNokrgYjMqmKndv2rSnekmSK2DjL",
            "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
          ]
        }
      },
      "code": "0x0..."
    }
  }
}

JSON based GenesisConfig in ChainSpec:

Patch

The ChainSpec can now be built using genesis config JSON patch (which contains some key-value pairs meant to override runtime's genesis config default values). This can be achieved with with_genesis_patch method of the builder:

let chain_spec = ChainSpec::builder()
    .with_code(substrate_test_runtime::wasm_binary_unwrap())
    .with_genesis_config_patch(json!({
        "babe": {
            "epochConfig": {
                "c": [
                    7,
                    10
                ],
                "allowed_slots": "PrimaryAndSecondaryPlainSlots"
            }
        }
    }))
    .build()

Resulting ChainSpec instance can be converted to raw version of chain spec JSON file. This was not changed and can be done with chain_spec.as_json(true) method. Sample output is here. The runtime's GenesisBuilder::build_config API is called during this conversion.

The ChainSpec instance can also be written to chain spec JSON file in human readable form. The resulting chain spec file will contain the genesis config patch (partial genesis config). Sample output is here

Full Config

It is also possible to build ChainSpec using full genesis config JSON (containing all the genesis config key-value pairs). No defaults will be used in this approach. The sample code is as follows:

let chain_spec = ChainSpec::builder()
    .with_code(substrate_test_runtime::wasm_binary_unwrap())
    .with_genesis_config(json!({
        "babe": {
            "authorities": [
                [ "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", 1 ],
                ...
            ],
            "epochConfig": {
                "allowed_slots": "PrimaryAndSecondaryPlainSlots",
                "c": [ 3, 10 ]
            }
        },
        "balances": {
            "balances": [
                [ "5D34dL5prEUaGNQtPPZ3yN5Y6BnkfXunKXXz6fo7ZJbLwRRH", 100000000000000000 ],
            ],
            ...
        },
        "substrateTest": {
            "authorities": [
                "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
                ...
            ]
        },
        "system": {}
    }))
    .build()

Again, resulting ChainSpec instance can be converted to the raw version of chain spec JSON file (which involves calling GenesisBuilder::build_config runtime method) .
It can be also stored in human readable version, sample output here.

chain-spec-builder util

New commands allowing to interact with arbitrary WASM runtime blob were added. Use cases are as follows:

Get default config from runtime

Queries the default genesis config from the provided runtime.wasm and uses it in the chain spec. Can also store runtime's default genesis config in given file (-d):

chain-spec-builder runtime -r runtime.wasm default -d /dev/stdout

Note: GenesisBuilder::create_default_config runtime function is called.

Generate raw storage chain spec using genesis config patch

Patches the runtime's default genesis config with provided patch.json and generates raw storage (-s) version of chain spec:

chain-spec-builder runtime -s -r runtime.wasm patch -p patch.json

Note: GenesisBuilder::build_config runtime function is called.

Generate raw storage chain spec using full genesis config

Build the chain spec using provided full genesis config json file. No defaults will be used:

chain-spec-builder runtime -s -r runtime.wasm full -c full-genesis-config.json

Note: GenesisBuilder::build_config runtime function is called.

Generate human readable chain spec using genesis config patch

chain-spec-builder runtime -r runtime.wasm patch -p patch.json

Note: No runtime API is called.

Generate human readable chain spec using full genesis config

chain-spec-builder runtime -r runtime.wasm full -c full-genesis-config.json

Note: No runtime API is called.

Some extra utils:

• verify: allows to verify if human readable chain spec is valid (precisely: all required fields in genesis config are initialized),
• edit, allows to:
  • update the code in given chain spec,
  • convert given chain spec to the raw chain spec,

Some open questions/issues:

• naming .with_no_genesis_defaults + in chain spec json keys: JsonPatch / JsonFull,
• GenesisSource source for patch and full config.
• support for New/Generate commands in `chain-spec-builder? (IMO we can remove them).

Step towards: #25

[michalkucharczyk]

Continuation of work done in: paritytech/substrate#14562

[michalkucharczyk]

bot fmt

[nazar-pc]

I no longer see a way to do this in more type-safe way without json!, which is very annoying. Even though generated GenesisConfig is JSON-serializable, so I constructed it explicitly and then tried to serialize, serialization failed for me with this non-descriptive error:

Error("number out of range", line: 0, column: 0)

And it comes from pallet-balances 😞

Any recommendations?

[nazar-pc]

Well, json! also doesn't work with u128::MAX / 2 balance, while ChainSpec::from_genesis worked. Created an issue about this: #2963

[michalkucharczyk]

Can you try arbitrary_precision feature on serde_json crate in minimal runtime? (just trying out of my head, remember that I had similar issue).

[michalkucharczyk]

Please see the proposal of fix:
https://github.com/michalkucharczyk/polkadot-sdk/tree/mku-u128-balances-fix (michalkucharczyk@879917e)

./minimal-node build-spec --chain dev works fine (also --raw):

...
            [
              "5Fxune7f71ZbpP2FoY3mhYcmM596Erhv1gRue4nsPwkxMR4n",
              170141183460469231731687303715884105727
            ],
            [
              "5CUjxa4wVKMj3FqKdqAUf7zcEMr4MYAjXeWmUf44B41neLmJ",
              170141183460469231731687303715884105727
            ]
...

[nazar-pc]

If that works - great (it is a bit difficult to test due to a series of downstream patches on top of Substrate). Why did it work with ChainSpec::from_genesis though? It should have unified features of serde_json if one of the dependencies enabled arbitrary_precision.

[michalkucharczyk]

I think no deps are enabling this feature in substrate. It is runtime serializing big numbers, so it must be enabled there. Just guessing - maybe we should add it to frame somewhere.

ChainSpec::from_genesis assumed that all rust types (including RuntimeGenesisConfig) were available on native side of node (which is no longer a case as we want to get rid of this dep). So there was no json boundary to cross.

[michalkucharczyk]

(it is a bit difficult to test due to a series of downstream patches on top of Substrate

This single line in runtime/Cargo.tom should do the job:

serde_json = { version = "1.0.108", default-features = false, features = ["alloc", "arbitrary_precision"] }