Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

GenesisBuilder runtime API #14131

Merged
merged 32 commits into from
Jun 26, 2023
Merged

GenesisBuilder runtime API #14131

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
926f828
GenesisConfigBuilder: preliminary API proposal
michalkucharczyk May 12, 2023
f5b4cbe
fmt
michalkucharczyk May 12, 2023
d24fbba
comment removed
michalkucharczyk May 12, 2023
139341a
build_default_config removed
michalkucharczyk May 15, 2023
587770b
Update client/genesis-builder/src/lib.rs
michalkucharczyk May 16, 2023
78f467b
config -> gensis_config
michalkucharczyk May 16, 2023
75c1f73
GenesisConfigBuilder: helper added
michalkucharczyk May 17, 2023
5756776
moved to primitives
michalkucharczyk May 18, 2023
1b3f8ab
licesne changed to apache-2.0
michalkucharczyk May 18, 2023
0b3e084
Cargo.toml: name/path to genesis-builder updated
michalkucharczyk May 18, 2023
fcd2ee9
helper removed
michalkucharczyk May 18, 2023
bed8d05
Merge remote-tracking branch 'origin/master' into mku-genesis-builder…
michalkucharczyk May 30, 2023
5ae93f6
sp-sd version bumped
michalkucharczyk May 30, 2023
77c6504
Merge remote-tracking branch 'origin/master' into mku-genesis-builder…
May 31, 2023
cc91d46
sp-std bump
michalkucharczyk May 31, 2023
679a970
naming + new function
michalkucharczyk Jun 5, 2023
aa1f77c
fix
michalkucharczyk Jun 6, 2023
43a5d41
build_from_patch_json -> build_with_patch
michalkucharczyk Jun 6, 2023
82bb745
fix
michalkucharczyk Jun 6, 2023
8ca9b44
Merge remote-tracking branch 'origin/master' into mku-genesis-builder…
Jun 6, 2023
f9926bd
Cargo.lock updated
michalkucharczyk Jun 6, 2023
706f3f2
readme: license updated
michalkucharczyk Jun 7, 2023
b35c443
Update primitives/genesis-builder/src/lib.rs
michalkucharczyk Jun 7, 2023
14a251b
Update primitives/genesis-builder/src/lib.rs
michalkucharczyk Jun 7, 2023
2435e42
Update primitives/genesis-builder/Cargo.toml
michalkucharczyk Jun 7, 2023
8b62fcf
Cargo.lock updated
michalkucharczyk Jun 7, 2023
2547761
removed redundant function
michalkucharczyk Jun 8, 2023
a1b64de
GenesisConfigBuilder API: no_defaults function added
michalkucharczyk Jun 21, 2023
750ddb2
Cargo.lock updated
michalkucharczyk Jun 21, 2023
cfbd0d2
Merge remote-tracking branch 'origin/master' into mku-genesis-builder…
Jun 21, 2023
4c615f0
GenesisConfigBuilder API: patching fn removed
michalkucharczyk Jun 26, 2023
b02e21e
trigger CI job
michalkucharczyk Jun 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ members = [
"primitives/database",
"primitives/debug-derive",
"primitives/externalities",
"primitives/genesis-builder",
"primitives/inherents",
"primitives/io",
"primitives/keyring",
Expand Down
26 changes: 26 additions & 0 deletions primitives/genesis-builder/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
[package]
name = "sp-genesis-builder"
version = "0.1.0-dev"
michalkucharczyk marked this conversation as resolved.
Show resolved Hide resolved
authors = ["Parity Technologies <admin@parity.io>"]
edition = "2021"
license = "Apache-2.0"
homepage = "https://substrate.io"
repository = "https://github.com/paritytech/substrate/"
description = "Substrate GenesisConfig builder API"
readme = "README.md"

[package.metadata.docs.rs]
targets = ["x86_64-unknown-linux-gnu"]

[dependencies]
sp-api = { version = "4.0.0-dev", default-features = false, path = "../../primitives/api" }
sp-std = { version = "5.0.0", default-features = false, path = "../../primitives/std" }
serde_json = { version = "1.0.85", default-features = false, features = ["alloc"] }

[features]
default = [ "std" ]
std = [
"sp-api/std",
"sp-std/std",
"serde_json/std",
]
3 changes: 3 additions & 0 deletions primitives/genesis-builder/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Substrate genesis builder

License: GPL-3.0-or-later WITH Classpath-exception-2.0
michalkucharczyk marked this conversation as resolved.
Show resolved Hide resolved
43 changes: 43 additions & 0 deletions primitives/genesis-builder/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// This file is part of Substrate.

// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0

// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#![cfg_attr(not(feature = "std"), no_std)]

//! Substrate genesis config builder
//!
//! This Runtime API allows to construct `GenesisConfig`, in particular:
//! - serialize the runtime default `GenesisConfig` struct into json format,
//! - put the GenesisConfig struct into the storage. Internally this operation calls
//! `GenesisBuild::build` function
//! for all runtime pallets, which is typically provided by pallet's author.
//! - deserialize the GenesisConfig from given json blob and put GenesisConfig into the state
//! storage. Allows to build
//! customized configuration.
michalkucharczyk marked this conversation as resolved.
Show resolved Hide resolved
//!
//! Providing externalities with empty storage and putting GenesisConfig into storage allows to
michalkucharczyk marked this conversation as resolved.
Show resolved Hide resolved
//! catch and build the raw storage of `GenesisConfig` which is the foundation for genesis block.

sp_api::decl_runtime_apis! {
/// API to interact with GenesisConfig for the runtime
pub trait GenesisBuilder {
/// Instantiate default `GenesisConfig` and serializes it to json blob.
fn default_genesis_config_as_json() -> sp_std::vec::Vec<u8>;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fn default_genesis_config_as_json() -> sp_std::vec::Vec<u8>;
fn generate_as_json(name: sp_runtime::RuntimeString) -> sp_std::vec::Vec<u8>;

Naming is still not really good, but yeah... We also need to supply some kind of name or whatever we want to call it to distinguish which genesis config we want to generate. Here we also have different kind of genesis configs: https://github.com/paritytech/substrate/tree/2c3b923423fac829b02842fbb9a0016b55c417df/bin/node/cli/src/chain_spec.rs

Copy link
Contributor Author

@michalkucharczyk michalkucharczyk May 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I am missing something, but I don't see the need for extra parameter to construct default GenesisConfig. GenesisConfig of the runtime is just a struct containing all GenesisConfigs pallets, so all required parametrization is already self-contained in config (AFAIU it).

e.g. the generated GenesisConfig for kitchensink-runtime:

pub struct GenesisConfig {
    pub system: SystemConfig,
    pub babe: BabeConfig,
    pub indices: IndicesConfig,
    pub balances: BalancesConfig,
    pub transaction_payment: TransactionPaymentConfig,
    pub staking: StakingConfig,
    pub session: SessionConfig,
    pub democracy: DemocracyConfig,
    pub council: CouncilConfig,
    pub technical_committee: TechnicalCommitteeConfig,
    pub elections: ElectionsConfig,
    pub technical_membership: TechnicalMembershipConfig,
    pub grandpa: GrandpaConfig,
    pub treasury: TreasuryConfig,
    pub sudo: SudoConfig,
    pub im_online: ImOnlineConfig,
    pub authority_discovery: AuthorityDiscoveryConfig,
    pub society: SocietyConfig,
    pub vesting: VestingConfig,
    pub assets: AssetsConfig,
    pub transaction_storage: TransactionStorageConfig,
    pub alliance_motion: AllianceMotionConfig,
    pub alliance: AllianceConfig,
    pub nomination_pools: NominationPoolsConfig,
}

What I see in chain_spec.rs are different ChainSpecs: development_config, local_testnet_config.

Each of them has specific genesis: local_testnet_genesis, development_config_genesis. Those specific gensis are just differently parametrized GenesisConfig instances of kitchensink_runtime created in shared function testnet_genesis.

With new GenesisBuilder API approach, that custom GenesisConfig will be embedded in the form of pre-configured json files customized on top of the json for default GensisConfig.

If I missed something, would you please be more precise on where the extra parameter is actually required?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With new GenesisBuilder API approach, that custom GenesisConfig will be embedded in the form of pre-configured json files customized on top of the json for default GensisConfig.

Could you give an example of this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Examples for test-runtime.

What I mean by default GenesisConfig:

substrate/test-utils/runtime/src/lib.rs

Line 1340 in e147d24

let expected = r#"{"system":{"code":"0x"},"babe":{"authorities":[],"epochConfig":null},"substrateTest":{"authorities":[]},"balances":{"balances":[]}}"#;

What I mean by GenesisConfig customised on top of default:

substrate/test-utils/runtime/src/test_json/default_genesis_config.json

Line 1 in e147d24

{"system":{"code":"0x52"},"babe":{"authorities":[["5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",1],["5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",1],["5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",1]],"epochConfig":{"c":[3,10],"allowed_slots":"PrimaryAndSecondaryPlainSlots"}},"substrateTest":{"authorities":["5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY","5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty","5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y"]},"balances":{"balances":[["5D34dL5prEUaGNQtPPZ3yN5Y6BnkfXunKXXz6fo7ZJbLwRRH",100000000000000000],["5GBNeWRhZc2jXu7D55rBimKYDk8PGk8itRYFTPfC8RJLKG5o",100000000000000000],["5Dfis6XL8J2P6JHUnUtArnFWndn62SydeP8ee8sG2ky9nfm9",100000000000000000],["5F4H97f7nQovyrbiq4ZetaaviNwThSVcFobcA5aGab6167dK",100000000000000000],["5DiDShBWa1fQx6gLzpf3SFBhMinCoyvHM1BWjPNsmXS8hkrW",100000000000000000],["5EFb84yH9tpcFuiKUcsmdoF7xeeY3ajG1ZLQimxQoFt9HMKR",100000000000000000],["5DZLHESsfGrJ5YzT3HuRPXsSNb589xQ4Unubh1mYLodzKdVY",100000000000000000],["5GHJzqvG6tXnngCpG7B12qjUvbo5e4e9z8Xjidk3CQZHxTPZ",100000000000000000],["5CUnSsgAyLND3bxxnfNhgWXSe9Wn676JzLpGLgyJv858qhoX",100000000000000000],["5CVKn7HAZW1Ky4r7Vkgsr7VEW88C2sHgUNDiwHY9Ct2hjU8q",100000000000000000],["5H673aukQ4PeDe1U2nuv1bi32xDEziimh3PZz7hDdYUB7TNz",100000000000000000],["5HTe9L15LJryjUAt1jZXZCBPnzbbGnpvFwbjE3NwCWaAqovf",100000000000000000],["5D7LFzGpMwHPyDBavkRbWSKWTtJhCaPPZ379wWLT23bJwXJz",100000000000000000],["5CLepMARnEgtVR1EkUuJVUvKh97gzergpSxUU3yKGx1v6EwC",100000000000000000],["5Chb2UhfvZpmjjEziHbFbotM4quX32ZscRV6QJBt1rUKzz51",100000000000000000],["5HmRp3i3ZZk7xsAvbi8hyXVP6whSMnBJGebVC4FsiZVhx52e",100000000000000000],["5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",100000000000000000],["5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",100000000000000000],["5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",100000000000000000]]}}

Since we won't have GenesisConfig on the native side of the node (at least this is my current understanding), we will not be able to hard-code the GenesisConfig in the form of rust variable. The alternative for this will be json string.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

substrate/bin/node/cli/src/chain_spec.rs

Lines 297 to 379 in 9d0200b

GenesisConfig {
system: SystemConfig { code: wasm_binary_unwrap().to_vec() },
balances: BalancesConfig {
balances: endowed_accounts.iter().cloned().map(|x| (x, ENDOWMENT)).collect(),
},
indices: IndicesConfig { indices: vec![] },
session: SessionConfig {
keys: initial_authorities
.iter()
.map(|x| {
(
x.0.clone(),
x.0.clone(),
session_keys(x.2.clone(), x.3.clone(), x.4.clone(), x.5.clone()),
)
})
.collect::<Vec<_>>(),
},
staking: StakingConfig {
validator_count: initial_authorities.len() as u32,
minimum_validator_count: initial_authorities.len() as u32,
invulnerables: initial_authorities.iter().map(|x| x.0.clone()).collect(),
slash_reward_fraction: Perbill::from_percent(10),
stakers,
..Default::default()
},
democracy: DemocracyConfig::default(),
elections: ElectionsConfig {
members: endowed_accounts
.iter()
.take((num_endowed_accounts + 1) / 2)
.cloned()
.map(|member| (member, STASH))
.collect(),
},
council: CouncilConfig::default(),
technical_committee: TechnicalCommitteeConfig {
members: endowed_accounts
.iter()
.take((num_endowed_accounts + 1) / 2)
.cloned()
.collect(),
phantom: Default::default(),
},
sudo: SudoConfig { key: Some(root_key) },
babe: BabeConfig {
authorities: vec![],
epoch_config: Some(kitchensink_runtime::BABE_GENESIS_EPOCH_CONFIG),
},
im_online: ImOnlineConfig { keys: vec![] },
authority_discovery: AuthorityDiscoveryConfig { keys: vec![] },
grandpa: GrandpaConfig { authorities: vec![] },
technical_membership: Default::default(),
treasury: Default::default(),
society: SocietyConfig {
members: endowed_accounts
.iter()
.take((num_endowed_accounts + 1) / 2)
.cloned()
.collect(),
pot: 0,
max_members: 999,
},
vesting: Default::default(),
assets: pallet_assets::GenesisConfig {
// This asset is used by the NIS pallet as counterpart currency.
assets: vec![(9, get_account_id_from_seed::<sr25519::Public>("Alice"), true, 1)],
..Default::default()
},
transaction_storage: Default::default(),
transaction_payment: Default::default(),
alliance: Default::default(),
alliance_motion: Default::default(),
nomination_pools: NominationPoolsConfig {
min_create_bond: 10 * DOLLARS,
min_join_bond: 1 * DOLLARS,
..Default::default()
},
glutton: GluttonConfig {
compute: Default::default(),
storage: Default::default(),
trash_data_count: Default::default(),
},

This is the stuff I'm "worried" about.

But I think we should be able to solve this with some tricks. However this would require to write there some json code, but I think this is doable.

I imagine something like this:

let config = json! {{
     "session": {
         "keys":  initial_authorities.map().collect(),
     },
      "staking": {
          "validator_count": validator_count,
      },
}};

// It should be able to handle that we only pass parts of the config and the rest should be filled by the default values.
Runtime::build_from_json(config);

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW this json macro already exists: https://docs.rs/serde_json/latest/serde_json/macro.json.html

Copy link
Contributor Author

@michalkucharczyk michalkucharczyk Jun 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in: 679a970

We would not need to put this functionality into the runtime, we can easily implement it outside the runtime, using functions already defined:

let json_str = Runtime::default_as_json();
let mut json: Value = serde_json::from_str(json_str).unwrap();
merge(&mut json, json!({
     "session": {
         "keys":  initial_authorities.map().collect(),
     },
      "staking": {
          "validator_count": validator_count,
      },
});
Runtime::build_from_json(json.to_string());

However I do understand that build_with_patch is a nice convenient function.


/// Deserialize the `GenesisConfig` from given json blob and put it into the storage.
fn build_genesis_config_from_json(json: sp_std::vec::Vec<u8>);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fn build_genesis_config_from_json(json: sp_std::vec::Vec<u8>);
fn build_from_json(json: sp_std::vec::Vec<u8>);

}
}