-
Notifications
You must be signed in to change notification settings - Fork 856
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GenesisConfig
in a native runtime free world
#25
Comments
Required support for serde addressed in paritytech/substrate#13027 |
Is anyone really using runtime-formatted JSON specs? All we really need is raw spec which does not require serde. |
I think zombienet relies on meddling with the JSON chain spec and other testing tools probably as well. |
Yeah and this will open a lot of possibilities. We will be able to ship json chain specs without them breaking, as the wasm file that loads them stays the same. We will be able to write tools that can modify chain specs very easily etc. It improves the user experience quite a lot. |
I see it is a nice to have but not essential. We can always load metadata from wasm and use it to convert between the raw values and JSON. polkadot.js makes this relatively easy. If this feature doesn't need to be part of Substrate, maybe it doesn't need to be part of Substrate? |
Good point, if that works good enough we don't need serde/json support. |
Chopsticks already implemented this feature and it works well. Custom storage override: Code that convert yaml/json to hex: |
Yeah to hex is easy, the other way around is more the problem. We would also then "disable" the |
edited: question moved to: https://github.com/paritytech/substrate/pull/14131/files#r1192701289 |
This issue has been mentioned on Polkadot Forum. There might be relevant details there: |
This change is follow-up of: paritytech/substrate#14108 It is a step towards: https://github.com/paritytech/substrate/issues/13334
This change is follow-up of: paritytech/substrate#14108 It is a step towards: https://github.com/paritytech/substrate/issues/13334
* pallets: implement Default for GenesisConfig in no_std This change is follow-up of: paritytech/substrate#14108 It is a step towards: https://github.com/paritytech/substrate/issues/13334 * Cargo.lock updated * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <>
* pallets: implement Default for GenesisConfig in no_std This change is follow-up of: paritytech/substrate#14108 It is a step towards: https://github.com/paritytech/substrate/issues/13334 * ".git/.scripts/commands/fmt/fmt.sh" * update lockfile for {"substrate", "polkadot"} --------- Co-authored-by: command-bot <>
@bkchr I need the native runtime for the |
Can you give an example on what exactly you are doing there? Especially as |
@bkchr I'm using the This is because my pallet creates projects that go through several stages as time (measured in blocks) moves forward. Now to test the UI, we need to start a local parachain with one project in each stage, and for this, every time I go forward in time, I need to see which project needs to resume execution.
The previous example is run through a function, such that in testing if we want to create a project at state 3 and then do some tests, we call Each state has a certain time it can remain alive. After that, it moves into either the success or failed state. If we use the normal synchronous way of instantiating our test projects, then by the time we advance blocks to create a new project in another state, the previous instantiated project states become invalid. The current way which is already working, is by creating an async project creation function, which blocks execution whenever it wants to advance blocks. Before blocking though, it notifies an orchestrator at which block it wants to be awakened at. The orchestrator then starts advancing blocks until it reaches the lowest block required by one of the instantiating project functions and hands over execution to it. A simple way for one test would be to hardcode each transition of each project instead of relying on calling functions and expecting them to hand over execution between themselves. But then I wouldn't be able to pass in any info at Genesis to start any number of projects at any state. If this is too confusing, I can give you access to our repo at Polimec and you can look through the code yourself. I couldn't get the Genesis builder working using the multithreaded Tokio runtime since each task wasn't executing under the context of the substrate storage. But using the single-threaded runtime did the trick. |
@bkchr We just open-sourced our code, here is where we are using async in the builder: https://github.com/Polimec/polimec-node/blob/da9d1ee0062ead7a62f815647813ada48e4c2250/pallets/funding/src/lib.rs#L1386 As mentioned before, we're using the single-threaded Tokio runtime and it works just fine. |
@JuaniRios why don't you run this async code in the node and then pass the processed data into the runtime? |
@bkchr the purpose of the async code is to create genesis storage to use in testing. Since this is the main use case of the genesis builder, I believe it makes sense to be inside the runtime code. How would you see us achieving the same thing with the code moved to the node? Also I heard there that soon we would move to an "omninode", so we would stop being able to put our own code into the substrate node, and just use a generic one imported as a crate |
If this is only for testing - maybe you could use Once you have a genesis storage it should be easy to convert it to runtime |
@michalkucharczyk thanks for the reply! So basically do what the genesis builder is already doing, but hardcoding the logic in the node instead of calling the code from the runtime? What would be the advantage of this? If polkadot-sdk removes the native runtime, then calling EDIT: |
Stupid question, could we use async to run multiple extrinsics simultaneously if they don't read/write from the same storage? Would something like this need to be programmed in the runtime itself, or could the node know each extrinsic which storage items it will touch and so know which other extrinsics it can batch concurrently? |
There will probably be such a node, but no one forces you to use this. This is more to make it easier to create parachains. However, with the omninode you would not have the native runtime either and so your "hack" wouldn't work. I honestly still don't get why you need to use async there and what the code is doing. I also don't want to dig into this. As I already told you, you can move this code somewhere else and then add whatever data this async code is generating to your
This is a complete different topic and we don't support this right now. I know that there exists some frameworks supporting "parallel" execution of transactions. However, it is questionable if you really need this right now. |
Yea I don't need it right now, but thought it might be an interesting idea for future optimizations |
Removal of native runtime in node means that node is no longer coupled with runtime and is able to do all the work with just But your runtime is still just a crate, which maybe used in whatever-you-want environment (e.g. wasm runtime or tests). You can write tools or tests (or even your flavor of node) that are using your runtime crate. Hope this addresses your doubts. |
@michalkucharczyk thanks for your detailed answer :) Last question, will the "std" feature config be removed from the |
I don't think so, however I am not an authority in this area :) I looked into code again, you perhaps don't even need to use |
I thought part of the move to native-runtime free was to remove "std" from the macro expansion of the This is how the current expansion looks: #[cfg(feature = "std")]
impl<T: Config> frame_support::sp_runtime::BuildStorage for GenesisConfig<T> where
T: Config + pallet_balances::Config<Balance=BalanceOf<T>>,
<T as Config>::AllPalletsWithoutSystem:
OnFinalize<BlockNumberFor<T>> + OnIdle<BlockNumberFor<T>> + OnInitialize<BlockNumberFor<T>>,
<T as Config>::RuntimeEvent: From<Event<T>> + TryInto<Event<T>> + Parameter + Member,
<T as pallet_balances::Config>::Balance: Into<BalanceOf<T>>,
{
fn assimilate_storage(&self, storage: &mut sp_runtime::Storage) -> std::result::Result<(), std::string::String> {
frame_support::BasicExternalities::execute_with_storage(storage, || {
self.build();
Ok(())
})
}
} And also the #[cfg(any(feature = "std", test))]
impl self::sp_api_hidden_includes_construct_runtime::hidden_include::sp_runtime::BuildStorage for RuntimeGenesisConfig {
fn assimilate_storage(&self, storage: &mut self::sp_api_hidden_includes_construct_runtime::hidden_include::sp_runtime::Storage ) -> std::result::Result<(), String> {
self::sp_api_hidden_includes_construct_runtime::hidden_include::BasicExternalities::execute_with_storage(storage, || {
<Self as self::sp_api_hidden_includes_construct_runtime::hidden_include::traits::BuildGenesisConfig>::build(&self);
Ok(())
})
}
} The default node uses this struct with "std" enabled to build the genesis storage. So when std is removed, my code won't work anymore. |
edited: Yeah, we need
I don't see a problem here. Features are not passed by macro (which is just putting a |
I understand what you are saying, but if you remove the "std" cfg from the I guess one solution would be like you suggested to have another genesis builder logic in my node, where I am calling it with "std" compiled. If we generate the genesis storage by calling the code directly, does that mean that it will always be a "native runtime"? |
I messed with naming so I edited my previous comment.
FWIS you already have this logic implemented. All you have to do is to call
If you need this only for testing, then I would not call it native runtime. You could think of it as some kind of util generating test chain-specs. Think of native runtime as runtime code which is linked with node (thus name native), and is also a crate dependency for node which makes all internal (public) types and functions available in rust. |
@michalkucharczyk Thank you for the answers, much more clear now! Yes I am already calling If I never move to the "omninode", does this mean that my "std" code will still run, even if the native-runtime is "removed"? |
Shall be fine. |
This PR implements [`GenesisBuilder` API](https://github.com/paritytech/polkadot-sdk/blob/a414ea7515c9cdc81f1d12410e646afc148250e8/substrate/primitives/genesis-builder/src/lib.rs#L38) for all the runtimes in polkadot repo. Step towards: paritytech#25 --------- Co-authored-by: ordian <write@reusable.software>
The runtime now can provide a number of predefined presets of `RuntimeGenesisConfig` struct. This presets are intended to be used in different deployments, e.g.: `local`, `staging`, etc, and should be included into the corresponding chain-specs. Having `GenesisConfig` presets in runtime allows to fully decouple node from runtime types (the problem is described in #1984). **Summary of changes:** - The `GenesisBuilder` API was adjusted to enable this functionality (and provide better naming - #150): ```rust fn preset_names() -> Vec<PresetId>; fn get_preset(id: Option<PresetId>) -> Option<serde_json::Value>; //`None` means default fn build_state(value: serde_json::Value); pub struct PresetId(Vec<u8>); ``` - **Breaking change**: Old `create_default_config` method was removed, `build_config` was renamed to `build_state`. As a consequence a node won't be able to interact with genesis config for older runtimes. The cleanup was made for sake of API simplicity. Also IMO maintaining compatibility with old API is not so crucial. - Reference implementation was provided for `substrate-test-runtime` and `rococo` runtimes. For rococo new [`genesis_configs_presets`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/polkadot/runtime/rococo/src/genesis_config_presets.rs#L530) module was added and is used in `GenesisBuilder` [_presets-related_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/polkadot/runtime/rococo/src/lib.rs#L2462-L2485) methods. - The `chain-spec-builder` util was also improved and allows to ([_doc_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/substrate/bin/utils/chain-spec-builder/src/lib.rs#L19)): - list presets provided by given runtime (`list-presets`), - display preset or default config provided by the runtime (`display-preset`), - build chain-spec using named preset (`create ... named-preset`), - The `ChainSpecBuilder` is extended with [`with_genesis_config_preset_name`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/substrate/client/chain-spec/src/chain_spec.rs#L447) method which allows to build chain-spec using named preset provided by the runtime. Sample usage on the node side [here](https://github.com/paritytech/polkadot-sdk/blob/2caffaae803e08a3d5b46c860e8016da023ff4ce/polkadot/node/service/src/chain_spec.rs#L404). Implementation of #1984. fixes: #150 part of: #25 --------- Co-authored-by: Sebastian Kunert <skunert49@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
The runtime now can provide a number of predefined presets of `RuntimeGenesisConfig` struct. This presets are intended to be used in different deployments, e.g.: `local`, `staging`, etc, and should be included into the corresponding chain-specs. Having `GenesisConfig` presets in runtime allows to fully decouple node from runtime types (the problem is described in #1984). **Summary of changes:** - The `GenesisBuilder` API was adjusted to enable this functionality (and provide better naming - #150): ```rust fn preset_names() -> Vec<PresetId>; fn get_preset(id: Option<PresetId>) -> Option<serde_json::Value>; //`None` means default fn build_state(value: serde_json::Value); pub struct PresetId(Vec<u8>); ``` - **Breaking change**: Old `create_default_config` method was removed, `build_config` was renamed to `build_state`. As a consequence a node won't be able to interact with genesis config for older runtimes. The cleanup was made for sake of API simplicity. Also IMO maintaining compatibility with old API is not so crucial. - Reference implementation was provided for `substrate-test-runtime` and `rococo` runtimes. For rococo new [`genesis_configs_presets`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/polkadot/runtime/rococo/src/genesis_config_presets.rs#L530) module was added and is used in `GenesisBuilder` [_presets-related_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/polkadot/runtime/rococo/src/lib.rs#L2462-L2485) methods. - The `chain-spec-builder` util was also improved and allows to ([_doc_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/substrate/bin/utils/chain-spec-builder/src/lib.rs#L19)): - list presets provided by given runtime (`list-presets`), - display preset or default config provided by the runtime (`display-preset`), - build chain-spec using named preset (`create ... named-preset`), - The `ChainSpecBuilder` is extended with [`with_genesis_config_preset_name`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/substrate/client/chain-spec/src/chain_spec.rs#L447) method which allows to build chain-spec using named preset provided by the runtime. Sample usage on the node side [here](https://github.com/paritytech/polkadot-sdk/blob/2caffaae803e08a3d5b46c860e8016da023ff4ce/polkadot/node/service/src/chain_spec.rs#L404). Implementation of #1984. fixes: #150 part of: #25 --------- Co-authored-by: Sebastian Kunert <skunert49@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
The runtime now can provide a number of predefined presets of `RuntimeGenesisConfig` struct. This presets are intended to be used in different deployments, e.g.: `local`, `staging`, etc, and should be included into the corresponding chain-specs. Having `GenesisConfig` presets in runtime allows to fully decouple node from runtime types (the problem is described in paritytech#1984). **Summary of changes:** - The `GenesisBuilder` API was adjusted to enable this functionality (and provide better naming - paritytech#150): ```rust fn preset_names() -> Vec<PresetId>; fn get_preset(id: Option<PresetId>) -> Option<serde_json::Value>; //`None` means default fn build_state(value: serde_json::Value); pub struct PresetId(Vec<u8>); ``` - **Breaking change**: Old `create_default_config` method was removed, `build_config` was renamed to `build_state`. As a consequence a node won't be able to interact with genesis config for older runtimes. The cleanup was made for sake of API simplicity. Also IMO maintaining compatibility with old API is not so crucial. - Reference implementation was provided for `substrate-test-runtime` and `rococo` runtimes. For rococo new [`genesis_configs_presets`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/polkadot/runtime/rococo/src/genesis_config_presets.rs#L530) module was added and is used in `GenesisBuilder` [_presets-related_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/polkadot/runtime/rococo/src/lib.rs#L2462-L2485) methods. - The `chain-spec-builder` util was also improved and allows to ([_doc_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/substrate/bin/utils/chain-spec-builder/src/lib.rs#L19)): - list presets provided by given runtime (`list-presets`), - display preset or default config provided by the runtime (`display-preset`), - build chain-spec using named preset (`create ... named-preset`), - The `ChainSpecBuilder` is extended with [`with_genesis_config_preset_name`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/substrate/client/chain-spec/src/chain_spec.rs#L447) method which allows to build chain-spec using named preset provided by the runtime. Sample usage on the node side [here](https://github.com/paritytech/polkadot-sdk/blob/2caffaae803e08a3d5b46c860e8016da023ff4ce/polkadot/node/service/src/chain_spec.rs#L404). Implementation of paritytech#1984. fixes: paritytech#150 part of: paritytech#25 --------- Co-authored-by: Sebastian Kunert <skunert49@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Tiny refactoring * Upgrade polkadot-sdk * Make the log consistent with upstream * Nit * Add import_params to Run * Add profiling profile * Nit * Add rocksdb feature into subcoin-node * Avoid cloning the payload using into_payload() * Update Cargo.lock * Nit
…m-umbrella Remove snowbridge from umbrella
The
GenesisConfig
is currently one of the only things for that we still require the native runtime. It isn't compiled for wasm right now. We also require the native runtime to load aGenesisConfig
from a JSON file and as theGenesisConfig
can change with newer native runtimes we got the raw chain spec that only contains the key/value storage pairs of the genesis block. When we remove the native runtime we need a new way to handle theGenesisConfig
.I propose that we put the
GenesisConfig
into the genesis wasm. This means that we start compiling theGenesisConfig
for wasm, but I would put this behind some rust feature. So, the normal production builds should not include theGenesisConfig
anymore. This is done to reduce the size of the wasm binary. I would also keep supporting the serialization of theGenesisConfig
into JSON. If theGenesisConfig
is part of the genesis wasm, we have some nice advantages. The first one being that we could keep the JSON serializedGenesisConfig
in the chain spec and even distribute this chain spec as we would call into the genesis wasm to parse and generate the genesis block. This would require that we move out the genesis wasm from the actual genesis field in the chain spec to some dedicated field from where we can load the wasm binary. Another advantage would be that we could finally make thechain-spec-builder
work properly. Currently this binary isn't really working for chain specs that aren't build from thekitchensink-runtime
. Thechain-spec-builder
could be passed some crate name that belongs to some runtime and then it would generate the chain spec for this runtime. We could also enable certain features in the runtime etc.To build what I outlined above, we would need the following:
GensisConfig
build in wasm. This will require supportingserde
in the wasm build to serialize/deserialize theGenesisConfig
to/from JSON.GenesisBuilder
(better name should be found) that can be used to build the genesis config, load it etc.code
.GenesisConfig
into the wasm binary. By default theGenesisConfig
should probably be compiled into the wasm binary. When doing a on-chain build there should exist a feature to disable theGenesisConfig
.Part of: #62
The text was updated successfully, but these errors were encountered: