Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate runtime API from metadata #294

Merged
merged 219 commits into from
Nov 3, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
219 commits
Select commit Hold shift + click to select a range
f43df0e
Remove test macro
ascjones Aug 26, 2021
01b7131
Remove client crate
ascjones Aug 26, 2021
ff6b9fc
Create tests crate and move pallet specific tests there
ascjones Aug 26, 2021
d7c1d22
Extract client, remove metadata and extra, more demolition
ascjones Aug 26, 2021
b755c29
Update substrate dependencies to git dependencies
ascjones Aug 26, 2021
05fe051
Remove Store stuff for now
ascjones Aug 26, 2021
5e91406
Comment out some Call usages
ascjones Aug 26, 2021
59aa4d7
Add back Runtime trait coped from original System trait
ascjones Aug 26, 2021
e337d34
Make subxt lib compile
ascjones Aug 27, 2021
8540533
Delete old proc macros and copy over type generation from chameleon
ascjones Aug 27, 2021
e917dac
WIP make transfer balance test pass
ascjones Sep 3, 2021
2ef017d
Change to subxt attribute macro
ascjones Sep 3, 2021
f3005cb
WIP provide user defined type substitutes
ascjones Sep 3, 2021
e4f1374
User defined type substitutes compile
ascjones Sep 6, 2021
b98b7aa
WIP submitting transactions
ascjones Sep 6, 2021
e20f123
WIP transfer balance test
ascjones Sep 6, 2021
2cf09c1
Fix macro
ascjones Sep 6, 2021
56cb2a6
Cargo fmt
ascjones Sep 6, 2021
c35d781
WIP generating storage hashers
ascjones Sep 7, 2021
4726db3
WIP add AccountData trait for fetching the nonce
ascjones Sep 7, 2021
f8f8b47
Support single type storage map keys
ascjones Sep 7, 2021
4b573f5
WIP impl AccountInfo retrieval
ascjones Sep 7, 2021
a6e7225
Fix up storage struct generation
ascjones Sep 8, 2021
d2bd669
Implement AccountData triait directly on storage entry
ascjones Sep 8, 2021
11248d0
Borrow storage map key and convert account id
ascjones Sep 8, 2021
b9ad905
Implement storage fetch client methods
ascjones Sep 8, 2021
f98d95a
Remove legacy metadata storage key construction
ascjones Sep 8, 2021
a8ca64a
Rename CheckEra to CheckMortality
ascjones Sep 8, 2021
9eac486
Substitute perthings types for compact impls
ascjones Sep 8, 2021
a5281fc
Fmt
ascjones Sep 9, 2021
bbb4530
Downgrade dyn-clone for cargo-contract compat
ascjones Sep 9, 2021
2d7bed8
Scale-fo 1.0
ascjones Sep 9, 2021
8ab7976
scale-info 1.0
ascjones Sep 9, 2021
6ac3f42
Remove special range handling
ascjones Sep 9, 2021
c935dc1
Restore wildcard type params
ascjones Sep 9, 2021
13f69a3
Frame metadata 14.0
ascjones Sep 9, 2021
af7c4ce
WIP decoding events
ascjones Sep 9, 2021
4e47acd
WIP more dynamically decoding events
ascjones Sep 10, 2021
7e2ed94
Fmt
ascjones Sep 10, 2021
a2bb550
Decode events, handle errors
ascjones Sep 10, 2021
0b233e0
Uncomment some tests
ascjones Sep 13, 2021
09c5888
Remove unused get_mod function
ascjones Sep 13, 2021
93c121d
Fix some warnings
ascjones Sep 13, 2021
a503911
Fix some more warnings
ascjones Sep 13, 2021
b362dfd
Fix some more warnings
ascjones Sep 13, 2021
8357031
Add tests mod
ascjones Sep 13, 2021
e5333f1
Rename node-runtime tests mod to frame
ascjones Sep 13, 2021
5405fb7
Fix some warnings
ascjones Sep 13, 2021
1034b68
Fmt
ascjones Sep 13, 2021
8556bc0
WIP generate storage client with getters
ascjones Sep 13, 2021
ed0d1bc
Storage client compiling
ascjones Sep 13, 2021
327a215
Generate storage client api
ascjones Sep 14, 2021
e70534e
Fix up system account query account ids
ascjones Sep 14, 2021
d75d1f0
WIP generating tx api fns
ascjones Sep 14, 2021
634f21f
Only generate tx api fields when calls available
ascjones Sep 14, 2021
cedf27e
Fix tx api call fns
ascjones Sep 15, 2021
690e079
Fmt
ascjones Sep 15, 2021
b82d4e8
WIP generate event structs
ascjones Sep 16, 2021
8342c31
call functions not async
ascjones Sep 16, 2021
4cb16a3
Derive Eq for comparison on generated types
ascjones Sep 16, 2021
97308a9
Generate event structs
ascjones Sep 17, 2021
f8f2da6
Fix call name
ascjones Sep 17, 2021
4d0e392
Fmt
ascjones Sep 17, 2021
a264cf3
Update node runtime metadata to substrate c000780db
ascjones Sep 17, 2021
2eb657a
Download latest substrate release for integration testing
ascjones Sep 17, 2021
c89791f
Fix event decoding
ascjones Sep 17, 2021
537c633
Remove unused imports
ascjones Sep 17, 2021
73490a3
Fix plain storage access, total_issuance pass
ascjones Sep 17, 2021
4ab94d5
Fmt
ascjones Sep 17, 2021
9a6c98c
Restore contracts tests
ascjones Sep 17, 2021
599863e
Backoff connecting to substrate node
ascjones Sep 17, 2021
b99685a
Add required TypeInfo impls for local SignedExtension impls
ascjones Sep 17, 2021
f3ae886
Remove unnecessary assert formatting
ascjones Sep 20, 2021
a5c51af
Fix handling of DispatchError
ascjones Sep 20, 2021
92f8665
Refactor contracts tests
ascjones Sep 20, 2021
7da7f56
Troubleshooting contract not found
ascjones Sep 20, 2021
01558f5
Remove more client feature stuff
ascjones Sep 22, 2021
0203365
Fix dynamic event variant decoding, write consumed index to output
ascjones Sep 22, 2021
184c096
Fmt
ascjones Sep 22, 2021
e778231
Use substrate branch with heavy dependency removed
ascjones Sep 24, 2021
ba71cc0
Remove sp-rcp dependency, define types locally
ascjones Sep 24, 2021
cadb26f
Ignore cargo timeing files
ascjones Sep 24, 2021
2766b40
Use my branch for substrate test deps
ascjones Sep 24, 2021
1b04411
Fix storage key type gen
ascjones Sep 24, 2021
af1ec84
Comment out fetching contract info
ascjones Sep 27, 2021
bbad328
Merge branch 'master' into aj-metadata-vnext
ascjones Sep 27, 2021
c8ebd1c
Add key iteration, extract storage client from main client
ascjones Sep 27, 2021
0cf8ec7
Debugging key generation
ascjones Sep 27, 2021
d68e7dd
Use substrate master branch
ascjones Sep 27, 2021
a6f8eb4
Fix call test
ascjones Sep 27, 2021
20aee21
Remove TypeSegmenter and dynclone dependency
ascjones Sep 29, 2021
3de2815
Publicly expose Rpc mod
ascjones Sep 29, 2021
c5ef0fa
Unused import warnings
ascjones Sep 29, 2021
b2f0ddf
Add getter for runtime metadata
ascjones Sep 29, 2021
e8256e0
Add pallet and event indices for raw events
ascjones Sep 29, 2021
d0dda23
Add is_call and is_event convenience trait functions
ascjones Oct 1, 2021
6198ac2
Add missing docs
ascjones Oct 4, 2021
cbbf0e1
Refactor tests crate
ascjones Oct 4, 2021
76d7b11
Restore remaining client tests
ascjones Oct 4, 2021
c86a9ce
Fmt
ascjones Oct 4, 2021
d439722
Fix warnings
ascjones Oct 4, 2021
997d6b7
Restore get_mod as test helper and fmt
ascjones Oct 4, 2021
44ec197
Use client references for api calls
ascjones Oct 4, 2021
950845f
Fix api usages with methods
ascjones Oct 4, 2021
3cb4000
Use Bytes for RawEvent debug
ascjones Oct 4, 2021
602f62b
Update metadata
ascjones Oct 5, 2021
4d1de1b
Restoring some Balances tests
ascjones Oct 5, 2021
384f88e
Populate runtime storage metadata
ascjones Oct 5, 2021
b340bc2
Restore balances lock test
ascjones Oct 5, 2021
f1b7b85
Restore Balances error test
ascjones Oct 5, 2021
ec90ecc
Fmt
ascjones Oct 5, 2021
e4907d4
Restore transfer subscription API
ascjones Oct 5, 2021
bab2aef
Staking test
ascjones Oct 5, 2021
696ee63
Restore another staking test
ascjones Oct 5, 2021
560ceb5
Restore another staking test
ascjones Oct 5, 2021
c5915b9
Restore another staking test
ascjones Oct 5, 2021
5e9b7d2
Partially restore chill_works_for_controller_only staking test
ascjones Oct 5, 2021
34e1da5
Fix fetching Optional storage entries
ascjones Oct 6, 2021
442fe24
Restore staking bond test
ascjones Oct 6, 2021
35a3141
Restore remaining staking tests
ascjones Oct 6, 2021
83af04e
Fmt
ascjones Oct 6, 2021
b95ce3d
Restore sudo tests
ascjones Oct 6, 2021
79355d3
Add some system tests
ascjones Oct 6, 2021
5e7b7e7
Fmt
ascjones Oct 6, 2021
368ec3b
Resolve some todos
ascjones Oct 6, 2021
f819cb6
Remove pass through rpc methods on Client, expose via rpc() getter
ascjones Oct 6, 2021
903b153
Remove more rpc pass through methods
ascjones Oct 6, 2021
d984dd5
Remove submit tx pass through rpc methods
ascjones Oct 6, 2021
a9d1e8f
Add some comments to SubmittableExtrinsic methods
ascjones Oct 7, 2021
ccb1491
Construct the runtime api from the client
ascjones Oct 7, 2021
ffc1a3c
Fmt
ascjones Oct 7, 2021
f46c944
Use From trait instead of new for AccountData query
ascjones Oct 8, 2021
f834035
Rename subxt_proc_macro crate to subxt_macro
ascjones Oct 12, 2021
fc80bb0
Fix AccountData From impl
ascjones Oct 12, 2021
72dc1f9
Extract codegen crate from macro crate
ascjones Oct 12, 2021
6563914
Fmt
ascjones Oct 12, 2021
c66ac9f
Replace chameleon hidden field name
ascjones Oct 12, 2021
7594bc5
Extract StructDef for generating structs
ascjones Oct 12, 2021
d0b0bb1
More refactoring of StructDef, moving towards sharing with typegen
ascjones Oct 12, 2021
2c2dd9f
Replace explicit tests crate with single implicit integration tests c…
ascjones Oct 12, 2021
7d9884d
Rename from substrate-subxt to subxt
ascjones Oct 13, 2021
cc0212d
Fix runtime path relative to root Cargo.toml
ascjones Oct 13, 2021
ea4666f
Move RpcClient creation to RpcClient
ascjones Oct 13, 2021
851f612
WIP get examples to compile
ascjones Oct 13, 2021
69e6ed6
Rename Runtime to Config trait
ascjones Oct 13, 2021
a30f9f8
WIP implementing default Config
ascjones Oct 14, 2021
d5bec8c
WIP implementing default extrinsic extras
ascjones Oct 15, 2021
567ce0e
fix metadata constants (#299)
paulormart Oct 15, 2021
a672dc4
Move DefaultConfig definition and impl to macro
ascjones Oct 15, 2021
518c028
Extract type substitute parsing to ir mod
ascjones Oct 15, 2021
699ce18
Extract calls, events and storage from api generation
ascjones Oct 18, 2021
021c186
Add some hardcoded type substitute defaults
ascjones Oct 18, 2021
99a83c7
Fmt
ascjones Oct 18, 2021
27c5b92
Add utility pallet tests (#300)
paulormart Oct 18, 2021
4df1313
Add polkadot runtime metadata for example
ascjones Oct 18, 2021
da5bb8d
Merge branch 'aj-metadata-vnext' of github.com:paritytech/substrate-s…
ascjones Oct 18, 2021
c2e7ab7
Fix system errors and fmt
ascjones Oct 18, 2021
8108f00
Add subxt-cli crate
ascjones Oct 18, 2021
fd18fcd
Add metadata and codegen subcommands
ascjones Oct 19, 2021
643c8f8
Make subxt-cli codegen command work
ascjones Oct 19, 2021
77ae552
Fmt
ascjones Oct 19, 2021
a0447ff
Add polkadot codegen test
ascjones Oct 19, 2021
562b593
Comment about how to run codegen
ascjones Oct 19, 2021
398c63c
Derive AsCompact for structs with single concrete unsigned int field
ascjones Oct 21, 2021
192fa39
Fix bitvec codegen, adds as non optional dependency
ascjones Oct 21, 2021
adf8ebd
Regenerate polkadot api with bitvec fix
ascjones Oct 21, 2021
e430d1b
Edition 2021
ascjones Oct 21, 2021
90312bd
Fix polkadot codegen with bitvec
ascjones Oct 22, 2021
b2d04bb
Polkadot balance transfer is working
ascjones Oct 22, 2021
d1566e8
Fix fetch remote
ascjones Oct 22, 2021
927aa00
Fix transfer_subscribe example
ascjones Oct 22, 2021
b26cd2f
Fix submit_and_watch example
ascjones Oct 22, 2021
091b255
Fmt
ascjones Oct 22, 2021
cdb881e
Generate storage iter method for iterating over keys
ascjones Oct 22, 2021
da71f69
Fmt
ascjones Oct 22, 2021
23be116
Fix existential deposit test
ascjones Oct 22, 2021
9d8c9a1
Fix staking tests
ascjones Oct 25, 2021
b500ac4
Add option for custom generated type derives
ascjones Oct 28, 2021
43f59e4
Add generated type derives for test runtime api
ascjones Oct 28, 2021
c19d2e6
Fmt
ascjones Oct 28, 2021
1fd0d5f
Copy WrapperTypeOpaque from substrate, add Encode/Decode
ascjones Oct 30, 2021
0c45452
Fmt
ascjones Oct 30, 2021
57f7a0b
Extract type generator to module, separate & fix tests
ascjones Oct 30, 2021
4b45985
Fully qualified primitive and prelude types
ascjones Oct 30, 2021
7caec49
Fix up remaining type gen tests
ascjones Oct 30, 2021
d4a4ce0
Skip formatting of generated polkadot example code
ascjones Oct 30, 2021
cc1fdab
Remove empty utility test file.
ascjones Oct 30, 2021
b570e39
Newline
ascjones Oct 30, 2021
9fd9540
Update cli/src/main.rs
ascjones Oct 30, 2021
4657b57
Rename subxt-cli executable to subxt
ascjones Oct 30, 2021
36d5e57
Update src/client.rs
ascjones Oct 30, 2021
d250904
Merge remote-tracking branch 'origin/aj-metadata-vnext' into aj-metad…
ascjones Oct 30, 2021
9818d0d
Add some code docs to TypeGenerator.
ascjones Oct 30, 2021
13915e2
Extract TypePath to own file
ascjones Oct 30, 2021
69ee2da
Extract type def generation to separate file
ascjones Oct 30, 2021
e8a8376
Renamed ModuleType to TypeDefGen
ascjones Oct 30, 2021
4f3de30
Fmt
ascjones Oct 30, 2021
1d1fffe
Factor out type parameter from final_key
ascjones Oct 31, 2021
b301a85
Fix some type paths
ascjones Oct 31, 2021
c1f6d53
Resolve some todos
ascjones Oct 31, 2021
a97fb2e
Resolve some panic todos in events
ascjones Oct 31, 2021
d58834b
Add EventsDecodingError
ascjones Oct 31, 2021
d0f5e69
Decode compact composite types with a single primitive field
ascjones Oct 31, 2021
9da0bcc
Decode compact composite types with a single primitive field
ascjones Oct 31, 2021
346504b
Update src/metadata.rs
ascjones Nov 2, 2021
2f36a89
Remove Perbill compact substitute types
ascjones Nov 3, 2021
6306b9f
Remove todos regarding maintaining Rust code items, promoted to follo…
ascjones Nov 3, 2021
5dd6cf9
Remove todo regarding overridding default config impl
ascjones Nov 3, 2021
fc30e90
Remove todo regarding overridding default Extra
ascjones Nov 3, 2021
63bb5ea
Remove todo regarding AccountData storage type defintion
ascjones Nov 3, 2021
84fe1a0
Remove todo regarding borrowing storage key arguments
ascjones Nov 3, 2021
6886f4c
Remove type substitution tests todo
ascjones Nov 3, 2021
63641b4
Remove `Box` field name type hack todo
ascjones Nov 3, 2021
ba2e3a8
Remove Compact todo
ascjones Nov 3, 2021
06a6ddf
Remove sudo todos
ascjones Nov 3, 2021
0d07b0c
Remove BitVec implementation todo
ascjones Nov 3, 2021
f0bb889
Fmt
ascjones Nov 3, 2021
2c93f3c
Add health warning to README
ascjones Nov 3, 2021
641ac27
Fix up health warning
ascjones Nov 3, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions codegen/src/api/storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@ fn generate_storage_entry_fns(
let pallet_name = &pallet.name;
let storage_name = &storage_entry.name;
let fn_name = format_ident!("{}", storage_entry.name.to_snake_case());
let fn_name_iter = format_ident!("{}_iter", fn_name);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to break things with bad storage entry naming, e.g. a storage entry named account_iter and another named account (which would gain an account_iter method)? A very contrived example, I know. I guess the real question is: do we need to check Idents for clashes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed it is entirely possible to break this. In the end the compiler will check for clashes for us so I don't see it as a high priority to solve at this point.

let storage_entry_ty = match storage_entry.ty {
StorageEntryType::Plain(ref ty) => ty,
StorageEntryType::Map { ref value, .. } => value,
Expand Down Expand Up @@ -189,10 +190,23 @@ fn generate_storage_entry_fns(
}
};

let client_iter_fn = if matches!(storage_entry.ty, StorageEntryType::Map { .. }) {
quote! (
pub async fn #fn_name_iter(
&self,
hash: ::core::option::Option<T::Hash>,
) -> ::core::result::Result<::subxt::KeyIter<'a, T, #entry_struct_ident>, ::subxt::Error> {
self.client.storage().iter(hash).await
}
)
} else {
quote! ()
};

let key_args = fields
.iter()
.map(|(field_name, field_type)| quote!( #field_name: #field_type )); // todo: [AJ] borrow non build-inf types?
let client_fn = quote! {
let client_fns = quote! {
pub async fn #fn_name(
&self,
#( #key_args, )*
Expand All @@ -201,7 +215,9 @@ fn generate_storage_entry_fns(
let entry = #constructor;
self.client.storage().#fetch(&entry, hash).await
}

#client_iter_fn
};

(storage_entry_type, client_fn)
(storage_entry_type, client_fns)
}
31 changes: 23 additions & 8 deletions examples/fetch_all_accounts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,35 @@
// You should have received a copy of the GNU General Public License
// along with subxt. If not, see <http://www.gnu.org/licenses/>.

use subxt::{
system::AccountStoreExt,
ClientBuilder,
DefaultNodeRuntime,
};
//! To run this example, a local polkadot node should be running.
//!
//! E.g.
//! ```bash
//! curl "https://github.com/paritytech/polkadot/releases/download/v0.9.11/polkadot" --output /usr/local/bin/polkadot --location
//! polkadot --dev --tmp
//! ```

use subxt::ClientBuilder;

#[subxt::subxt(runtime_metadata_path = "examples/polkadot_metadata.scale")]
pub mod polkadot {}

#[async_std::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
env_logger::init();

let client = ClientBuilder::<DefaultNodeRuntime>::new().build().await?;
let mut iter = client.account_iter(None).await?;
let api = ClientBuilder::new()
.build()
.await?
.to_runtime_api::<polkadot::RuntimeApi<polkadot::DefaultConfig>>();

let mut iter = api
.storage()
.system()
.account_iter(None).await?;

while let Some((key, account)) = iter.next().await? {
println!("{:?}: {}", key, account.data.free);
println!("{}: {}", hex::encode(key), account.data.free);
}
Ok(())
}
1 change: 1 addition & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ pub use crate::{
SystemProperties,
},
storage::{
KeyIter,
StorageEntry,
StorageEntryKey,
StorageMapKey,
Expand Down