Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Metadata V15: Enrich extrinsic type info for decoding #14123

Merged
merged 29 commits into from
Jun 28, 2023
Merged

Metadata V15: Enrich extrinsic type info for decoding #14123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
da550a7
metadata-ir: Add extrinsic type info to decode address, call, sig
lexnv May 11, 2023
a51ea7b
frame-metadata: Point to unreleased branch
lexnv May 11, 2023
888ce4a
metadata-ir: Include addrees, call, signature in V15 conversion
lexnv May 11, 2023
49995b4
metadata-ir: Include extra ty
lexnv May 11, 2023
0babb53
construct_runtime: Extract address,call,sig,extra ty from tx type
lexnv May 11, 2023
768c515
frame/tests: Check metadata populates xt types correctly
lexnv May 11, 2023
ab66814
metadata-ir/tests: Add extra fields on ExtrinsicMetadataIR
lexnv May 11, 2023
52f1480
Merge remote-tracking branch 'origin/master' into lexnv/extrinsic_dec…
lexnv May 11, 2023
6610ac2
primitives/traits: Expand the `Extrinsic::SignaturePayload`
lexnv May 15, 2023
25617e0
primitives: Adjust to new `Extrinsic` associated types
lexnv May 15, 2023
10bbe10
frame/metadata: Simplify metadata generation
lexnv May 15, 2023
b823f5c
frame/example: Adjust to new interface
lexnv May 15, 2023
a94cd09
frame/tests: Adjust `extrinsic_metadata_ir_types`
lexnv May 16, 2023
1f17361
Merge remote-tracking branch 'origin/master' into lexnv/extrinsic_dec…
lexnv May 29, 2023
86e39c1
Revert the additional Extrinsic' associated types
lexnv May 29, 2023
903ebd7
primitives: Add `SignaturePayload` marker trait
lexnv May 29, 2023
37f6a09
primitives: Implement SignaturePayload for empty tuple
lexnv May 29, 2023
16b81b6
Adjust to new SignaturePayload trait
lexnv May 29, 2023
cd5c0a3
tests: Adjust `extrinsic_metadata_ir_types` to new interface
lexnv May 29, 2023
60db7bc
frame/support: Adjust pallet test
lexnv May 29, 2023
6e4f749
frame: Add Extrinsic length prefix to the metadata
lexnv May 30, 2023
0ad90f3
primitives: Populate `ExtrinsicMetadataIR` with `len_ty`
lexnv May 30, 2023
7d55ba1
Update primitives/runtime/src/traits.rs
lexnv Jun 2, 2023
03383be
Apply cargo fmt
lexnv Jun 2, 2023
0e87313
v15: Remove len type of the extrinsic
lexnv Jun 2, 2023
2c307b8
cargo: Update frame-metadata
lexnv Jun 2, 2023
e183783
Merge remote-tracking branch 'origin/master' into lexnv/extrinsic_dec…
lexnv Jun 2, 2023
1747586
Merge remote-tracking branch 'origin/master' into lexnv/extrinsic_dec…
Jun 5, 2023
9fd69cd
Merge remote-tracking branch 'origin/master' into lexnv/extrinsic_dec…
lexnv Jun 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion frame/support/procedural/src/construct_runtime/expand/metadata.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,11 +96,29 @@ pub fn expand_runtime_metadata(
// `Deref` needs a reference for resolving the function call.
let rt = #runtime;

let ty = #scrate::scale_info::meta_type::<#extrinsic>();
let address_ty = #scrate::scale_info::meta_type::<
<<#extrinsic as #scrate::sp_runtime::traits::Extrinsic>::SignaturePayload as #scrate::sp_runtime::traits::SignaturePayload>::SignatureAddress
>();
let call_ty = #scrate::scale_info::meta_type::<
<#extrinsic as #scrate::sp_runtime::traits::Extrinsic>::Call
>();
let signature_ty = #scrate::scale_info::meta_type::<
<<#extrinsic as #scrate::sp_runtime::traits::Extrinsic>::SignaturePayload as #scrate::sp_runtime::traits::SignaturePayload>::Signature
>();
let extra_ty = #scrate::scale_info::meta_type::<
<<#extrinsic as #scrate::sp_runtime::traits::Extrinsic>::SignaturePayload as #scrate::sp_runtime::traits::SignaturePayload>::SignatureExtra
>();

#scrate::metadata_ir::MetadataIR {
pallets: #scrate::sp_std::vec![ #(#pallets),* ],
extrinsic: #scrate::metadata_ir::ExtrinsicMetadataIR {
ty: #scrate::scale_info::meta_type::<#extrinsic>(),
ty,
version: <#extrinsic as #scrate::sp_runtime::traits::ExtrinsicMetadata>::VERSION,
address_ty,
call_ty,
signature_ty,
extra_ty,
signed_extensions: <
<
#extrinsic as #scrate::sp_runtime::traits::ExtrinsicMetadata
Expand Down
8 changes: 6 additions & 2 deletions frame/support/src/traits/misc.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -893,7 +893,8 @@ pub trait ExtrinsicCall: sp_runtime::traits::Extrinsic {
#[cfg(feature = "std")]
impl<Call, Extra> ExtrinsicCall for sp_runtime::testing::TestXt<Call, Extra>
where
Call: codec::Codec + Sync + Send,
Call: codec::Codec + Sync + Send + TypeInfo,
Extra: TypeInfo,
{
fn call(&self) -> &Self::Call {
&self.call
Expand All @@ -903,7 +904,10 @@ where
impl<Address, Call, Signature, Extra> ExtrinsicCall
for sp_runtime::generic::UncheckedExtrinsic<Address, Call, Signature, Extra>
where
Extra: sp_runtime::traits::SignedExtension,
Address: TypeInfo,
Call: TypeInfo,
Signature: TypeInfo,
Extra: sp_runtime::traits::SignedExtension + TypeInfo,
{
fn call(&self) -> &Self::Call {
&self.function
Expand Down
28 changes: 26 additions & 2 deletions frame/support/test/tests/pallet.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,10 @@ use sp_io::{
hashing::{blake2_128, twox_128, twox_64},
TestExternalities,
};
use sp_runtime::{DispatchError, ModuleError};
use sp_runtime::{
traits::{Extrinsic as ExtrinsicT, SignaturePayload as SignaturePayloadT},
DispatchError, ModuleError,
};

parameter_types! {
/// Used to control if the storage version should be updated.
Expand Down Expand Up @@ -1700,6 +1703,28 @@ fn metadata_ir_pallet_runtime_docs() {
assert_eq!(pallet.docs, expected);
}

#[test]
fn extrinsic_metadata_ir_types() {
let ir = Runtime::metadata_ir().extrinsic;

assert_eq!(meta_type::<<<UncheckedExtrinsic as ExtrinsicT>::SignaturePayload as SignaturePayloadT>::SignatureAddress>(), ir.address_ty);
assert_eq!(meta_type::<u64>(), ir.address_ty);

assert_eq!(meta_type::<<UncheckedExtrinsic as ExtrinsicT>::Call>(), ir.call_ty);
assert_eq!(meta_type::<RuntimeCall>(), ir.call_ty);

assert_eq!(
meta_type::<
<<UncheckedExtrinsic as ExtrinsicT>::SignaturePayload as SignaturePayloadT>::Signature,
>(),
ir.signature_ty
);
assert_eq!(meta_type::<()>(), ir.signature_ty);

assert_eq!(meta_type::<<<UncheckedExtrinsic as ExtrinsicT>::SignaturePayload as SignaturePayloadT>::SignatureExtra>(), ir.extra_ty);
assert_eq!(meta_type::<frame_system::CheckNonZeroSender<Runtime>>(), ir.extra_ty);
}

#[test]
fn test_pallet_runtime_docs() {
let docs = crate::pallet::Pallet::<Runtime>::pallet_documentation_metadata();
Expand All @@ -1713,7 +1738,6 @@ fn test_pallet_info_access() {
assert_eq!(<System as frame_support::traits::PalletInfoAccess>::name(), "System");
assert_eq!(<Example as frame_support::traits::PalletInfoAccess>::name(), "Example");
assert_eq!(<Example2 as frame_support::traits::PalletInfoAccess>::name(), "Example2");

assert_eq!(<System as frame_support::traits::PalletInfoAccess>::index(), 0);
assert_eq!(<Example as frame_support::traits::PalletInfoAccess>::index(), 1);
assert_eq!(<Example2 as frame_support::traits::PalletInfoAccess>::index(), 2);
Expand Down
4 changes: 4 additions & 0 deletions primitives/metadata-ir/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@ mod test {
extrinsic: ExtrinsicMetadataIR {
ty: meta_type::<()>(),
version: 0,
address_ty: meta_type::<()>(),
call_ty: meta_type::<()>(),
signature_ty: meta_type::<()>(),
extra_ty: meta_type::<()>(),
signed_extensions: vec![],
},
ty: meta_type::<()>(),
Expand Down
14 changes: 14 additions & 0 deletions primitives/metadata-ir/src/types.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,9 +155,19 @@ impl IntoPortable for PalletMetadataIR {
#[derive(Clone, PartialEq, Eq, Encode, Debug)]
pub struct ExtrinsicMetadataIR<T: Form = MetaForm> {
/// The type of the extrinsic.
///
/// Note: Field used for metadata V14 only.
pub ty: T::Type,
/// Extrinsic version.
pub version: u8,
/// The type of the address that signes the extrinsic
pub address_ty: T::Type,
/// The type of the outermost Call enum.
pub call_ty: T::Type,
/// The type of the extrinsic's signature.
pub signature_ty: T::Type,
/// The type of the outermost Extra enum.
pub extra_ty: T::Type,
/// The signed extensions in the order they appear in the extrinsic.
pub signed_extensions: Vec<SignedExtensionMetadataIR<T>>,
}
Expand All @@ -169,6 +179,10 @@ impl IntoPortable for ExtrinsicMetadataIR {
ExtrinsicMetadataIR {
ty: registry.register_type(&self.ty),
version: self.version,
address_ty: registry.register_type(&self.address_ty),
call_ty: registry.register_type(&self.call_ty),
signature_ty: registry.register_type(&self.signature_ty),
extra_ty: registry.register_type(&self.extra_ty),
signed_extensions: registry.map_into_portable(self.signed_extensions),
}
}
Expand Down
9 changes: 4 additions & 5 deletions primitives/metadata-ir/src/v15.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,12 +101,11 @@ impl From<ExtrinsicMetadataIR> for ExtrinsicMetadata {
fn from(ir: ExtrinsicMetadataIR) -> Self {
ExtrinsicMetadata {
version: ir.version,
address_ty: ir.address_ty,
call_ty: ir.call_ty,
signature_ty: ir.signature_ty,
extra_ty: ir.extra_ty,
signed_extensions: ir.signed_extensions.into_iter().map(Into::into).collect(),
// Note: These fields are populated by complementary PR: https://github.com/paritytech/substrate/pull/14123.
address_ty: ir.ty,
call_ty: ir.ty,
signature_ty: ir.ty,
extra_ty: ir.ty,
}
}
}
Expand Down
21 changes: 16 additions & 5 deletions primitives/runtime/src/generic/unchecked_extrinsic.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ use crate::{
generic::CheckedExtrinsic,
traits::{
self, Checkable, Extrinsic, ExtrinsicMetadata, IdentifyAccount, MaybeDisplay, Member,
SignedExtension,
SignaturePayload, SignedExtension,
},
transaction_validity::{InvalidTransaction, TransactionValidityError},
OpaqueExtrinsic,
Expand All @@ -40,6 +40,9 @@ use sp_std::{fmt, prelude::*};
/// the decoding fails.
const EXTRINSIC_FORMAT_VERSION: u8 = 4;

/// The `SingaturePayload` of `UncheckedExtrinsic`.
type UncheckedSignaturePayload<Address, Signature, Extra> = (Address, Signature, Extra);

/// A extrinsic right from the external world. This is unchecked and so
/// can contain a signature.
#[derive(PartialEq, Eq, Clone)]
Expand All @@ -50,11 +53,19 @@ where
/// The signature, address, number of extrinsics have come before from
/// the same signer and an era describing the longevity of this transaction,
/// if this is a signed extrinsic.
pub signature: Option<(Address, Signature, Extra)>,
pub signature: Option<UncheckedSignaturePayload<Address, Signature, Extra>>,
/// The function that should be called.
pub function: Call,
}

impl<Address: TypeInfo, Signature: TypeInfo, Extra: TypeInfo> SignaturePayload
for UncheckedSignaturePayload<Address, Signature, Extra>
{
type SignatureAddress = Address;
type Signature = Signature;
type SignatureExtra = Extra;
}

/// Manual [`TypeInfo`] implementation because of custom encoding. The data is a valid encoded
/// `Vec<u8>`, but requires some logic to extract the signature and payload.
///
Expand Down Expand Up @@ -103,12 +114,12 @@ impl<Address, Call, Signature, Extra: SignedExtension>
}
}

impl<Address, Call, Signature, Extra: SignedExtension> Extrinsic
for UncheckedExtrinsic<Address, Call, Signature, Extra>
impl<Address: TypeInfo, Call: TypeInfo, Signature: TypeInfo, Extra: SignedExtension + TypeInfo>
Extrinsic for UncheckedExtrinsic<Address, Call, Signature, Extra>
{
type Call = Call;

type SignaturePayload = (Address, Signature, Extra);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, it is being done already 🙈

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, a type SignaturePayloadOf<E> = (<E as Extrinsic>::X, <E as Extrinsic>::Y, <E as Extrinsic>::Z) will help prevent potential error where the order of these 3 might be mistaken.

type SignaturePayload = UncheckedSignaturePayload<Address, Signature, Extra>;

fn is_signed(&self) -> Option<bool> {
Some(self.signature.is_some())
Expand Down
19 changes: 15 additions & 4 deletions primitives/runtime/src/testing.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ use crate::{
scale_info::TypeInfo,
traits::{
self, Applyable, BlakeTwo256, Checkable, DispatchInfoOf, Dispatchable, OpaqueKeys,
PostDispatchInfoOf, SignedExtension, ValidateUnsigned,
PostDispatchInfoOf, SignaturePayload, SignedExtension, ValidateUnsigned,
},
transaction_validity::{TransactionSource, TransactionValidity, TransactionValidityError},
ApplyExtrinsicResultWithInfo, KeyTypeId,
Expand Down Expand Up @@ -279,14 +279,23 @@ where
}
}

/// The signature payload of a `TestXt`.
type TxSingaturePayload<Extra> = (u64, Extra);

impl<Extra: TypeInfo> SignaturePayload for TxSingaturePayload<Extra> {
type SignatureAddress = u64;
type Signature = ();
type SignatureExtra = Extra;
}

/// Test transaction, tuple of (sender, call, signed_extra)
/// with index only used if sender is some.
///
/// If sender is some then the transaction is signed otherwise it is unsigned.
#[derive(PartialEq, Eq, Clone, Encode, Decode, TypeInfo)]
pub struct TestXt<Call, Extra> {
/// Signature of the extrinsic.
pub signature: Option<(u64, Extra)>,
pub signature: Option<TxSingaturePayload<Extra>>,
/// Call of the extrinsic.
pub call: Call,
}
Expand Down Expand Up @@ -331,9 +340,11 @@ impl<Call: Codec + Sync + Send, Context, Extra> Checkable<Context> for TestXt<Ca
}
}

impl<Call: Codec + Sync + Send, Extra> traits::Extrinsic for TestXt<Call, Extra> {
impl<Call: Codec + Sync + Send + TypeInfo, Extra: TypeInfo> traits::Extrinsic
for TestXt<Call, Extra>
{
type Call = Call;
type SignaturePayload = (u64, Extra);
type SignaturePayload = TxSingaturePayload<Extra>;

fn is_signed(&self) -> Option<bool> {
Some(self.signature.is_some())
Expand Down
29 changes: 27 additions & 2 deletions primitives/runtime/src/traits.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1242,14 +1242,14 @@ pub trait Block: Clone + Send + Sync + Codec + Eq + MaybeSerialize + Debug + 'st
/// Something that acts like an `Extrinsic`.
pub trait Extrinsic: Sized {
/// The function call.
type Call;
type Call: TypeInfo;

/// The payload we carry for signed extrinsics.
///
/// Usually it will contain a `Signature` and
/// may include some additional data that are specific to signed
/// extrinsics.
type SignaturePayload;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of breaking most of the code we could just do

type SignaturePayload: SignaturePayload;

trait SignaturePayload {
      type Address: TypeInfo;
      type Signature: TypeInfo;
      type Extra: TypeInfo;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I like this approach more! Thanks for the feedback!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bkchr What's your opinion on having the extrinsic.ty in the metadata after this change (the message James mentioned above)?

We should have enough type info by relying on:

  • address_ty
  • call_ty
  • signature_ty
  • extra_ty

The extrinsic.ty has a manual implementation of TypeInfo that adds the above ty as generic types, but since we explicitly expose that info with this PR, I believe we should remove the extrinsic.ty.

That is unless we'll break other things

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea if someone is using this. However, frankly speaking by only having the types you listed you don't know the encoding of the Extrinsic type. While currently it may also already not being correct as it probably doesn't include the Compact<u32> length in from of the extrinsinc encoding?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! I'll add the TypeDefCompact<u32> type to the extrinsic metadata to provide even better decoding abilities to end users. Considering this, I'll remove the extrinsic_ty from the metadata, unless encountering some edge-cases from UX teams.

The type only contained the address, call, signature and extra on its type info definition, and users should be able to obtain those details (and more) directly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iirc the extrinsic type was quite useless anyway before; it mostly just said "hey we want to encode the extrinsic to some vec of bytes". It doesn't provide any useful information on how those bytes need to be formed etc.

There are already a bunch of things we must just assume already about encoding extrinsics, ie the order of the signature stuff, how the not-really-scale Option for the signature is encoded (ie it has a version number &&'d to it) etc. I've always assumed this format information to be encoded in the tx version (4 atm); if a transaction has this version, it must be encoded in this way.

The metadata, in my mind, only needs to provide the information that might vary from chain to chain (eg the actual additional params etc). This tx version can encode the actual format that this stuff is turned into an extrinsic.

All of this to say, what would adding a TypeDefCompact<u32> to the metadata help with? I'd assume it to be part of the V4 extrinsic format I guess, but perhaps I'm wrong on this :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are already a bunch of things we must just assume already about encoding extrinsics, ie the order of the signature stuff, how the not-really-scale Option for the signature is encoded (ie it has a version number &&'d to it) etc. I've always assumed this format information to be encoded in the tx version (4 atm); if a transaction has this version, it must be encoded in this way.

Good point.

The metadata, in my mind, only needs to provide the information that might vary from chain to chain (eg the actual additional params etc). This tx version can encode the actual format that this stuff is turned into an extrinsic.

Just because every body is currently using our UncheckedExtrinsic doesn't mean that this will always be the case. However, it makes sense that this is a type that we can not depict in as type metadata.

type SignaturePayload: SignaturePayload;

/// Is this `Extrinsic` signed?
/// If no information are available about signed/unsigned, `None` should be returned.
Expand All @@ -1268,6 +1268,31 @@ pub trait Extrinsic: Sized {
}
}

/// Something that acts like a [`SignaturePayload`](Extrinsic::SignaturePayload) of an
/// [`Extrinsic`].
pub trait SignaturePayload {
/// The type of the address that signed the extrinsic.
///
/// Particular to a signed extrinsic.
type SignatureAddress: TypeInfo;

/// The signature type of the extrinsic.
///
/// Particular to a signed extrinsic.
type Signature: TypeInfo;

/// The additional data that is specific to the signed extrinsic.
///
/// Particular to a signed extrinsic.
type SignatureExtra: TypeInfo;
}

impl SignaturePayload for () {
type SignatureAddress = ();
type Signature = ();
type SignatureExtra = ();
}

/// Implementor is an [`Extrinsic`] and provides metadata about this extrinsic.
pub trait ExtrinsicMetadata {
/// The format version of the `Extrinsic`.
Expand Down