FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` #2280

gavofyork · 2023-11-11T15:38:43Z

Introduces a new trait TransactionExtension to replace SignedExtension. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have General for the former and Bare for the latter. (Ultimately, the latter will be phased out as a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:

Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned")
- Bare transactions (deprecated): Gossiped, validated with ValidateUnsigned (deprecated) and the _bare_compat bits of TransactionExtension (deprecated).
- Inherents: Not gossiped, validated with ProvideInherent.
Extended (Extra data): Gossiped, validated via TransactionExtension.
- Signed transactions (with a hardcoded signature).
- General transactions (without a hardcoded signature).

TransactionExtension differs from SignedExtension because:

A signature on the underlying transaction may validly not be present.
It may alter the origin during validation.
pre_dispatch is renamed to prepare and need not contain the checks present in validate.
validate and prepare is passed an Origin rather than a AccountId.
validate may pass arbitrary information into prepare via a new user-specifiable type Val.
AdditionalSigned/additional_signed is renamed to Implicit/implicit. It is encoded for the entire transaction and passed in to each extension as a new argument to validate. This facilitates the ability of extensions to acts as underlying crypto.

There is a new DispatchTransaction trait which contains only default function impls and is impl'ed for any TransactionExtension impler. It provides several utility functions which reduce some of the tedium from using TransactionExtension (indeed, none of its regular functions should now need to be called directly).

Three transaction version discriminator ("versions") are now permissible:

0b000000100: Bare (used to be called "Unsigned"): contains Signature or Extra (extension data). After bare transactions are no longer supported, this will strictly identify an Inherents only.
0b100000100: Old-school "Signed" Transaction: contains Signature and Extra (extension data).
0b010000100: New-school "General" Transaction: contains Extra (extension data), but no Signature.

For the New-school General Transaction, it becomes trivial for authors to publish extensions to the mechanism for authorizing an Origin, e.g. through new kinds of key-signing schemes, ZK proofs, pallet state, mutations over pre-authenticated origins or any combination of the above.

Code Migration

NOW: Getting it to build

Wrap your SignedExtensions in AsTransactionExtension. This should be accompanied by renaming your aggregate type in line with the new terminology. E.g. Before:

/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;

After:

/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;

You'll also need to alter any transaction building logic to add a .into() to make the conversion happen. E.g. Before:

fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}

After:

fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}

SOON: Migrating to `TransactionExtension`

Most SignedExtensions can be trivially converted to become a TransactionExtension. There are a few things to know.

Instead of a single trait like SignedExtension, you should now implement two traits individually: TransactionExtensionBase and TransactionExtension.
Weights are now a thing and must be provided via the new function fn weight.

`TransactionExtensionBase`

This trait takes care of anything which is not dependent on types specific to your runtime, most notably Call.

AdditionalSigned/additional_signed is renamed to Implicit/implicit.
Weight must be returned by implementing the weight function. If your extension is associated with a pallet, you'll probably want to do this via the pallet's existing benchmarking infrastructure.

`TransactionExtension`

Generally:

pre_dispatch is now prepare and you should not reexecute the validate functionality in there!
You don't get an account ID any more; you get an origin instead. If you need to presume an account ID, then you can use the trait function AsSystemOriginSigner::as_system_origin_signer.
You get an additional ticket, similar to Pre, called Val. This defines data which is passed from validate into prepare. This is important since you should not be duplicating logic from validate to prepare, you need a way of passing your working from the former into the latter. This is it.
This trait takes two type parameters: Call and Context. Call is the runtime call type which used to be an associated type; you can just move it to become a type parameter for your trait impl. Context is not currently used and you can safely implement over it as an unbounded type.
There's no AccountId associated type any more. Just remove it.

Regarding validate:

You get three new parameters in validate; all can be ignored when migrating from SignedExtension.
validate returns a tuple on success; the second item in the tuple is the new ticket type Self::Val which gets passed in to prepare. If you use any information extracted during validate (off-chain and on-chain, non-mutating) in prepare (on-chain, mutating) then you can pass it through with this. For the tuple's last item, just return the origin argument.

Regarding prepare:

This is renamed from pre_dispatch, but there is one change:
FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM validate!!
(This is different to SignedExtension which was required to run the same checks in pre_dispatch as in validate.)

Regarding post_dispatch:

Since there are no unsigned transactions handled by TransactionExtension, Pre is always defined, so the first parameter is Self::Pre rather than Option<Self::Pre>.

If you make use of SignedExtension::validate_unsigned or SignedExtension::pre_dispatch_unsigned, then:

Just use the regular versions of these functions instead.
Have your logic execute in the case that the origin is None.
Ensure your transaction creation logic creates a General Transaction rather than a Bare Transaction; this means having to include all TransactionExtensions' data.
ValidateUnsigned can still be used (for now) if you need to be able to construct transactions which contain none of the extension data, however these will be phased out in stage 2 of the Transactions Horizon, so you should consider moving to an extension-centric design.

TODO

xlc · 2023-11-16T06:35:32Z

Should we have a RFC to describe the extrinsic format change? Yeah I can read the diff but I prefer to read a spec in this case.

The API changes can be considered internal details and no need to be spec'ed but extrinsic format is part of external interface and should be discussed and documented.

…gav-tx-ext

gavofyork · 2023-11-20T10:44:46Z

Should we have a RFC to describe the extrinsic format change? Yeah I can read the diff but I prefer to read a spec in this case.

The API changes can be considered internal details and no need to be spec'ed but extrinsic format is part of external interface and should be discussed and documented.

This PR makes no alterations to the protocol of the Polkadot Relay or system chains as defined by the extrinsic datagrams which are acceptable or their meaning.

Furthermore, we do not (yet) have a convention of blanket requiring preapproval via an RFC on arbitrarily small changes which would alter transaction formats used by the Polkadot Relay and system chains. If you feel that such a move would be in the interests of Polkadot, I suggest you consider writing an RFC to propose it.

…gav-tx-ext

kianenigma · 2024-03-05T10:06:12Z

substrate/frame/support/procedural/src/construct_runtime/expand/origin.rs

@@ -153,6 +153,10 @@ pub fn expand_outer_origin(
 				self.filter = #scrate::__private::sp_std::rc::Rc::new(Box::new(filter));
 			}

+			fn set_caller(&mut self, caller: OriginCaller) {


I suppose this goes back to the new tx extensions being able to alter origin?

If this is the killer feature, it should be documented in reference_docs::transaction_extensions.

This enables TransactionExtensions to alter origins during validation/preparation. This should be documented in an example of new-school transaction impl using TransactionExtension. Opened #3593 to address this.

kianenigma · 2024-03-05T10:10:08Z

substrate/primitives/runtime/src/generic/checked_extrinsic.rs

+	General(Extension),
+}
+
+// TODO: Rename ValidateUnsigned to ValidateInherent


I see that you have made follow-up issues, but don't see these being addressed in any of them. Would be good to have an issue for them too, even if they are small. They can be grouped into one issue.

This is a bit tough because the next step in Extrinsic Horizon is to get rid of unsigned transactions, and ValidateUnsigned would have to be removed and its checks moved somewhere else. With step 1 done and before step 2, renaming ValidateUnsigned to ValidateInherent wouldn't really make sense because unsigned transactions still exist and would be validated using this trait. This will probably just be removed in step 2, but we should clear up these todo's, which basically just describe a subset of the changes required in step 2.

kianenigma · 2024-03-05T10:14:55Z

substrate/primitives/runtime/src/generic/checked_extrinsic.rs

+				I::pre_dispatch(&self.function)?;
+				// TODO: Remove below once `pre_dispatch_unsigned` is removed from `LegacyExtension`
+				//   or `LegacyExtension` is removed.
+				#[allow(deprecated)]


for these, you can make issues with the deprecation label and use the process as per https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/DEPRECATION_CHECKLIST.md

Again, this should happen when we do step 2 of #2415 .

kianenigma · 2024-03-05T10:20:58Z

substrate/primitives/runtime/src/generic/unchecked_extrinsic.rs

-	/// `None` if it is unsigned or an inherent.
-	pub signature: Option<UncheckedSignaturePayload<Address, Signature, Extra>>,
+#[derive(PartialEq, Eq, Clone, Debug)]
+pub struct UncheckedExtrinsic<Address, Call, Signature, Extension> {


Are the docs for this type up to date still? Especially in regard to extrinsic types etc.

We reference this type + the diagram linked to it a lot as the source of truth for "types of extrinsic"

This is still true for old-school extrinsics but incomplete given the general transaction type. This doc comment along with the mermaid file need to be updated. Opened #3592 for this.

bkchr

I'm not yet through all the changes, but here are already some comments.

bkchr · 2024-03-05T11:04:15Z

substrate/primitives/runtime/src/traits/mod.rs

 	) -> Result<(), TransactionValidityError> {
-		for_tuples!( #( Tuple::pre_dispatch_unsigned(call, info, len)?; )* );
 		Ok(())


You changed the implementation.

It was before:

Self::validate_unsigned(call, info, len).map(|_| ()).map_err(Into::into)

Even when this is deprecated, you should not just change it.

As a default impl for a deprecated trait, I think it's ok to restore the previous behavior and call validate_unsigned 👍 .

bkchr · 2024-03-05T11:04:37Z

substrate/primitives/runtime/src/traits/mod.rs

@@ -1762,6 +1713,7 @@ pub trait GetNodeBlockType {
 /// function is called right before dispatching the call wrapped by an unsigned extrinsic. The
 /// [`validate_unsigned`](Self::validate_unsigned) function is mainly being used in the context of
 /// the transaction pool to check the validity of the call wrapped by an unsigned extrinsic.
+// TODO: Rename to ValidateBareTransaction (or just remove).


This should be removed.

I'll remove the comment, it will be addressed anyway in step 2 of #2415

bkchr · 2024-03-05T11:07:50Z

substrate/primitives/metadata-ir/src/types.rs

@@ -166,10 +166,11 @@ pub struct ExtrinsicMetadataIR<T: Form = MetaForm> {
 	pub call_ty: T::Type,
 	/// The type of the extrinsic's signature.
 	pub signature_ty: T::Type,
-	/// The type of the outermost Extra enum.
+	/// The type of the outermost Extra/Extensions enum.
+	// TODO: metadata-v16: rename this to `extension_ty`.
 	pub extra_ty: T::Type,


This can just be removed, as the type can be constructed using the extensions.

bkchr · 2024-03-05T11:17:19Z

substrate/primitives/runtime/src/generic/unchecked_extrinsic.rs

+	/// An old-school transaction extrinsic which includes a signature of some hard-coded crypto.
+	#[codec(index = 0b10000100)]
+	Signed(Address, Signature, Extension),
+	/// A new-school transaction extrinsic which does not include a signature.


These docs could be improved.

Generally people don't care what is old school or new school. It should mention an example what it is, IMO.

bkchr · 2024-03-05T11:17:50Z

substrate/primitives/runtime/src/generic/unchecked_extrinsic.rs

+	/// NOTE: In the future, once we remove `ValidateUnsigned`, this will only serve Inherent
+	/// extrinsics and thus can be renamed to `Inherent`.


Not sure why this is added here. Why not directly rename it to Inherent?

Because it still serves both unsigned extrinsics as well as inherents. Both are represented by this Bare variant but my understanding is that they are not equivalent.

bkchr · 2024-03-06T15:06:51Z

substrate/frame/support/src/transaction_extensions.rs

+)]
+#[codec(encode_bound())]
+#[codec(decode_bound())]
+pub struct VerifyMultiSignature<V: Verify>


Missing docs on what this does.

I'll address this and this by opening an issue to document this extension.

bkchr · 2024-03-06T16:09:40Z

polkadot/runtime/common/src/claims.rs

@@ -166,6 +174,13 @@ pub mod pallet {
 	#[pallet::without_storage_info]
 	pub struct Pallet<T>(_);

+	#[cfg(feature = "runtime-benchmarks")]
+	/// Helper trait to benchmark the `PrevalidateAttests` transaction extension.
+	pub trait BenchmarkHelperTrait<RuntimeCall, DispatchInfo> {


I don't really get why we need this trait at all. You have your default implementation, not sure why you can not directly use this one?

bkchr · 2024-03-06T16:17:24Z

substrate/primitives/runtime/src/transaction_validity.rs

@@ -82,6 +82,8 @@ pub enum InvalidTransaction {
 	MandatoryValidation,
 	/// The sending address is disabled or known to be invalid.
 	BadSigner,
+	/// The implicit data was unable to be calculated.
+	IndeterminateImplicit,


Basically this breaks the runtime api interface, because the host will may fail to decode this error. However, as this is an error, it is "fine".

bkchr · 2024-03-07T08:45:46Z

substrate/primitives/runtime/src/traits/transaction_extension/dispatch_transaction.rs

+		info: &Self::Info,
+		len: usize,
+	) -> Result<(ValidTransaction, Self::Val, Self::Origin), TransactionValidityError>;
+	/// Prepare and validate a transaction, ready for dispatch.


It first validates and then prepares.

bkchr · 2024-03-07T08:47:25Z

substrate/primitives/runtime/src/traits/transaction_extension/dispatch_transaction.rs

+		len: usize,
+	) -> Self::Result;
+	/// Do everything which would be done in a [dispatch_transaction](Self::dispatch_transaction),
+	/// but instead of executing the call, execute `substitute` instead. Since this doesn't actually


You are partly explaining the code, aka "call can be passed as a reference". As someone wanting to use this trait, these information are quite useless.

georgepisaltu · 2024-03-08T12:07:53Z

@bkchr opened #3623 to address your comments

bkchr · 2024-03-09T07:23:44Z

substrate/primitives/runtime/src/generic/checked_extrinsic.rs

+				Extension::validate_bare_compat(&self.function, info, len)?;
+				#[allow(deprecated)]
+				Extension::pre_dispatch_bare_compat(&self.function, info, len)?;


Before it was said that you need to do the same operations in pre_dispatch_unsigned as in validate_unsigned. You now calling here both, which probably results in doing the same checks twice. I know that this is only for the compatibility, but still.

bkchr · 2024-03-09T07:30:41Z

substrate/primitives/runtime/src/traits/transaction_extension/mod.rs

+	/// validity against current state. It should perform all checks that determine a valid
+	/// transaction, that can pay for its execution and quickly eliminate ones that are stale or


This is called for all kinds of transactions, also General transactions which do not really pay for their execution.

General transactions indeed can't pay for their execution as things stand now. These transactions would either need to be feeless with some other means of approving/validating the origin to prevent spam or people who use this General format would need to write their own logic for a transaction payment extension.

#2415 (comment)

bkchr · 2024-03-09T07:40:55Z

substrate/primitives/runtime/src/generic/checked_extrinsic.rs

+		match self.format {
+			ExtrinsicFormat::Bare => {
+				let inherent_validation = I::validate_unsigned(source, &self.function)?;
+				#[allow(deprecated)]
+				let legacy_validation = Extension::validate_bare_compat(&self.function, info, len)?;
+				Ok(legacy_validation.combine_with(inherent_validation))
+			},
+			ExtrinsicFormat::Signed(ref signer, ref extension) => {
+				let origin = Some(signer.clone()).into();
+				extension.validate_only(origin, &self.function, info, len).map(|x| x.0)
+			},
+			ExtrinsicFormat::General(ref extension) =>
+				extension.validate_only(None.into(), &self.function, info, len).map(|x| x.0),


That the unsigned validation is not directly lifted to the TransactionExtension level is not good. IMO it is now just more confusing than before.

Another issue being that you call here just validate_only for the General transaction. How will you find out that a transaction is allowed being a General transaction? The old ValidateUnsigned logic required explicit approval for calls to be accepted as unsigned transaction. This will now be skipped here.

This basically means that I could now DOS the network with General transaction that are invalid. (Currently it would be prevented by the broken transaction-payment transaction extension. However, this is nothing we should count on.)

That the unsigned validation is not directly lifted to the TransactionExtension level is not good. IMO it is now just more confusing than before.

This is step 2 of Extrinsic Horizon. I think it would have been really difficult to fit these changes in this PR too.

This basically means that I could now DOS the network with General transaction that are invalid. (Currently it would be prevented by the broken transaction-payment transaction extension.

Well, not now because this PR only introduces the framework for General transactions, but they are not really supported anywhere else. It's true that the transaction payment extension would need to change to accomodate for General transactions as you can't really pay for a transaction without an account. These changes should follow naturally when we do step 2 mentioned above, where we deal with unsigned transactions as well.

but they are not really supported anywhere else

I can just construct them and send them to the chain. Any chain that will not use the tx-payment signed extension will (if they don't have any other schemes against these General transactions) be able to be dosed. Aka I can just send million of these transactions to the nodes and they will happily accept them.

bkchr · 2024-03-09T07:47:05Z

substrate/frame/transaction-payment/asset-tx-payment/src/lib.rs

 		use pallet_transaction_payment::ChargeTransactionPayment;
-		let (fee, _) = self.withdraw_fee(who, call, info, len)?;
+		let who = origin.as_system_origin_signer().ok_or(InvalidTransaction::BadSigner)?;


This is wrong. When this is called for a General transaction that has no signer, you will reject it.

It's true that the transaction payment extension only works with signed transactions right now and it needs to be changed to support General transactions. This check will be easier when we get to the point where Bare transactions are strictly inherents and everything else is a General transaction, where we will be able to probe into the origin type, account or pallet specific with specific validation.

bkchr · 2024-03-09T07:48:43Z

substrate/frame/transaction-payment/src/lib.rs

+		(ValidTransaction, Self::Val, <T::RuntimeCall as Dispatchable>::RuntimeOrigin),
+		TransactionValidityError,
+	> {
+		let who = frame_system::ensure_signed(origin.clone())


Same issue.

bkchr · 2024-03-09T07:56:51Z

substrate/primitives/runtime/src/traits/transaction_extension/mod.rs

+/// Implict
+#[macro_export]
+macro_rules! impl_tx_ext_default {


Not really sure we should have this macro.

But if we want to have it, it should have proper docs.

It's just a convenience macro to avoid writing boilerplate. It should state what the default impl is, I agree.

I mean I see what it does. However, users will not know what it does and what the expected syntax is.

bkchr · 2024-03-09T07:58:23Z

substrate/primitives/runtime/src/traits/transaction_extension/mod.rs

+/// Base for [TransactionExtension]s; this contains the associated types and does not require any
+/// generic parameterization.
+pub trait TransactionExtensionBase: TransactionExtensionInterior {


Why do we need this? I couldn't find anything where this is really required to be separate? Asking because I think that having two traits makes this more complicated to handle, as you need to implement both traits always.

…gnedExtension` (paritytech#2280)" This reverts commit fd5f929.

…gnedExtension` (#2280)" (#3665) This PR reverts #2280 which introduced `TransactionExtension` to replace `SignedExtension`. As a result of the discussion [here](#3623 (comment)), the changes will be reverted for now with plans to reintroduce the concept in the future. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io>

… for `SignedExtension` (paritytech#2280)" (paritytech#3665)" This reverts commit bbd51ce.

* master: (65 commits) collator protocol changes for elastic scaling (validator side) (#3302) Contracts use polkavm workspace deps (#3715) Add elastic scaling support in ParaInherent BenchBuilder (#3690) Removes `as [disambiguation_path]` from `derive_impl` usage (#3652) fix(paseo-spec): New Paseo Bootnodes (#3674) Improve Penpal runtime + emulated tests (#3543) Staking ledger bonding fixes (#3639) DescribeAllTerminal for HashedDescription (#3349) Increase timeout for assertions (#3680) Add subsystems regression tests to CI (#3527) Always print connectivity report (#3677) Revert "FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280)" (#3665) authority-discovery: Add log for debugging DHT authority records (#3668) Construct Runtime v2 (#1378) Support for `keyring` in runtimes (#2044) Add api-name in `cannot query the runtime API version` warning (#3653) Add a PolkaVM-based executor (#3458) Adds default config for assets pallet (#3637) Bump handlebars from 4.3.7 to 5.1.0 (#3248) [Collator Selection] Fix weight refund for `set_candidacy_bond` (#3643) ...

…gnedExtension` (paritytech#2280)" (paritytech#3665) This PR reverts paritytech#2280 which introduced `TransactionExtension` to replace `SignedExtension`. As a result of the discussion [here](paritytech#3623 (comment)), the changes will be reverted for now with plans to reintroduce the concept in the future. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io>

…nsion` (paritytech#2280) Closes paritytech#2160 First part of [Extrinsic Horizon](paritytech#2415) Introduces a new trait `TransactionExtension` to replace `SignedExtension`. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures. Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have __*General*__ for the former and __*Bare*__ for the latter. (Ultimately, the latter will be phased out as a type of transaction, and Bare will only be used for Inherents.) Types of extrinsic are now therefore: - Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned") - Bare transactions (deprecated): Gossiped, validated with `ValidateUnsigned` (deprecated) and the `_bare_compat` bits of `TransactionExtension` (deprecated). - Inherents: Not gossiped, validated with `ProvideInherent`. - Extended (Extra data): Gossiped, validated via `TransactionExtension`. - Signed transactions (with a hardcoded signature). - General transactions (without a hardcoded signature). `TransactionExtension` differs from `SignedExtension` because: - A signature on the underlying transaction may validly not be present. - It may alter the origin during validation. - `pre_dispatch` is renamed to `prepare` and need not contain the checks present in `validate`. - `validate` and `prepare` is passed an `Origin` rather than a `AccountId`. - `validate` may pass arbitrary information into `prepare` via a new user-specifiable type `Val`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. It is encoded *for the entire transaction* and passed in to each extension as a new argument to `validate`. This facilitates the ability of extensions to acts as underlying crypto. There is a new `DispatchTransaction` trait which contains only default function impls and is impl'ed for any `TransactionExtension` impler. It provides several utility functions which reduce some of the tedium from using `TransactionExtension` (indeed, none of its regular functions should now need to be called directly). Three transaction version discriminator ("versions") are now permissible: - 0b000000100: Bare (used to be called "Unsigned"): contains Signature or Extra (extension data). After bare transactions are no longer supported, this will strictly identify an Inherents only. - 0b100000100: Old-school "Signed" Transaction: contains Signature and Extra (extension data). - 0b010000100: New-school "General" Transaction: contains Extra (extension data), but no Signature. For the New-school General Transaction, it becomes trivial for authors to publish extensions to the mechanism for authorizing an Origin, e.g. through new kinds of key-signing schemes, ZK proofs, pallet state, mutations over pre-authenticated origins or any combination of the above. ## Code Migration ### NOW: Getting it to build Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be accompanied by renaming your aggregate type in line with the new terminology. E.g. Before: ```rust /// The SignedExtension to the basic transaction logic. pub type SignedExtra = ( /* snip */ MySpecialSignedExtension, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>; ``` After: ```rust /// The extension to the basic transaction logic. pub type TxExtension = ( /* snip */ AsTransactionExtension<MySpecialSignedExtension>, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>; ``` You'll also need to alter any transaction building logic to add a `.into()` to make the conversion happen. E.g. Before: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let extra: SignedExtra = ( /* snip */ MySpecialSignedExtension::new(/* snip */), ); let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), extra, ) } ``` After: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let tx_ext: TxExtension = ( /* snip */ MySpecialSignedExtension::new(/* snip */).into(), ); let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), tx_ext, ) } ``` ### SOON: Migrating to `TransactionExtension` Most `SignedExtension`s can be trivially converted to become a `TransactionExtension`. There are a few things to know. - Instead of a single trait like `SignedExtension`, you should now implement two traits individually: `TransactionExtensionBase` and `TransactionExtension`. - Weights are now a thing and must be provided via the new function `fn weight`. #### `TransactionExtensionBase` This trait takes care of anything which is not dependent on types specific to your runtime, most notably `Call`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. - Weight must be returned by implementing the `weight` function. If your extension is associated with a pallet, you'll probably want to do this via the pallet's existing benchmarking infrastructure. #### `TransactionExtension` Generally: - `pre_dispatch` is now `prepare` and you *should not reexecute the `validate` functionality in there*! - You don't get an account ID any more; you get an origin instead. If you need to presume an account ID, then you can use the trait function `AsSystemOriginSigner::as_system_origin_signer`. - You get an additional ticket, similar to `Pre`, called `Val`. This defines data which is passed from `validate` into `prepare`. This is important since you should not be duplicating logic from `validate` to `prepare`, you need a way of passing your working from the former into the latter. This is it. - This trait takes two type parameters: `Call` and `Context`. `Call` is the runtime call type which used to be an associated type; you can just move it to become a type parameter for your trait impl. `Context` is not currently used and you can safely implement over it as an unbounded type. - There's no `AccountId` associated type any more. Just remove it. Regarding `validate`: - You get three new parameters in `validate`; all can be ignored when migrating from `SignedExtension`. - `validate` returns a tuple on success; the second item in the tuple is the new ticket type `Self::Val` which gets passed in to `prepare`. If you use any information extracted during `validate` (off-chain and on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can pass it through with this. For the tuple's last item, just return the `origin` argument. Regarding `prepare`: - This is renamed from `pre_dispatch`, but there is one change: - FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM `validate`!! - (This is different to `SignedExtension` which was required to run the same checks in `pre_dispatch` as in `validate`.) Regarding `post_dispatch`: - Since there are no unsigned transactions handled by `TransactionExtension`, `Pre` is always defined, so the first parameter is `Self::Pre` rather than `Option<Self::Pre>`. If you make use of `SignedExtension::validate_unsigned` or `SignedExtension::pre_dispatch_unsigned`, then: - Just use the regular versions of these functions instead. - Have your logic execute in the case that the `origin` is `None`. - Ensure your transaction creation logic creates a General Transaction rather than a Bare Transaction; this means having to include all `TransactionExtension`s' data. - `ValidateUnsigned` can still be used (for now) if you need to be able to construct transactions which contain none of the extension data, however these will be phased out in stage 2 of the Transactions Horizon, so you should consider moving to an extension-centric design. ## TODO - [x] Introduce `CheckSignature` impl of `TransactionExtension` to ensure it's possible to have crypto be done wholly in a `TransactionExtension`. - [x] Deprecate `SignedExtension` and move all uses in codebase to `TransactionExtension`. - [x] `ChargeTransactionPayment` - [x] `DummyExtension` - [x] `ChargeAssetTxPayment` (asset-tx-payment) - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment) - [x] `CheckWeight` - [x] `CheckTxVersion` - [x] `CheckSpecVersion` - [x] `CheckNonce` - [x] `CheckNonZeroSender` - [x] `CheckMortality` - [x] `CheckGenesis` - [x] `CheckOnlySudoAccount` - [x] `WatchDummy` - [x] `PrevalidateAttests` - [x] `GenericSignedExtension` - [x] `SignedExtension` (chain-polkadot-bulletin) - [x] `RefundSignedExtensionAdapter` - [x] Implement `fn weight` across the board. - [ ] Go through all pre-existing extensions which assume an account signer and explicitly handle the possibility of another kind of origin. - [x] `CheckNonce` should probably succeed in the case of a non-account origin. - [x] `CheckNonZeroSender` should succeed in the case of a non-account origin. - [x] `ChargeTransactionPayment` and family should fail in the case of a non-account origin. - [ ] - [x] Fix any broken tests. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Nikhil Gupta <17176722+gupnik@users.noreply.github.com> Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Maciej <maciej.zyszkiewicz@parity.io> Co-authored-by: Javier Viola <javier@parity.io> Co-authored-by: Marcin S. <marcin@realemail.net> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Javier Bullrich <javier@bullrich.dev> Co-authored-by: Koute <koute@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: Vladimir Istyufeev <vladimir@parity.io> Co-authored-by: Ross Bulat <ross@parity.io> Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Sebastian Kunert <skunert49@gmail.com> Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com> Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com> Co-authored-by: Julian Eager <eagr@tutanota.com> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Dónal Murray <donal.murray@parity.io> Co-authored-by: yjh <yjh465402634@gmail.com> Co-authored-by: Tom Mi <tommi@niemi.lol> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Will | Paradox | ParaNodes.io <79228812+paradox-tt@users.noreply.github.com> Co-authored-by: Bastian Köcher <info@kchr.de> Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> Co-authored-by: Joshy Orndorff <git-user-email.h0ly5@simplelogin.com> Co-authored-by: PG Herveou <pgherveou@gmail.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Juan Girini <juangirini@gmail.com> Co-authored-by: bader y <ibnbassem@gmail.com> Co-authored-by: James Wilson <james@jsdw.me> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: asynchronous rob <rphmeier@gmail.com> Co-authored-by: Parth <desaiparth08@gmail.com> Co-authored-by: Andrew Jones <ascjones@gmail.com> Co-authored-by: Jonathan Udd <jonathan@dwellir.com> Co-authored-by: Serban Iorga <serban@parity.io> Co-authored-by: Egor_P <egor@parity.io> Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: Evgeny Snitko <evgeny@parity.io> Co-authored-by: Just van Stam <vstam1@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com> Co-authored-by: dzmitry-lahoda <dzmitry@lahoda.pro> Co-authored-by: zhiqiangxu <652732310@qq.com> Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com> Co-authored-by: Anwesh <anweshknayak@gmail.com> Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Jegor Sidorenko <5252494+jsidorenko@users.noreply.github.com> Co-authored-by: Muharem <ismailov.m.h@gmail.com> Co-authored-by: joepetrowski <joe@parity.io> Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com> Co-authored-by: Gabriel Facco de Arruda <arrudagates@gmail.com> Co-authored-by: Squirrel <gilescope@gmail.com> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: georgepisaltu <george.pisaltu@parity.io> Co-authored-by: command-bot <>

…gnedExtension` (paritytech#2280)" (paritytech#3665) This PR reverts paritytech#2280 which introduced `TransactionExtension` to replace `SignedExtension`. As a result of the discussion [here](paritytech#3623 (comment)), the changes will be reverted for now with plans to reintroduce the concept in the future. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io>

* Add two new zombienet tests for bridges (manual run) (#3072) extracted useful code from #2982 This PR: - adds test 2 for Rococo <> Westend bridge: checks that relayer doesn't submit any extra headers while there are no any messages; - adds test 3 for Rococo <> Westend bridge: checks that relayer doesn't submit any extra headers when there are messages; - fixes most of comments from #2439 (like: log names, ability to run specify test number when calling `run-tests.sh`). Right now of all our tests, only test 2 is working (until BHs will be upgraded to use async backing), so you can test it with `./bridges/zombienet/run-tests.sh --test 2` locally. (cherry picked from commit 2e6067d) * [cumulus] Improved check for sane bridge fees calculations (#3175) - [x] change constants when CI fails (should fail :) ) On the AssetHubRococo: 1701175800126 -> 1700929825257 = 0.15 % decreased. ``` Feb 02 12:59:05.520 ERROR bridges::estimate: `bridging::XcmBridgeHubRouterBaseFee` actual value: 1701175800126 for runtime: statemine-1006000 (statemine-0.tx14.au1) Feb 02 13:02:40.647 ERROR bridges::estimate: `bridging::XcmBridgeHubRouterBaseFee` actual value: 1700929825257 for runtime: statemine-1006000 (statemine-0.tx14.au1) ``` On the AssetHubWestend: 2116038876326 -> 1641718372993 = 22.4 % decreased. ``` Feb 02 12:56:00.880 ERROR bridges::estimate: `bridging::XcmBridgeHubRouterBaseFee` actual value: 2116038876326 for runtime: westmint-1006000 (westmint-0.tx14.au1) Feb 02 13:04:42.515 ERROR bridges::estimate: `bridging::XcmBridgeHubRouterBaseFee` actual value: 1641718372993 for runtime: westmint-1006000 (westmint-0.tx14.au1) ``` (cherry picked from commit 74b597f) * Enable async backing on all testnet system chains (#2949) Built on top of #2826 which was a trial run. Guide: https://github.com/w3f/polkadot-wiki/blob/master/docs/maintain/maintain-guides-async-backing.md --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io> Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: Dónal Murray <donal.murray@parity.io> Co-authored-by: Dmitry Sinyavin <dmitry.sinyavin@parity.io> Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com> Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com> Co-authored-by: Bastian Köcher <info@kchr.de> Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com> (cherry picked from commit 700d5f8) * Introduce submit_finality_proof_ex call to bridges GRANDPA pallet (#3225) backport of paritytech/parity-bridges-common#2821 (see detailed description there) (cherry picked from commit a462207) * Bridge zombienet tests refactoring (#3260) Related to #3242 Reorganizing the bridge zombienet tests in order to: - separate the environment spawning from the actual tests - offer better control over the tests and some possibility to orchestrate them as opposed to running everything from the zndsl file Only rewrote the asset transfer test using this new "framework". The old logic and old tests weren't functionally modified or deleted. The plan is to get feedback on this approach first and if this is agreed upon, migrate the other 2 tests later in separate PRs and also do other improvements later. (cherry picked from commit dfc8e46) * Bridges: add test 0002 to CI (#3310) Bridges: add test 0002 to CI (cherry picked from commit 1b66bb5) * Bridge zombienet tests - move all test scripts to the same folder (#3333) Related to #3242 (cherry picked from commit 5fc7622) * Lift dependencies to the workspace (Part 2/x) (#3366) Lifting some more dependencies to the workspace. Just using the most-often updated ones for now. It can be reproduced locally. ```sh $ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*" $ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix $ taplo format --config .config/taplo.toml ``` --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> (cherry picked from commit e89d0fc) * Add support for BHP local and BHK local (#3443) Related to #3400 Extracting small parts of #3429 into separate PR: - Add support for BHP local and BHK local - Increase the timeout for the bridge zomienet tests (cherry picked from commit e4b6b8c) * Bridge zombienet tests: move all "framework" files under one folder (#3462) Related to #3400 Moving all bridges testing "framework" files under one folder in order to be able to download the entire folder when we want to add tests in other repos No significant functional changes (cherry picked from commit 6fc1d41) * Bridge zombienet tests: Check amount received at destination (#3490) Related to #3475 (cherry picked from commit 2cdda0e) * FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280) Closes #2160 First part of [Extrinsic Horizon](#2415) Introduces a new trait `TransactionExtension` to replace `SignedExtension`. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures. Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have __*General*__ for the former and __*Bare*__ for the latter. (Ultimately, the latter will be phased out as a type of transaction, and Bare will only be used for Inherents.) Types of extrinsic are now therefore: - Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned") - Bare transactions (deprecated): Gossiped, validated with `ValidateUnsigned` (deprecated) and the `_bare_compat` bits of `TransactionExtension` (deprecated). - Inherents: Not gossiped, validated with `ProvideInherent`. - Extended (Extra data): Gossiped, validated via `TransactionExtension`. - Signed transactions (with a hardcoded signature). - General transactions (without a hardcoded signature). `TransactionExtension` differs from `SignedExtension` because: - A signature on the underlying transaction may validly not be present. - It may alter the origin during validation. - `pre_dispatch` is renamed to `prepare` and need not contain the checks present in `validate`. - `validate` and `prepare` is passed an `Origin` rather than a `AccountId`. - `validate` may pass arbitrary information into `prepare` via a new user-specifiable type `Val`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. It is encoded *for the entire transaction* and passed in to each extension as a new argument to `validate`. This facilitates the ability of extensions to acts as underlying crypto. There is a new `DispatchTransaction` trait which contains only default function impls and is impl'ed for any `TransactionExtension` impler. It provides several utility functions which reduce some of the tedium from using `TransactionExtension` (indeed, none of its regular functions should now need to be called directly). Three transaction version discriminator ("versions") are now permissible: - 0b000000100: Bare (used to be called "Unsigned"): contains Signature or Extra (extension data). After bare transactions are no longer supported, this will strictly identify an Inherents only. - 0b100000100: Old-school "Signed" Transaction: contains Signature and Extra (extension data). - 0b010000100: New-school "General" Transaction: contains Extra (extension data), but no Signature. For the New-school General Transaction, it becomes trivial for authors to publish extensions to the mechanism for authorizing an Origin, e.g. through new kinds of key-signing schemes, ZK proofs, pallet state, mutations over pre-authenticated origins or any combination of the above. Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be accompanied by renaming your aggregate type in line with the new terminology. E.g. Before: ```rust /// The SignedExtension to the basic transaction logic. pub type SignedExtra = ( /* snip */ MySpecialSignedExtension, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>; ``` After: ```rust /// The extension to the basic transaction logic. pub type TxExtension = ( /* snip */ AsTransactionExtension<MySpecialSignedExtension>, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>; ``` You'll also need to alter any transaction building logic to add a `.into()` to make the conversion happen. E.g. Before: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let extra: SignedExtra = ( /* snip */ MySpecialSignedExtension::new(/* snip */), ); let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), extra, ) } ``` After: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let tx_ext: TxExtension = ( /* snip */ MySpecialSignedExtension::new(/* snip */).into(), ); let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), tx_ext, ) } ``` Most `SignedExtension`s can be trivially converted to become a `TransactionExtension`. There are a few things to know. - Instead of a single trait like `SignedExtension`, you should now implement two traits individually: `TransactionExtensionBase` and `TransactionExtension`. - Weights are now a thing and must be provided via the new function `fn weight`. This trait takes care of anything which is not dependent on types specific to your runtime, most notably `Call`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. - Weight must be returned by implementing the `weight` function. If your extension is associated with a pallet, you'll probably want to do this via the pallet's existing benchmarking infrastructure. Generally: - `pre_dispatch` is now `prepare` and you *should not reexecute the `validate` functionality in there*! - You don't get an account ID any more; you get an origin instead. If you need to presume an account ID, then you can use the trait function `AsSystemOriginSigner::as_system_origin_signer`. - You get an additional ticket, similar to `Pre`, called `Val`. This defines data which is passed from `validate` into `prepare`. This is important since you should not be duplicating logic from `validate` to `prepare`, you need a way of passing your working from the former into the latter. This is it. - This trait takes two type parameters: `Call` and `Context`. `Call` is the runtime call type which used to be an associated type; you can just move it to become a type parameter for your trait impl. `Context` is not currently used and you can safely implement over it as an unbounded type. - There's no `AccountId` associated type any more. Just remove it. Regarding `validate`: - You get three new parameters in `validate`; all can be ignored when migrating from `SignedExtension`. - `validate` returns a tuple on success; the second item in the tuple is the new ticket type `Self::Val` which gets passed in to `prepare`. If you use any information extracted during `validate` (off-chain and on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can pass it through with this. For the tuple's last item, just return the `origin` argument. Regarding `prepare`: - This is renamed from `pre_dispatch`, but there is one change: - FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM `validate`!! - (This is different to `SignedExtension` which was required to run the same checks in `pre_dispatch` as in `validate`.) Regarding `post_dispatch`: - Since there are no unsigned transactions handled by `TransactionExtension`, `Pre` is always defined, so the first parameter is `Self::Pre` rather than `Option<Self::Pre>`. If you make use of `SignedExtension::validate_unsigned` or `SignedExtension::pre_dispatch_unsigned`, then: - Just use the regular versions of these functions instead. - Have your logic execute in the case that the `origin` is `None`. - Ensure your transaction creation logic creates a General Transaction rather than a Bare Transaction; this means having to include all `TransactionExtension`s' data. - `ValidateUnsigned` can still be used (for now) if you need to be able to construct transactions which contain none of the extension data, however these will be phased out in stage 2 of the Transactions Horizon, so you should consider moving to an extension-centric design. - [x] Introduce `CheckSignature` impl of `TransactionExtension` to ensure it's possible to have crypto be done wholly in a `TransactionExtension`. - [x] Deprecate `SignedExtension` and move all uses in codebase to `TransactionExtension`. - [x] `ChargeTransactionPayment` - [x] `DummyExtension` - [x] `ChargeAssetTxPayment` (asset-tx-payment) - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment) - [x] `CheckWeight` - [x] `CheckTxVersion` - [x] `CheckSpecVersion` - [x] `CheckNonce` - [x] `CheckNonZeroSender` - [x] `CheckMortality` - [x] `CheckGenesis` - [x] `CheckOnlySudoAccount` - [x] `WatchDummy` - [x] `PrevalidateAttests` - [x] `GenericSignedExtension` - [x] `SignedExtension` (chain-polkadot-bulletin) - [x] `RefundSignedExtensionAdapter` - [x] Implement `fn weight` across the board. - [ ] Go through all pre-existing extensions which assume an account signer and explicitly handle the possibility of another kind of origin. - [x] `CheckNonce` should probably succeed in the case of a non-account origin. - [x] `CheckNonZeroSender` should succeed in the case of a non-account origin. - [x] `ChargeTransactionPayment` and family should fail in the case of a non-account origin. - [ ] - [x] Fix any broken tests. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Nikhil Gupta <17176722+gupnik@users.noreply.github.com> Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Maciej <maciej.zyszkiewicz@parity.io> Co-authored-by: Javier Viola <javier@parity.io> Co-authored-by: Marcin S. <marcin@realemail.net> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Javier Bullrich <javier@bullrich.dev> Co-authored-by: Koute <koute@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: Vladimir Istyufeev <vladimir@parity.io> Co-authored-by: Ross Bulat <ross@parity.io> Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Sebastian Kunert <skunert49@gmail.com> Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com> Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com> Co-authored-by: Julian Eager <eagr@tutanota.com> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Dónal Murray <donal.murray@parity.io> Co-authored-by: yjh <yjh465402634@gmail.com> Co-authored-by: Tom Mi <tommi@niemi.lol> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Will | Paradox | ParaNodes.io <79228812+paradox-tt@users.noreply.github.com> Co-authored-by: Bastian Köcher <info@kchr.de> Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> Co-authored-by: Joshy Orndorff <git-user-email.h0ly5@simplelogin.com> Co-authored-by: PG Herveou <pgherveou@gmail.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Juan Girini <juangirini@gmail.com> Co-authored-by: bader y <ibnbassem@gmail.com> Co-authored-by: James Wilson <james@jsdw.me> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: asynchronous rob <rphmeier@gmail.com> Co-authored-by: Parth <desaiparth08@gmail.com> Co-authored-by: Andrew Jones <ascjones@gmail.com> Co-authored-by: Jonathan Udd <jonathan@dwellir.com> Co-authored-by: Serban Iorga <serban@parity.io> Co-authored-by: Egor_P <egor@parity.io> Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: Evgeny Snitko <evgeny@parity.io> Co-authored-by: Just van Stam <vstam1@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com> Co-authored-by: dzmitry-lahoda <dzmitry@lahoda.pro> Co-authored-by: zhiqiangxu <652732310@qq.com> Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com> Co-authored-by: Anwesh <anweshknayak@gmail.com> Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Jegor Sidorenko <5252494+jsidorenko@users.noreply.github.com> Co-authored-by: Muharem <ismailov.m.h@gmail.com> Co-authored-by: joepetrowski <joe@parity.io> Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com> Co-authored-by: Gabriel Facco de Arruda <arrudagates@gmail.com> Co-authored-by: Squirrel <gilescope@gmail.com> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: georgepisaltu <george.pisaltu@parity.io> Co-authored-by: command-bot <> (cherry picked from commit fd5f929) * Revert "FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280)" (#3665) This PR reverts #2280 which introduced `TransactionExtension` to replace `SignedExtension`. As a result of the discussion [here](#3623 (comment)), the changes will be reverted for now with plans to reintroduce the concept in the future. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io> (cherry picked from commit bbd51ce) * Increase timeout for assertions (#3680) Prevents timeouts in ci like https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5516019 (cherry picked from commit c4c9257) * Removes `as [disambiguation_path]` from `derive_impl` usage (#3652) Step in #171 This PR removes `as [disambiguation_path]` syntax from `derive_impl` usage across the polkadot-sdk as introduced in #3505 (cherry picked from commit 7099f6e) * Fix typo (#3691) (cherry picked from commit 6b1179f) * Bridge zombienet tests: remove unneeded accounts (#3700) Bridge zombienet tests: remove unneeded accounts (cherry picked from commit 0c6c837) * Fix typos (#3753) (cherry picked from commit 7241a8d) * Update polkadot-sdk refs * Fix dependency conflicts * Fix build * cargo fmt * Fix spellcheck test --------- Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com> Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com> Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com> Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com> Co-authored-by: jokess123 <163112061+jokess123@users.noreply.github.com> Co-authored-by: slicejoke <163888128+slicejoke@users.noreply.github.com>

…dExtension` (#3685) Original PR #2280 reverted in #3665 This PR reintroduces the reverted functionality with additional changes, related effort [here](#3623). Description is copied over from the original PR First part of [Extrinsic Horizon](#2415) Introduces a new trait `TransactionExtension` to replace `SignedExtension`. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures. Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have __*General*__ for the former and __*Bare*__ for the latter. (Ultimately, the latter will be phased out as a type of transaction, and Bare will only be used for Inherents.) Types of extrinsic are now therefore: - Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned") - Bare transactions (deprecated): Gossiped, validated with `ValidateUnsigned` (deprecated) and the `_bare_compat` bits of `TransactionExtension` (deprecated). - Inherents: Not gossiped, validated with `ProvideInherent`. - Extended (Extra data): Gossiped, validated via `TransactionExtension`. - Signed transactions (with a hardcoded signature) in extrinsic v4. - General transactions (without a hardcoded signature) in extrinsic v5. `TransactionExtension` differs from `SignedExtension` because: - A signature on the underlying transaction may validly not be present. - It may alter the origin during validation. - `pre_dispatch` is renamed to `prepare` and need not contain the checks present in `validate`. - `validate` and `prepare` is passed an `Origin` rather than a `AccountId`. - `validate` may pass arbitrary information into `prepare` via a new user-specifiable type `Val`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. It is encoded *for the entire transaction* and passed in to each extension as a new argument to `validate`. This facilitates the ability of extensions to acts as underlying crypto. There is a new `DispatchTransaction` trait which contains only default function impls and is impl'ed for any `TransactionExtension` impler. It provides several utility functions which reduce some of the tedium from using `TransactionExtension` (indeed, none of its regular functions should now need to be called directly). Three transaction version discriminator ("versions") are now permissible (RFC [here](polkadot-fellows/RFCs#84)) in extrinsic version 5: - 0b00000100 or 0b00000101: Bare (used to be called "Unsigned"): contains Signature or Extra (extension data). After bare transactions are no longer supported, this will strictly identify an Inherents only. Available in both extrinsic versions 4 and 5. - 0b10000100: Old-school "Signed" Transaction: contains Signature, Extra (extension data) and an extension version byte, introduced as part of [RFC99](https://github.com/polkadot-fellows/RFCs/blob/main/text/0099-transaction-extension-version.md). Still available as part of extrinsic v4. - 0b01000101: New-school "General" Transaction: contains Extra (extension data) and an extension version byte, as per RFC99, but no Signature. Only available in extrinsic v5. For the New-school General Transaction, it becomes trivial for authors to publish extensions to the mechanism for authorizing an Origin, e.g. through new kinds of key-signing schemes, ZK proofs, pallet state, mutations over pre-authenticated origins or any combination of the above. `UncheckedExtrinsic` still maintains encode/decode backwards compatibility with extrinsic version 4, where the first byte was encoded as: - 0b00000100 - Unsigned transactions - 0b10000100 - Old-school Signed transactions, without the extension version byte Now, `UncheckedExtrinsic` contains a `Preamble` and the actual call. The `Preamble` describes the type of extrinsic as follows: ```rust /// A "header" for extrinsics leading up to the call itself. Determines the type of extrinsic and /// holds any necessary specialized data. #[derive(Eq, PartialEq, Clone)] pub enum Preamble<Address, Signature, Extension> { /// An extrinsic without a signature or any extension. This means it's either an inherent or /// an old-school "Unsigned" (we don't use that terminology any more since it's confusable with /// the general transaction which is without a signature but does have an extension). /// /// NOTE: In the future, once we remove `ValidateUnsigned`, this will only serve Inherent /// extrinsics and thus can be renamed to `Inherent`. Bare(ExtrinsicVersion), /// An old-school transaction extrinsic which includes a signature of some hard-coded crypto. /// Available only on extrinsic version 4. Signed(Address, Signature, ExtensionVersion, Extension), /// A new-school transaction extrinsic which does not include a signature by default. The /// origin authorization, through signatures or other means, is performed by the transaction /// extension in this extrinsic. Available starting with extrinsic version 5. General(ExtensionVersion, Extension), } ``` ## Code Migration ### NOW: Getting it to build Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be accompanied by renaming your aggregate type in line with the new terminology. E.g. Before: ```rust /// The SignedExtension to the basic transaction logic. pub type SignedExtra = ( /* snip */ MySpecialSignedExtension, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>; ``` After: ```rust /// The extension to the basic transaction logic. pub type TxExtension = ( /* snip */ AsTransactionExtension<MySpecialSignedExtension>, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>; ``` You'll also need to alter any transaction building logic to add a `.into()` to make the conversion happen. E.g. Before: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let extra: SignedExtra = ( /* snip */ MySpecialSignedExtension::new(/* snip */), ); let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), extra, ) } ``` After: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let tx_ext: TxExtension = ( /* snip */ MySpecialSignedExtension::new(/* snip */).into(), ); let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), tx_ext, ) } ``` ### SOON: Migrating to `TransactionExtension` Most `SignedExtension`s can be trivially converted to become a `TransactionExtension`. There are a few things to know. - Instead of a single trait like `SignedExtension`, you should now implement two traits individually: `TransactionExtensionBase` and `TransactionExtension`. - Weights are now a thing and must be provided via the new function `fn weight`. #### `TransactionExtensionBase` This trait takes care of anything which is not dependent on types specific to your runtime, most notably `Call`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. - Weight must be returned by implementing the `weight` function. If your extension is associated with a pallet, you'll probably want to do this via the pallet's existing benchmarking infrastructure. #### `TransactionExtension` Generally: - `pre_dispatch` is now `prepare` and you *should not reexecute the `validate` functionality in there*! - You don't get an account ID any more; you get an origin instead. If you need to presume an account ID, then you can use the trait function `AsSystemOriginSigner::as_system_origin_signer`. - You get an additional ticket, similar to `Pre`, called `Val`. This defines data which is passed from `validate` into `prepare`. This is important since you should not be duplicating logic from `validate` to `prepare`, you need a way of passing your working from the former into the latter. This is it. - This trait takes a `Call` type parameter. `Call` is the runtime call type which used to be an associated type; you can just move it to become a type parameter for your trait impl. - There's no `AccountId` associated type any more. Just remove it. Regarding `validate`: - You get three new parameters in `validate`; all can be ignored when migrating from `SignedExtension`. - `validate` returns a tuple on success; the second item in the tuple is the new ticket type `Self::Val` which gets passed in to `prepare`. If you use any information extracted during `validate` (off-chain and on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can pass it through with this. For the tuple's last item, just return the `origin` argument. Regarding `prepare`: - This is renamed from `pre_dispatch`, but there is one change: - FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM `validate`!! - (This is different to `SignedExtension` which was required to run the same checks in `pre_dispatch` as in `validate`.) Regarding `post_dispatch`: - Since there are no unsigned transactions handled by `TransactionExtension`, `Pre` is always defined, so the first parameter is `Self::Pre` rather than `Option<Self::Pre>`. If you make use of `SignedExtension::validate_unsigned` or `SignedExtension::pre_dispatch_unsigned`, then: - Just use the regular versions of these functions instead. - Have your logic execute in the case that the `origin` is `None`. - Ensure your transaction creation logic creates a General Transaction rather than a Bare Transaction; this means having to include all `TransactionExtension`s' data. - `ValidateUnsigned` can still be used (for now) if you need to be able to construct transactions which contain none of the extension data, however these will be phased out in stage 2 of the Transactions Horizon, so you should consider moving to an extension-centric design. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Branislav Kontur <bkontur@gmail.com>

Create TransactionExtension and replace SignedExtension with it.

c220976

gavofyork requested review from a team November 11, 2023 15:38

gavofyork marked this pull request as draft November 11, 2023 15:39

gavofyork added the T1-FRAME This PR/Issue is related to core FRAME, the framework. label Nov 11, 2023

gupnik and others added 2 commits November 11, 2023 21:33

Fixes macro issue

cc315b8

Fixes and extract AdditionalSigned.

4b4e31f

gavofyork added 4 commits November 17, 2023 15:17

Final bits

c951a14

Merge remote-tracking branch 'origin/master' into gav-tx-ext

9c9bf85

Merge branch 'gav-tx-ext' of github.com:paritytech/polkadot-sdk into …

bf93801

…gav-tx-ext

Merge remote-tracking branch 'origin/master' into gav-tx-ext

0410dec

gavofyork closed this Nov 20, 2023

gavofyork reopened this Nov 20, 2023

gavofyork added 2 commits November 20, 2023 16:36

It builds.

549c285

Builds with all features

dab1499

gavofyork mentioned this pull request Nov 20, 2023

Extrinsic Horizon #2415

Open

gavofyork added 12 commits November 20, 2023 23:25

Migrate some Extensions.

399962f

Merge branch 'master' into gav-tx-ext

52c2d83

Merge branch 'gav-tx-ext' of github.com:paritytech/polkadot-sdk into …

16c20c4

…gav-tx-ext

AdditionalSigned integrated as implicit

acd1be4

Make Verifysignature TxExt work,

25353dd

TransactionExtension has Call as a type param

9aa7028

Cleanups

3d89109

Remove unneeded code

d229571

Impl TE for DisallowSigned

149e118

Remove noop test

2e80e78

Move over CheckIfFeeless

0bce1be

Port CheckSpecVersion

a1e81c1

kianenigma reviewed Mar 5, 2024

View reviewed changes

This was referenced Mar 6, 2024

Revisit reference docs for TransactionExtension #3591

Open

Update UncheckedExtrinsic documentation #3592

Open

Add example usage in docs of TransactionExtension use with general, new-school transactions #3593

Open

ascjones mentioned this pull request Mar 7, 2024

Merkleized metadata polkadot-fellows/RFCs#78

Merged

bkchr reviewed Mar 7, 2024

View reviewed changes

georgepisaltu mentioned this pull request Mar 8, 2024

TransactionExtension follow up refactoring and fixes #3623

Closed

jsdw mentioned this pull request Mar 8, 2024

TransactionExtension: Update naming paritytech/subxt#1468

Open

bkchr reviewed Mar 9, 2024

View reviewed changes

georgepisaltu added a commit to georgepisaltu/polkadot-sdk that referenced this pull request Mar 12, 2024

Revert "FRAME: Create TransactionExtension as a replacement for `Si…

d9781da

…gnedExtension` (paritytech#2280)" This reverts commit fd5f929.

georgepisaltu mentioned this pull request Mar 12, 2024

Revert "FRAME: Create TransactionExtension as a replacement for SignedExtension (#2280)" #3665

Merged

Niederb mentioned this pull request Mar 13, 2024

Rename node-template-runtime dependency scs/substrate-api-client#751

Merged

georgepisaltu added a commit to georgepisaltu/polkadot-sdk that referenced this pull request Mar 13, 2024

Revert "Revert "FRAME: Create TransactionExtension as a replacement…

53c0379

… for `SignedExtension` (paritytech#2280)" (paritytech#3665)" This reverts commit bbd51ce.

georgepisaltu mentioned this pull request Mar 13, 2024

FRAME: Reintroduce TransactionExtension as a replacement for SignedExtension #3685

Merged

muharem mentioned this pull request Mar 15, 2024

FRAME: Meta Transaction (extension based version) #3712

Closed

nazar-pc mentioned this pull request Apr 5, 2024

Update Substrate to 1.8.0 autonomys/subspace#2667

Merged

1 task

Niederb mentioned this pull request Apr 29, 2024

Better wasm support scs/substrate-api-client#766

Merged

		/// NOTE: In the future, once we remove `ValidateUnsigned`, this will only serve Inherent
		/// extrinsics and thus can be renamed to `Inherent`.

		/// validity against current state. It should perform all checks that determine a valid
		/// transaction, that can pay for its execution and quickly eliminate ones that are stale or

FRAME: Create TransactionExtension as a replacement for SignedExtension #2280

FRAME: Create TransactionExtension as a replacement for SignedExtension #2280

Conversation

gavofyork commented Nov 11, 2023 • edited by georgepisaltu Loading

Code Migration

NOW: Getting it to build

SOON: Migrating to TransactionExtension

TransactionExtensionBase

TransactionExtension

TODO

xlc commented Nov 16, 2023 • edited Loading

gavofyork commented Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

kianenigma Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkchr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgepisaltu Mar 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgepisaltu commented Mar 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` #2280

FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` #2280

gavofyork commented Nov 11, 2023 •

edited by georgepisaltu

Loading

SOON: Migrating to `TransactionExtension`

`TransactionExtensionBase`

`TransactionExtension`

xlc commented Nov 16, 2023 •

edited

Loading

gavofyork commented Nov 20, 2023 •

edited

Loading

kianenigma Mar 5, 2024 •

edited

Loading

georgepisaltu Mar 8, 2024 •

edited

Loading