Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XcmDryRunApi - Dry-running extrinsics to get their XCM effects #3872

Merged
merged 74 commits into from
May 8, 2024

Conversation

franciscoaguirre
Copy link
Contributor

@franciscoaguirre franciscoaguirre commented Mar 27, 2024

(Note: followed-up by #4621 to use call instead of extrinsic)

Context

Estimating fees for XCM execution and sending has been an area with bad UX.
The addition of the XcmPaymentApi exposed the necessary components to be able to estimate XCM fees correctly, however, that was not the full story.
The XcmPaymentApi works for estimating fees only if you know the specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics, they don't necessarily know the XCM program that's executed by them.

Main addition

A new runtime API is introduced, the XcmDryRunApi, that given an extrinsic, or an XCM program, returns its effects:

  • Execution result
  • Local XCM (in the case of an extrinsic)
  • Forwarded XCMs
  • List of events

This API can be used on its own for dry-running purposes, for double-checking or testing, but it mainly shines when used in conjunction with the XcmPaymentApi.
UIs can use these two APIs to estimate transfers.

How it works

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under xcm-fee-payment-runtime-api.
There's a real-world test using Westend and AssetHubWestend under cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs.
Added both a test for a simple teleport between chains and a reserve transfer asset between two parachains going through a reserve.

The steps to follow:

  • Use XcmDryRunApi::dry_run_extrinsic to get local XCM program and forwarded messages
  • For each forwarded message
    • Use XcmPaymentApi::query_delivery_fee LOCALLY to get the delivery fees
    • Use XcmPaymentApi::query_xcm_weight ON THE DESTINATION to get the remote execution weight
    • (optional) Use XcmPaymentApi::query_acceptable_payment_assets ON THE DESTINATION to know on which assets the execution fees can be paid
    • Use XcmPaymentApi::query_weight_to_asset_fee ON THE DESTINATION to convert weight to the actual remote execution fees
    • Use XcmDryRunApi::dry_run_xcm ON THE DESTINATION to know if a new message will be forwarded, if so, continue

Dear reviewer

The changes in this PR are grouped as follows, and in order of importance:

  • Addition of new runtime API
    • Definition, mock and simple tests: polkadot/xcm/xcm-fee-payment-runtime-api/*
    • Implemented on Westend, Asset Hub Westend and Penpal, will implement on every runtime in a following PR
  • Addition of a new config item to the XCM executor for recording xcms about to be executed
    • Definition: polkadot/xcm/xcm-executor/*
    • Implementation: polkadot/xcm/pallet-xcm/*
    • had to update all runtime xcm_config.rs files with type XcmRecorder = XcmPallet;
  • Addition of a new trait for inspecting the messages in queues
    • Definition: polkadot/xcm/xcm-builder/src/routing.rs
    • Implemented it on all routers:
      • ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
      • ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on implementation in cumulus/pallets/parachain-system/src/lib.rs)
      • XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
      • Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
  • More complicated and useful tests:
    • cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

Next steps

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have the new API.
UIs can test on these runtimes to create better experiences around cross-chain operations.

Next:

  • Add XcmDryRunApi to all system parachains
  • Integrate xcm fee estimation in all emulated tests
  • Get this on the fellowship runtimes

@franciscoaguirre franciscoaguirre added the T6-XCM This PR/Issue is related to XCM. label Mar 27, 2024
@franciscoaguirre franciscoaguirre self-assigned this Mar 27, 2024
Copy link
Contributor

@xlc xlc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How will this play with orml-xtokens?

orml-xtokens provided calls that accepts CurrencyId instead of Location and it internally figure out the right Location to use.

one way to achieve similar result without too coupling is dry run the extrinsic whatever it is, and read out the ump / xcmp queue and we know exactly the XCM that will be sent

polkadot/xcm/xcm-fee-payment-runtime-api/src/transfers.rs Outdated Show resolved Hide resolved
@franciscoaguirre
Copy link
Contributor Author

The idea was that both pallet-xcm and orml-tokens could provide functions for returning the messages they send.
Then each runtime would implement this API, calling the underlying function from the pallet they use for XCM transfers.

Dry-running the extrinsic definitely sounds like a better option.
In the case of reserve asset transfers, you'd also need to dry-run the XCMs to know what other messages they send.
I think it's better for UIs since they only have access to the extrinsic.

@franciscoaguirre
Copy link
Contributor Author

We can know what messages are put in the queue, but we can't know the message that is executed locally by the extrinsic.

@mrshiposha
Copy link
Contributor

mrshiposha commented Mar 28, 2024

It seems the pallet developers should provide some support for their extrinsics for such an API anyway.
However, it may be possible to simplify this API for clients...

For instance,

pub trait XcmProgramsApi {
    fn query_xcm_programs(xt: <Block as BlockT>::Extrinsic) -> Result<Vec<(Location, Xcm<()>)>, /* ErrT */>;
}

And then, when implementing this API for a chain:

impl XcmProgramsApi<Block> for Runtime {
    fn query_xcm_programs(xt: <Block as sp_api::BlockT>::Extrinsic) -> Result<Vec<(Location, Xcm<()>)>, /* ErrT */> {
        match xt.0.function {
            RuntimeCall::PalletXcm(pallet_xcm::Call::transfer_assets { /* params */ }) => { todo!("return the programs") },
            RuntimeCall::XTokens(pallet_xtokens::Call::transfer { /* params */ }) => { todo!("return the programs") },
            _ => Err(/* whatever */),
        }
    }
}

So, the client could provide an extrinsic to this API and retrieve the XCM programs. The chain could determine the programs based on the provided extrinsic by itself.

This way, we can return the program that will be executed by the extrinsic locally and dry-run the extrinsic as well to return the programs from the ump / xcmp queue. Also, to simplify the API implementation, the pallets could provide public functions for handling the todo!("return the programs")

Note: an extrinsic should be signed, but the client can use the signFake from PolkadotJS to analyze the extrinsic using the API first. It's similar to how the paymentInfo is implemented for the SubmittableExtrinsic: https://github.com/polkadot-js/api/blob/7874a8e7234219e47abca6a37ae0558aa4731751/packages/api/src/submittable/createClass.ts#L167-L169.

But maybe I'm missing something. I didn't try to implement this :)
Just sharing thoughts.

@mrshiposha
Copy link
Contributor

mrshiposha commented Mar 28, 2024

Hmm, I think that dry-running an extrinsic to learn what messages will be sent where is not enough. It covers the case for transferring assets from Chain A to Chain B. However, it doesn't cover this case: Chain A --> Reserve --> Chain B.

However, if we have the XCM program that the extrinsic wants to execute locally on the first chain, we can easily analyze the program to determine all the locations and messages. We just need to analyze the initial program and look at the InitiateReserveWithdraw and other similar instructions to recursively extract all the locations and messages. Such an analysis of an XCM program can be done in a common function that all chains can reuse when implementing the API from the comment above

@acatangiu
Copy link
Contributor

acatangiu commented Mar 28, 2024

I was discussing something similar with @franciscoaguirre, exploring something similar:

pub struct XcmDryRunEffects {
    locally_executed_xcm: VersionedXcm<()>,
    forwarded_xcms: Vec<(VersionedLocation, VersionedXcm<()>)>,
    execution_result: DispatchResult,
    // events ?
}

pub trait DryRunXcmApi {
    /// query side effects for executing extrinsic
    fn query_xcm_for_extrinsic(
        xt: <Block as BlockT>::Extrinsic,
    ) -> Result<XcmDryRunEffects, /* ErrT */>;

    /// query side effects for executing (incoming) XCM
    fn query_execute_xcm(
        xcm: Xcm<()>,
        using_origin: Location,
    ) -> Result<XcmDryRunEffects, /* ErrT */>;
}

We want to:

  • find out locally paid fees as part of locally executing an XCM either coming from some other location or coming from some locally run extrinsic
  • for same as above, find out any outgoing XCMs and their destinations as a result
  • be able to repeat the same operations on said destinations for these outgoing XCMs.

Would this be enough to cover all needs?

@acatangiu acatangiu mentioned this pull request Mar 28, 2024
@franciscoaguirre
Copy link
Contributor Author

franciscoaguirre commented Mar 28, 2024

Very draft still but I'm working on the dry-running idea

@franciscoaguirre franciscoaguirre changed the title Runtime API for getting transfer XCMs for usage with new XcmPaymentApi XcmDryRunApi - Dry-running extrinsics to get their XCM effects Mar 29, 2024
@franciscoaguirre
Copy link
Contributor Author

I have now both the dry_run_extrinsic and dry_run_xcm calls. The only problem I see with the extrinsic one is we should have a way to get the actual xcm program that's being sent to the executor.
Other than that, the API looks much better.
It would need something to solve that situation and then some code cleanup.

@franciscoaguirre
Copy link
Contributor Author

franciscoaguirre commented Apr 9, 2024

@xlc I managed to simplify the way of dry-running an extrinsic and getting both the local message and the remote messages. I think this'll make it easy to be used with xtokens. Can you take a look and tell me what you think?

Also, Here I'm dry-running both an extrinsic and an XCM, which look like #230 and #486. I could move the extrinsic one to a more generic runtime API. What are your thoughts on trying to tackle these issues here?

Copy link
Contributor

@acatangiu acatangiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great! 🚀

Only have impl details comments, see below

polkadot/xcm/pallet-xcm/src/lib.rs Outdated Show resolved Hide resolved
polkadot/xcm/pallet-xcm/src/lib.rs Outdated Show resolved Hide resolved
polkadot/xcm/pallet-xcm/src/lib.rs Outdated Show resolved Hide resolved
polkadot/xcm/pallet-xcm/src/lib.rs Outdated Show resolved Hide resolved
polkadot/xcm/xcm-executor/src/config.rs Outdated Show resolved Hide resolved
polkadot/xcm/xcm-fee-payment-runtime-api/src/dry_run.rs Outdated Show resolved Hide resolved
polkadot/xcm/xcm-fee-payment-runtime-api/tests/mock.rs Outdated Show resolved Hide resolved
polkadot/xcm/xcm-fee-payment-runtime-api/tests/mock.rs Outdated Show resolved Hide resolved
ntn-x2 pushed a commit to KILTprotocol/polkadot-sdk that referenced this pull request Jul 12, 2024
…tytech#4621)

Follow-up to the new `XcmDryRunApi` runtime API introduced in
paritytech#3872.

Taking an extrinsic means the frontend has to sign first to dry-run and
once again to submit.
This is bad UX which is solved by taking an `origin` and a `call`.
This also has the benefit of being able to dry-run as any account, since
it needs no signature.

This is a breaking change since I changed `dry_run_extrinsic` to
`dry_run_call`, however, this API is still only on testnets.
The crates are bumped accordingly.

As a part of this PR, I changed the name of the API from `XcmDryRunApi`
to just `DryRunApi`, since it can be used for general dry-running :)

Step towards paritytech#690.

Example of calling the API with PAPI, not the best code, just testing :)

```ts
// We just build a call, the arguments make it look very big though.
const call = localApi.tx.XcmPallet.transfer_assets({
  dest: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.Parachain(1000)) }),
  beneficiary: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.AccountId32({ network: undefined, id: Binary.fromBytes(encodeAccount(account.address)) })) }),
  weight_limit: XcmV3WeightLimit.Unlimited(),
  assets: XcmVersionedAssets.V4([{
    id: { parents: 0, interior: XcmV4Junctions.Here() },
    fun: XcmV3MultiassetFungibility.Fungible(1_000_000_000_000n) }
  ]),
  fee_asset_item: 0,
});
// We call the API passing in a signed origin 
const result = await localApi.apis.XcmDryRunApi.dry_run_call(
  WestendRuntimeOriginCaller.system(DispatchRawOrigin.Signed(account.address)),
  call.decodedCall
);
if (result.success && result.value.execution_result.success) {
  // We find the forwarded XCM we want. The first one going to AssetHub in this case.
  const xcmsToAssetHub = result.value.forwarded_xcms.find(([location, _]) => (
    location.type === "V4" &&
      location.value.parents === 0 &&
      location.value.interior.type === "X1"
      && location.value.interior.value.type === "Parachain"
      && location.value.interior.value.value === 1000
  ))!;

  // We can even find the delivery fees for that forwarded XCM.
  const deliveryFeesQuery = await localApi.apis.XcmPaymentApi.query_delivery_fees(xcmsToAssetHub[0], xcmsToAssetHub[1][0]);

  if (deliveryFeesQuery.success) {
    const amount = deliveryFeesQuery.value.type === "V4" && deliveryFeesQuery.value.value[0].fun.type === "Fungible" && deliveryFeesQuery.value.value[0].fun.value.valueOf() || 0n;
    // We store them in state somewhere.
    setDeliveryFees(formatAmount(BigInt(amount)));
  }
}
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
ntn-x2 pushed a commit to KILTprotocol/polkadot-sdk that referenced this pull request Jul 13, 2024
…tytech#4621)

Follow-up to the new `XcmDryRunApi` runtime API introduced in
paritytech#3872.

Taking an extrinsic means the frontend has to sign first to dry-run and
once again to submit.
This is bad UX which is solved by taking an `origin` and a `call`.
This also has the benefit of being able to dry-run as any account, since
it needs no signature.

This is a breaking change since I changed `dry_run_extrinsic` to
`dry_run_call`, however, this API is still only on testnets.
The crates are bumped accordingly.

As a part of this PR, I changed the name of the API from `XcmDryRunApi`
to just `DryRunApi`, since it can be used for general dry-running :)

Step towards paritytech#690.

Example of calling the API with PAPI, not the best code, just testing :)

```ts
// We just build a call, the arguments make it look very big though.
const call = localApi.tx.XcmPallet.transfer_assets({
  dest: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.Parachain(1000)) }),
  beneficiary: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.AccountId32({ network: undefined, id: Binary.fromBytes(encodeAccount(account.address)) })) }),
  weight_limit: XcmV3WeightLimit.Unlimited(),
  assets: XcmVersionedAssets.V4([{
    id: { parents: 0, interior: XcmV4Junctions.Here() },
    fun: XcmV3MultiassetFungibility.Fungible(1_000_000_000_000n) }
  ]),
  fee_asset_item: 0,
});
// We call the API passing in a signed origin 
const result = await localApi.apis.XcmDryRunApi.dry_run_call(
  WestendRuntimeOriginCaller.system(DispatchRawOrigin.Signed(account.address)),
  call.decodedCall
);
if (result.success && result.value.execution_result.success) {
  // We find the forwarded XCM we want. The first one going to AssetHub in this case.
  const xcmsToAssetHub = result.value.forwarded_xcms.find(([location, _]) => (
    location.type === "V4" &&
      location.value.parents === 0 &&
      location.value.interior.type === "X1"
      && location.value.interior.value.type === "Parachain"
      && location.value.interior.value.value === 1000
  ))!;

  // We can even find the delivery fees for that forwarded XCM.
  const deliveryFeesQuery = await localApi.apis.XcmPaymentApi.query_delivery_fees(xcmsToAssetHub[0], xcmsToAssetHub[1][0]);

  if (deliveryFeesQuery.success) {
    const amount = deliveryFeesQuery.value.type === "V4" && deliveryFeesQuery.value.value[0].fun.type === "Fungible" && deliveryFeesQuery.value.value[0].fun.value.valueOf() || 0n;
    // We store them in state somewhere.
    setDeliveryFees(formatAmount(BigInt(amount)));
  }
}
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
fgamundi pushed a commit to moondance-labs/polkadot-sdk that referenced this pull request Jul 17, 2024
…ytech#3872)

Estimating fees for XCM execution and sending has been an area with bad
UX.
The addition of the
[XcmPaymentApi](paritytech#3607)
exposed the necessary components to be able to estimate XCM fees
correctly, however, that was not the full story.
The `XcmPaymentApi` works for estimating fees only if you know the
specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics,
they don't necessarily know the XCM program that's executed by them.

A new runtime API is introduced, the `XcmDryRunApi`, that given an
extrinsic, or an XCM program, returns its effects:
- Execution result
- Local XCM (in the case of an extrinsic)
- Forwarded XCMs
- List of events

This API can be used on its own for dry-running purposes, for
double-checking or testing, but it mainly shines when used in
conjunction with the `XcmPaymentApi`.
UIs can use these two APIs to estimate transfers.

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under
`xcm-fee-payment-runtime-api`.
There's a real-world test using Westend and AssetHubWestend under
`cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs`.
Added both a test for a simple teleport between chains and a reserve
transfer asset between two parachains going through a reserve.

The steps to follow:
- Use `XcmDryRunApi::dry_run_extrinsic` to get local XCM program and
forwarded messages
- For each forwarded message
- Use `XcmPaymentApi::query_delivery_fee` LOCALLY to get the delivery
fees
- Use `XcmPaymentApi::query_xcm_weight` ON THE DESTINATION to get the
remote execution weight
- (optional) Use `XcmPaymentApi::query_acceptable_payment_assets` ON THE
DESTINATION to know on which assets the execution fees can be paid
- Use `XcmPaymentApi::query_weight_to_asset_fee` ON THE DESTINATION to
convert weight to the actual remote execution fees
- Use `XcmDryRunApi::dry_run_xcm` ON THE DESTINATION to know if a new
message will be forwarded, if so, continue

The changes in this PR are grouped as follows, and in order of
importance:
- Addition of new runtime API
- Definition, mock and simple tests:
polkadot/xcm/xcm-fee-payment-runtime-api/*
- Implemented on Westend, Asset Hub Westend and Penpal, will implement
on every runtime in a following PR
- Addition of a new config item to the XCM executor for recording xcms
about to be executed
  - Definition: polkadot/xcm/xcm-executor/*
  - Implementation: polkadot/xcm/pallet-xcm/*
- had to update all runtime xcm_config.rs files with `type XcmRecorder =
XcmPallet;`
- Addition of a new trait for inspecting the messages in queues
  - Definition: polkadot/xcm/xcm-builder/src/routing.rs
  - Implemented it on all routers:
    - ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
- ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on
implementation in cumulus/pallets/parachain-system/src/lib.rs)
    - XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
    - Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
- More complicated and useful tests:
-
cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have
the new API.
UIs can test on these runtimes to create better experiences around
cross-chain operations.

Next:
- Add XcmDryRunApi to all system parachains
- Integrate xcm fee estimation in all emulated tests
- Get this on the fellowship runtimes

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
fgamundi pushed a commit to moondance-labs/polkadot-sdk that referenced this pull request Jul 17, 2024
…tytech#4621)

Follow-up to the new `XcmDryRunApi` runtime API introduced in
paritytech#3872.

Taking an extrinsic means the frontend has to sign first to dry-run and
once again to submit.
This is bad UX which is solved by taking an `origin` and a `call`.
This also has the benefit of being able to dry-run as any account, since
it needs no signature.

This is a breaking change since I changed `dry_run_extrinsic` to
`dry_run_call`, however, this API is still only on testnets.
The crates are bumped accordingly.

As a part of this PR, I changed the name of the API from `XcmDryRunApi`
to just `DryRunApi`, since it can be used for general dry-running :)

Step towards paritytech#690.

Example of calling the API with PAPI, not the best code, just testing :)

```ts
// We just build a call, the arguments make it look very big though.
const call = localApi.tx.XcmPallet.transfer_assets({
  dest: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.Parachain(1000)) }),
  beneficiary: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.AccountId32({ network: undefined, id: Binary.fromBytes(encodeAccount(account.address)) })) }),
  weight_limit: XcmV3WeightLimit.Unlimited(),
  assets: XcmVersionedAssets.V4([{
    id: { parents: 0, interior: XcmV4Junctions.Here() },
    fun: XcmV3MultiassetFungibility.Fungible(1_000_000_000_000n) }
  ]),
  fee_asset_item: 0,
});
// We call the API passing in a signed origin
const result = await localApi.apis.XcmDryRunApi.dry_run_call(
  WestendRuntimeOriginCaller.system(DispatchRawOrigin.Signed(account.address)),
  call.decodedCall
);
if (result.success && result.value.execution_result.success) {
  // We find the forwarded XCM we want. The first one going to AssetHub in this case.
  const xcmsToAssetHub = result.value.forwarded_xcms.find(([location, _]) => (
    location.type === "V4" &&
      location.value.parents === 0 &&
      location.value.interior.type === "X1"
      && location.value.interior.value.type === "Parachain"
      && location.value.interior.value.value === 1000
  ))!;

  // We can even find the delivery fees for that forwarded XCM.
  const deliveryFeesQuery = await localApi.apis.XcmPaymentApi.query_delivery_fees(xcmsToAssetHub[0], xcmsToAssetHub[1][0]);

  if (deliveryFeesQuery.success) {
    const amount = deliveryFeesQuery.value.type === "V4" && deliveryFeesQuery.value.value[0].fun.type === "Fungible" && deliveryFeesQuery.value.value[0].fun.value.valueOf() || 0n;
    // We store them in state somewhere.
    setDeliveryFees(formatAmount(BigInt(amount)));
  }
}
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
# Conflicts:
#	cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs
#	cumulus/parachains/runtimes/assets/asset-hub-rococo/src/lib.rs
#	cumulus/parachains/runtimes/assets/asset-hub-westend/src/lib.rs
#	cumulus/parachains/runtimes/testing/penpal/src/lib.rs
#	polkadot/runtime/rococo/src/lib.rs
#	polkadot/runtime/westend/src/lib.rs
#	polkadot/xcm/xcm-fee-payment-runtime-api/tests/fee_estimation.rs
#	polkadot/xcm/xcm-fee-payment-runtime-api/tests/mock.rs
fgamundi pushed a commit to moondance-labs/polkadot-sdk that referenced this pull request Jul 17, 2024
…ytech#3872)

Estimating fees for XCM execution and sending has been an area with bad
UX.
The addition of the
[XcmPaymentApi](paritytech#3607)
exposed the necessary components to be able to estimate XCM fees
correctly, however, that was not the full story.
The `XcmPaymentApi` works for estimating fees only if you know the
specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics,
they don't necessarily know the XCM program that's executed by them.

A new runtime API is introduced, the `XcmDryRunApi`, that given an
extrinsic, or an XCM program, returns its effects:
- Execution result
- Local XCM (in the case of an extrinsic)
- Forwarded XCMs
- List of events

This API can be used on its own for dry-running purposes, for
double-checking or testing, but it mainly shines when used in
conjunction with the `XcmPaymentApi`.
UIs can use these two APIs to estimate transfers.

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under
`xcm-fee-payment-runtime-api`.
There's a real-world test using Westend and AssetHubWestend under
`cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs`.
Added both a test for a simple teleport between chains and a reserve
transfer asset between two parachains going through a reserve.

The steps to follow:
- Use `XcmDryRunApi::dry_run_extrinsic` to get local XCM program and
forwarded messages
- For each forwarded message
- Use `XcmPaymentApi::query_delivery_fee` LOCALLY to get the delivery
fees
- Use `XcmPaymentApi::query_xcm_weight` ON THE DESTINATION to get the
remote execution weight
- (optional) Use `XcmPaymentApi::query_acceptable_payment_assets` ON THE
DESTINATION to know on which assets the execution fees can be paid
- Use `XcmPaymentApi::query_weight_to_asset_fee` ON THE DESTINATION to
convert weight to the actual remote execution fees
- Use `XcmDryRunApi::dry_run_xcm` ON THE DESTINATION to know if a new
message will be forwarded, if so, continue

The changes in this PR are grouped as follows, and in order of
importance:
- Addition of new runtime API
- Definition, mock and simple tests:
polkadot/xcm/xcm-fee-payment-runtime-api/*
- Implemented on Westend, Asset Hub Westend and Penpal, will implement
on every runtime in a following PR
- Addition of a new config item to the XCM executor for recording xcms
about to be executed
  - Definition: polkadot/xcm/xcm-executor/*
  - Implementation: polkadot/xcm/pallet-xcm/*
- had to update all runtime xcm_config.rs files with `type XcmRecorder =
XcmPallet;`
- Addition of a new trait for inspecting the messages in queues
  - Definition: polkadot/xcm/xcm-builder/src/routing.rs
  - Implemented it on all routers:
    - ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
- ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on
implementation in cumulus/pallets/parachain-system/src/lib.rs)
    - XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
    - Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
- More complicated and useful tests:
-
cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have
the new API.
UIs can test on these runtimes to create better experiences around
cross-chain operations.

Next:
- Add XcmDryRunApi to all system parachains
- Integrate xcm fee estimation in all emulated tests
- Get this on the fellowship runtimes

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
fgamundi pushed a commit to moondance-labs/polkadot-sdk that referenced this pull request Jul 17, 2024
…tytech#4621)

Follow-up to the new `XcmDryRunApi` runtime API introduced in
paritytech#3872.

Taking an extrinsic means the frontend has to sign first to dry-run and
once again to submit.
This is bad UX which is solved by taking an `origin` and a `call`.
This also has the benefit of being able to dry-run as any account, since
it needs no signature.

This is a breaking change since I changed `dry_run_extrinsic` to
`dry_run_call`, however, this API is still only on testnets.
The crates are bumped accordingly.

As a part of this PR, I changed the name of the API from `XcmDryRunApi`
to just `DryRunApi`, since it can be used for general dry-running :)

Step towards paritytech#690.

Example of calling the API with PAPI, not the best code, just testing :)

```ts
// We just build a call, the arguments make it look very big though.
const call = localApi.tx.XcmPallet.transfer_assets({
  dest: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.Parachain(1000)) }),
  beneficiary: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.AccountId32({ network: undefined, id: Binary.fromBytes(encodeAccount(account.address)) })) }),
  weight_limit: XcmV3WeightLimit.Unlimited(),
  assets: XcmVersionedAssets.V4([{
    id: { parents: 0, interior: XcmV4Junctions.Here() },
    fun: XcmV3MultiassetFungibility.Fungible(1_000_000_000_000n) }
  ]),
  fee_asset_item: 0,
});
// We call the API passing in a signed origin
const result = await localApi.apis.XcmDryRunApi.dry_run_call(
  WestendRuntimeOriginCaller.system(DispatchRawOrigin.Signed(account.address)),
  call.decodedCall
);
if (result.success && result.value.execution_result.success) {
  // We find the forwarded XCM we want. The first one going to AssetHub in this case.
  const xcmsToAssetHub = result.value.forwarded_xcms.find(([location, _]) => (
    location.type === "V4" &&
      location.value.parents === 0 &&
      location.value.interior.type === "X1"
      && location.value.interior.value.type === "Parachain"
      && location.value.interior.value.value === 1000
  ))!;

  // We can even find the delivery fees for that forwarded XCM.
  const deliveryFeesQuery = await localApi.apis.XcmPaymentApi.query_delivery_fees(xcmsToAssetHub[0], xcmsToAssetHub[1][0]);

  if (deliveryFeesQuery.success) {
    const amount = deliveryFeesQuery.value.type === "V4" && deliveryFeesQuery.value.value[0].fun.type === "Fungible" && deliveryFeesQuery.value.value[0].fun.value.valueOf() || 0n;
    // We store them in state somewhere.
    setDeliveryFees(formatAmount(BigInt(amount)));
  }
}
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
# Conflicts:
#	cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs
#	cumulus/parachains/runtimes/assets/asset-hub-rococo/src/lib.rs
#	cumulus/parachains/runtimes/assets/asset-hub-westend/src/lib.rs
#	cumulus/parachains/runtimes/testing/penpal/src/lib.rs
#	polkadot/runtime/rococo/src/lib.rs
#	polkadot/runtime/westend/src/lib.rs
#	polkadot/xcm/xcm-fee-payment-runtime-api/tests/fee_estimation.rs
#	polkadot/xcm/xcm-fee-payment-runtime-api/tests/mock.rs
fellowship-merge-bot bot pushed a commit to polkadot-fellows/runtimes that referenced this pull request Jul 22, 2024
Adds the paritytech/polkadot-sdk#3607 and
paritytech/polkadot-sdk#3872 to all runtimes.
These APIs work together to allow dry running and estimating execution
and delivery fees for XCMs.

This PR doesn't allow querying the price in assets different than the
relay token for asset hubs. Can be done in a following PR.
Also tracking #388
and #389 for future
improvemens.

Old PR was #359, this
one targets main already updated to Polkadot SDK 1.13.

## Dear reviewer

Although there are a lot of changes, the main one is the addition of the
two aforementioned APIs to all relays and system parachains. The rest of
the files are tests, which all have the same format.
@Polkadot-Forum
Copy link

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-technical-summit-xcm/9384/1

girazoki pushed a commit to moondance-labs/polkadot-sdk that referenced this pull request Aug 1, 2024
…ytech#3872)

Estimating fees for XCM execution and sending has been an area with bad
UX.
The addition of the
[XcmPaymentApi](paritytech#3607)
exposed the necessary components to be able to estimate XCM fees
correctly, however, that was not the full story.
The `XcmPaymentApi` works for estimating fees only if you know the
specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics,
they don't necessarily know the XCM program that's executed by them.

A new runtime API is introduced, the `XcmDryRunApi`, that given an
extrinsic, or an XCM program, returns its effects:
- Execution result
- Local XCM (in the case of an extrinsic)
- Forwarded XCMs
- List of events

This API can be used on its own for dry-running purposes, for
double-checking or testing, but it mainly shines when used in
conjunction with the `XcmPaymentApi`.
UIs can use these two APIs to estimate transfers.

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under
`xcm-fee-payment-runtime-api`.
There's a real-world test using Westend and AssetHubWestend under
`cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs`.
Added both a test for a simple teleport between chains and a reserve
transfer asset between two parachains going through a reserve.

The steps to follow:
- Use `XcmDryRunApi::dry_run_extrinsic` to get local XCM program and
forwarded messages
- For each forwarded message
- Use `XcmPaymentApi::query_delivery_fee` LOCALLY to get the delivery
fees
- Use `XcmPaymentApi::query_xcm_weight` ON THE DESTINATION to get the
remote execution weight
- (optional) Use `XcmPaymentApi::query_acceptable_payment_assets` ON THE
DESTINATION to know on which assets the execution fees can be paid
- Use `XcmPaymentApi::query_weight_to_asset_fee` ON THE DESTINATION to
convert weight to the actual remote execution fees
- Use `XcmDryRunApi::dry_run_xcm` ON THE DESTINATION to know if a new
message will be forwarded, if so, continue

The changes in this PR are grouped as follows, and in order of
importance:
- Addition of new runtime API
- Definition, mock and simple tests:
polkadot/xcm/xcm-fee-payment-runtime-api/*
- Implemented on Westend, Asset Hub Westend and Penpal, will implement
on every runtime in a following PR
- Addition of a new config item to the XCM executor for recording xcms
about to be executed
  - Definition: polkadot/xcm/xcm-executor/*
  - Implementation: polkadot/xcm/pallet-xcm/*
- had to update all runtime xcm_config.rs files with `type XcmRecorder =
XcmPallet;`
- Addition of a new trait for inspecting the messages in queues
  - Definition: polkadot/xcm/xcm-builder/src/routing.rs
  - Implemented it on all routers:
    - ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
- ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on
implementation in cumulus/pallets/parachain-system/src/lib.rs)
    - XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
    - Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
- More complicated and useful tests:
-
cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have
the new API.
UIs can test on these runtimes to create better experiences around
cross-chain operations.

Next:
- Add XcmDryRunApi to all system parachains
- Integrate xcm fee estimation in all emulated tests
- Get this on the fellowship runtimes

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
girazoki pushed a commit to moondance-labs/polkadot-sdk that referenced this pull request Aug 1, 2024
…tytech#4621)

Follow-up to the new `XcmDryRunApi` runtime API introduced in
paritytech#3872.

Taking an extrinsic means the frontend has to sign first to dry-run and
once again to submit.
This is bad UX which is solved by taking an `origin` and a `call`.
This also has the benefit of being able to dry-run as any account, since
it needs no signature.

This is a breaking change since I changed `dry_run_extrinsic` to
`dry_run_call`, however, this API is still only on testnets.
The crates are bumped accordingly.

As a part of this PR, I changed the name of the API from `XcmDryRunApi`
to just `DryRunApi`, since it can be used for general dry-running :)

Step towards paritytech#690.

Example of calling the API with PAPI, not the best code, just testing :)

```ts
// We just build a call, the arguments make it look very big though.
const call = localApi.tx.XcmPallet.transfer_assets({
  dest: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.Parachain(1000)) }),
  beneficiary: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.AccountId32({ network: undefined, id: Binary.fromBytes(encodeAccount(account.address)) })) }),
  weight_limit: XcmV3WeightLimit.Unlimited(),
  assets: XcmVersionedAssets.V4([{
    id: { parents: 0, interior: XcmV4Junctions.Here() },
    fun: XcmV3MultiassetFungibility.Fungible(1_000_000_000_000n) }
  ]),
  fee_asset_item: 0,
});
// We call the API passing in a signed origin
const result = await localApi.apis.XcmDryRunApi.dry_run_call(
  WestendRuntimeOriginCaller.system(DispatchRawOrigin.Signed(account.address)),
  call.decodedCall
);
if (result.success && result.value.execution_result.success) {
  // We find the forwarded XCM we want. The first one going to AssetHub in this case.
  const xcmsToAssetHub = result.value.forwarded_xcms.find(([location, _]) => (
    location.type === "V4" &&
      location.value.parents === 0 &&
      location.value.interior.type === "X1"
      && location.value.interior.value.type === "Parachain"
      && location.value.interior.value.value === 1000
  ))!;

  // We can even find the delivery fees for that forwarded XCM.
  const deliveryFeesQuery = await localApi.apis.XcmPaymentApi.query_delivery_fees(xcmsToAssetHub[0], xcmsToAssetHub[1][0]);

  if (deliveryFeesQuery.success) {
    const amount = deliveryFeesQuery.value.type === "V4" && deliveryFeesQuery.value.value[0].fun.type === "Fungible" && deliveryFeesQuery.value.value[0].fun.value.valueOf() || 0n;
    // We store them in state somewhere.
    setDeliveryFees(formatAmount(BigInt(amount)));
  }
}
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
# Conflicts:
#	cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs
#	cumulus/parachains/runtimes/assets/asset-hub-rococo/src/lib.rs
#	cumulus/parachains/runtimes/assets/asset-hub-westend/src/lib.rs
#	cumulus/parachains/runtimes/testing/penpal/src/lib.rs
#	polkadot/runtime/rococo/src/lib.rs
#	polkadot/runtime/westend/src/lib.rs
#	polkadot/xcm/xcm-fee-payment-runtime-api/tests/fee_estimation.rs
#	polkadot/xcm/xcm-fee-payment-runtime-api/tests/mock.rs
TarekkMA pushed a commit to moonbeam-foundation/polkadot-sdk that referenced this pull request Aug 2, 2024
…ytech#3872)

# Context

Estimating fees for XCM execution and sending has been an area with bad
UX.
The addition of the
[XcmPaymentApi](paritytech#3607)
exposed the necessary components to be able to estimate XCM fees
correctly, however, that was not the full story.
The `XcmPaymentApi` works for estimating fees only if you know the
specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics,
they don't necessarily know the XCM program that's executed by them.

# Main addition

A new runtime API is introduced, the `XcmDryRunApi`, that given an
extrinsic, or an XCM program, returns its effects:
- Execution result
- Local XCM (in the case of an extrinsic)
- Forwarded XCMs
- List of events

This API can be used on its own for dry-running purposes, for
double-checking or testing, but it mainly shines when used in
conjunction with the `XcmPaymentApi`.
UIs can use these two APIs to estimate transfers.

# How it works

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under
`xcm-fee-payment-runtime-api`.
There's a real-world test using Westend and AssetHubWestend under
`cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs`.
Added both a test for a simple teleport between chains and a reserve
transfer asset between two parachains going through a reserve.

The steps to follow:
- Use `XcmDryRunApi::dry_run_extrinsic` to get local XCM program and
forwarded messages
- For each forwarded message
- Use `XcmPaymentApi::query_delivery_fee` LOCALLY to get the delivery
fees
- Use `XcmPaymentApi::query_xcm_weight` ON THE DESTINATION to get the
remote execution weight
- (optional) Use `XcmPaymentApi::query_acceptable_payment_assets` ON THE
DESTINATION to know on which assets the execution fees can be paid
- Use `XcmPaymentApi::query_weight_to_asset_fee` ON THE DESTINATION to
convert weight to the actual remote execution fees
- Use `XcmDryRunApi::dry_run_xcm` ON THE DESTINATION to know if a new
message will be forwarded, if so, continue

# Dear reviewer

The changes in this PR are grouped as follows, and in order of
importance:
- Addition of new runtime API
- Definition, mock and simple tests:
polkadot/xcm/xcm-fee-payment-runtime-api/*
- Implemented on Westend, Asset Hub Westend and Penpal, will implement
on every runtime in a following PR
- Addition of a new config item to the XCM executor for recording xcms
about to be executed
  - Definition: polkadot/xcm/xcm-executor/*
  - Implementation: polkadot/xcm/pallet-xcm/*
- had to update all runtime xcm_config.rs files with `type XcmRecorder =
XcmPallet;`
- Addition of a new trait for inspecting the messages in queues
  - Definition: polkadot/xcm/xcm-builder/src/routing.rs
  - Implemented it on all routers:
    - ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
- ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on
implementation in cumulus/pallets/parachain-system/src/lib.rs)
    - XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
    - Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
- More complicated and useful tests:
-
cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

## Next steps

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have
the new API.
UIs can test on these runtimes to create better experiences around
cross-chain operations.

Next:
- Add XcmDryRunApi to all system parachains
- Integrate xcm fee estimation in all emulated tests
- Get this on the fellowship runtimes

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
TarekkMA pushed a commit to moonbeam-foundation/polkadot-sdk that referenced this pull request Aug 2, 2024
…tytech#4621)

Follow-up to the new `XcmDryRunApi` runtime API introduced in
paritytech#3872.

Taking an extrinsic means the frontend has to sign first to dry-run and
once again to submit.
This is bad UX which is solved by taking an `origin` and a `call`.
This also has the benefit of being able to dry-run as any account, since
it needs no signature.

This is a breaking change since I changed `dry_run_extrinsic` to
`dry_run_call`, however, this API is still only on testnets.
The crates are bumped accordingly.

As a part of this PR, I changed the name of the API from `XcmDryRunApi`
to just `DryRunApi`, since it can be used for general dry-running :)

Step towards paritytech#690.

Example of calling the API with PAPI, not the best code, just testing :)

```ts
// We just build a call, the arguments make it look very big though.
const call = localApi.tx.XcmPallet.transfer_assets({
  dest: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.Parachain(1000)) }),
  beneficiary: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.AccountId32({ network: undefined, id: Binary.fromBytes(encodeAccount(account.address)) })) }),
  weight_limit: XcmV3WeightLimit.Unlimited(),
  assets: XcmVersionedAssets.V4([{
    id: { parents: 0, interior: XcmV4Junctions.Here() },
    fun: XcmV3MultiassetFungibility.Fungible(1_000_000_000_000n) }
  ]),
  fee_asset_item: 0,
});
// We call the API passing in a signed origin 
const result = await localApi.apis.XcmDryRunApi.dry_run_call(
  WestendRuntimeOriginCaller.system(DispatchRawOrigin.Signed(account.address)),
  call.decodedCall
);
if (result.success && result.value.execution_result.success) {
  // We find the forwarded XCM we want. The first one going to AssetHub in this case.
  const xcmsToAssetHub = result.value.forwarded_xcms.find(([location, _]) => (
    location.type === "V4" &&
      location.value.parents === 0 &&
      location.value.interior.type === "X1"
      && location.value.interior.value.type === "Parachain"
      && location.value.interior.value.value === 1000
  ))!;

  // We can even find the delivery fees for that forwarded XCM.
  const deliveryFeesQuery = await localApi.apis.XcmPaymentApi.query_delivery_fees(xcmsToAssetHub[0], xcmsToAssetHub[1][0]);

  if (deliveryFeesQuery.success) {
    const amount = deliveryFeesQuery.value.type === "V4" && deliveryFeesQuery.value.value[0].fun.type === "Fungible" && deliveryFeesQuery.value.value[0].fun.value.valueOf() || 0n;
    // We store them in state somewhere.
    setDeliveryFees(formatAmount(BigInt(amount)));
  }
}
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
TarekkMA pushed a commit to moonbeam-foundation/polkadot-sdk that referenced this pull request Aug 2, 2024
…aritytech#4634)

Depends on paritytech#4621.

Implemented the
[`XcmPaymentApi`](paritytech#3607)
and [`DryRunApi`](paritytech#3872)
on all system parachains.

More scenarios can be tested on both rococo and westend if all system
parachains implement this APIs.
The objective is for all XCM-enabled runtimes to implement them.
After demonstrating fee estimation in a UI on the testnets, come the
fellowship runtimes.

Step towards paritytech#690.
@mrshiposha mrshiposha mentioned this pull request Aug 26, 2024
12 tasks
magecnion added a commit to freeverseio/laos that referenced this pull request Sep 12, 2024
@Polkadot-Forum
Copy link

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-dry-runner/10786/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T6-XCM This PR/Issue is related to XCM.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants