XcmDryRunApi - Dry-running extrinsics to get their XCM effects

[franciscoaguirre]

(Note: followed-up by #4621 to use call instead of extrinsic)

Context

Estimating fees for XCM execution and sending has been an area with bad UX.
The addition of the XcmPaymentApi exposed the necessary components to be able to estimate XCM fees correctly, however, that was not the full story.
The XcmPaymentApi works for estimating fees only if you know the specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics, they don't necessarily know the XCM program that's executed by them.

Main addition

A new runtime API is introduced, the XcmDryRunApi, that given an extrinsic, or an XCM program, returns its effects:

• Execution result
• Local XCM (in the case of an extrinsic)
• Forwarded XCMs
• List of events

This API can be used on its own for dry-running purposes, for double-checking or testing, but it mainly shines when used in conjunction with the XcmPaymentApi.
UIs can use these two APIs to estimate transfers.

How it works

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under xcm-fee-payment-runtime-api.
There's a real-world test using Westend and AssetHubWestend under cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs.
Added both a test for a simple teleport between chains and a reserve transfer asset between two parachains going through a reserve.

The steps to follow:

• Use XcmDryRunApi::dry_run_extrinsic to get local XCM program and forwarded messages
• For each forwarded message
  • Use XcmPaymentApi::query_delivery_fee LOCALLY to get the delivery fees
  • Use XcmPaymentApi::query_xcm_weight ON THE DESTINATION to get the remote execution weight
  • (optional) Use XcmPaymentApi::query_acceptable_payment_assets ON THE DESTINATION to know on which assets the execution fees can be paid
  • Use XcmPaymentApi::query_weight_to_asset_fee ON THE DESTINATION to convert weight to the actual remote execution fees
  • Use XcmDryRunApi::dry_run_xcm ON THE DESTINATION to know if a new message will be forwarded, if so, continue

Dear reviewer

The changes in this PR are grouped as follows, and in order of importance:

• Addition of new runtime API
  • Definition, mock and simple tests: polkadot/xcm/xcm-fee-payment-runtime-api/*
  • Implemented on Westend, Asset Hub Westend and Penpal, will implement on every runtime in a following PR
• Addition of a new config item to the XCM executor for recording xcms about to be executed
  • Definition: polkadot/xcm/xcm-executor/*
  • Implementation: polkadot/xcm/pallet-xcm/*
  • had to update all runtime xcm_config.rs files with type XcmRecorder = XcmPallet;
• Addition of a new trait for inspecting the messages in queues
  • Definition: polkadot/xcm/xcm-builder/src/routing.rs
  • Implemented it on all routers:
    • ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
    • ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on implementation in cumulus/pallets/parachain-system/src/lib.rs)
    • XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
    • Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
• More complicated and useful tests:
  • cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

Next steps

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have the new API.
UIs can test on these runtimes to create better experiences around cross-chain operations.

Next:

• Add XcmDryRunApi to all system parachains
• Integrate xcm fee estimation in all emulated tests
• Get this on the fellowship runtimes

[xlc]

How will this play with orml-xtokens?

orml-xtokens provided calls that accepts CurrencyId instead of Location and it internally figure out the right Location to use.

one way to achieve similar result without too coupling is dry run the extrinsic whatever it is, and read out the ump / xcmp queue and we know exactly the XCM that will be sent

[franciscoaguirre]

The idea was that both pallet-xcm and orml-tokens could provide functions for returning the messages they send.
Then each runtime would implement this API, calling the underlying function from the pallet they use for XCM transfers.

Dry-running the extrinsic definitely sounds like a better option.
In the case of reserve asset transfers, you'd also need to dry-run the XCMs to know what other messages they send.
I think it's better for UIs since they only have access to the extrinsic.

[franciscoaguirre]

We can know what messages are put in the queue, but we can't know the message that is executed locally by the extrinsic.

[mrshiposha]

It seems the pallet developers should provide some support for their extrinsics for such an API anyway.
However, it may be possible to simplify this API for clients...

For instance,

pub trait XcmProgramsApi {
    fn query_xcm_programs(xt: <Block as BlockT>::Extrinsic) -> Result<Vec<(Location, Xcm<()>)>, /* ErrT */>;
}

And then, when implementing this API for a chain:

impl XcmProgramsApi<Block> for Runtime {
    fn query_xcm_programs(xt: <Block as sp_api::BlockT>::Extrinsic) -> Result<Vec<(Location, Xcm<()>)>, /* ErrT */> {
        match xt.0.function {
            RuntimeCall::PalletXcm(pallet_xcm::Call::transfer_assets { /* params */ }) => { todo!("return the programs") },
            RuntimeCall::XTokens(pallet_xtokens::Call::transfer { /* params */ }) => { todo!("return the programs") },
            _ => Err(/* whatever */),
        }
    }
}

So, the client could provide an extrinsic to this API and retrieve the XCM programs. The chain could determine the programs based on the provided extrinsic by itself.

This way, we can return the program that will be executed by the extrinsic locally and dry-run the extrinsic as well to return the programs from the ump / xcmp queue. Also, to simplify the API implementation, the pallets could provide public functions for handling the todo!("return the programs")

Note: an extrinsic should be signed, but the client can use the signFake from PolkadotJS to analyze the extrinsic using the API first. It's similar to how the paymentInfo is implemented for the SubmittableExtrinsic: https://github.com/polkadot-js/api/blob/7874a8e7234219e47abca6a37ae0558aa4731751/packages/api/src/submittable/createClass.ts#L167-L169.

But maybe I'm missing something. I didn't try to implement this :)
Just sharing thoughts.

[mrshiposha]

Hmm, I think that dry-running an extrinsic to learn what messages will be sent where is not enough. It covers the case for transferring assets from Chain A to Chain B. However, it doesn't cover this case: Chain A --> Reserve --> Chain B.

However, if we have the XCM program that the extrinsic wants to execute locally on the first chain, we can easily analyze the program to determine all the locations and messages. We just need to analyze the initial program and look at the InitiateReserveWithdraw and other similar instructions to recursively extract all the locations and messages. Such an analysis of an XCM program can be done in a common function that all chains can reuse when implementing the API from the comment above

[acatangiu]

I was discussing something similar with @franciscoaguirre, exploring something similar:

pub struct XcmDryRunEffects {
    locally_executed_xcm: VersionedXcm<()>,
    forwarded_xcms: Vec<(VersionedLocation, VersionedXcm<()>)>,
    execution_result: DispatchResult,
    // events ?
}

pub trait DryRunXcmApi {
    /// query side effects for executing extrinsic
    fn query_xcm_for_extrinsic(
        xt: <Block as BlockT>::Extrinsic,
    ) -> Result<XcmDryRunEffects, /* ErrT */>;

    /// query side effects for executing (incoming) XCM
    fn query_execute_xcm(
        xcm: Xcm<()>,
        using_origin: Location,
    ) -> Result<XcmDryRunEffects, /* ErrT */>;
}

We want to:

• find out locally paid fees as part of locally executing an XCM either coming from some other location or coming from some locally run extrinsic
• for same as above, find out any outgoing XCMs and their destinations as a result
• be able to repeat the same operations on said destinations for these outgoing XCMs.

Would this be enough to cover all needs?

[franciscoaguirre]

Very draft still but I'm working on the dry-running idea

[franciscoaguirre]

I have now both the dry_run_extrinsic and dry_run_xcm calls. The only problem I see with the extrinsic one is we should have a way to get the actual xcm program that's being sent to the executor.
Other than that, the API looks much better.
It would need something to solve that situation and then some code cleanup.

[franciscoaguirre]

@xlc I managed to simplify the way of dry-running an extrinsic and getting both the local message and the remote messages. I think this'll make it easy to be used with xtokens. Can you take a look and tell me what you think?

Also, Here I'm dry-running both an extrinsic and an XCM, which look like #230 and #486. I could move the extrinsic one to a more generic runtime API. What are your thoughts on trying to tackle these issues here?

[acatangiu]

Looking great! 🚀

Only have impl details comments, see below

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-technical-summit-xcm/9384/1