Support for unordered transactions for CosmosSDK-based chains

[adizere]

Summary

For Cosmos SDK-based chains, the next version (i.e., likely SDK v0.51) will have support for executing unordered transactions.

This means the Hermes relayer will not need to include nonces anymore. Also, Hermes can not longer make ordering assumptions among transactions it is submitting (eg if nonce X has client update, it should be executed before nonce X+1 that depends on X).

On the positive side, if Hermes manages correctly the pipeline of submitted transactions, this feature should enable better UX by eliminating problems with "sequence mismatch".

The feature is unreleased as of today

https://github.com/cosmos/cosmos-sdk/blob/f431a5039cecd04c3d66f0c43b9df75b89d88037/CHANGELOG.md?plain=1#L43

Context

• the Hermes team has previously opened two related issues
  • Support for parallel nonces to allow for safe submission of concurrent transactions cosmos/cosmos-sdk#13009
  • gRPC Tx simulation sometimes fails with account sequence mismatch cosmos/cosmos-sdk#11597
• ADR in the SDK: adr: Un-Ordered Transaction Inclusion cosmos/cosmos-sdk#18553
• Implementation in the SDK is spread across two PRs:
  • feat: [ADR-070] Unordered Transactions (1/2) cosmos/cosmos-sdk#18641
  • feat: [ADR-070] Unordered Transactions (2/2) cosmos/cosmos-sdk#18739

Acceptance Criteria

• New design for dealing with unordered transactions in Hermes
• Test scenarios

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ancazamfir]

This means the Hermes relayer will not need to include nonces anymore. Also, Hermes can not longer make ordering assumptions

We will need to be very careful with any changes we make (if we do) when submitting transactions to SDK v0.51. And properly test.
For legacy hermes we serialize the transaction submission anyway (via the single chain runtime) so we cannot parallelize.

If we don't explicitly flag the Tx-es as unordered then the behavior should be the same as before (so we can still make the ordering assumption).

The only potential advantage would be to simulate as unordered so we don't run into nonce mismatch due to recheck (i.e. we get the got > expected error that we ignore today) and proceed with the Tx submission (we must not mark unordered here).
But then we may run into other simulation failures if IBC dependent transactions have not been rechecked and we will need to figure out what to do then (still ignore but we need to figure this out from other error types and not sure about the accuracy there).
Then we still want to catch the got < expected mismatch case that indicates multiple usage of same account from multiple instances (in this case we need to re-cache the account nonce).

So at first look it is not clear to me how to take advantage of the proposal and just hope the legacy way will still work fine.