Issues uncovered when testing with DEX liquidityd

[ancazamfir]

Crate

relayer, realyer-cli

Summary

Many thanks to @ebuchman and @andynog for testing this.

• liquidityd uses timestamp based timeout packets
  • add support for it (Add support for packet timeout based on the timeout_timestamp #937)
• hermes does not decode the timeout_timestamp from Tx result event when clearing packets...
  • therefore building a recv_packet with timestamp 0 while the proof collected from the source chain is for the packet with the proper value. (Add support for packet timeout based on the timeout_timestamp #937)
  • This causes validation failure on destination chain (Add support for packet timeout based on the timeout_timestamp #937)
  • hermes keeps retrying for this failure delaying active packets that are sent which sometimes may timeout by the time hermes gets to them deferred to Relayer error handling specification #712
• when multiple messages are sent with send_msgs() we use an algorithm that splits them in multiple Tx-es based on maximum allowed size and number of messages per Tx. This does not work well with gas configured with a low value.
  • one problem is that we stop at first message that increases the size of the Tx over the max but we include that message. (tbd)
  • the other is that we do not currently have a dynamic way to determine the required gas (Evaluate use of Tx simulate to determine the fees #930)
• when processing a batch of events hermes starts with the timeouts and then receive packets.
  • The order should be reversed such that any retries caused by the timeout processing do not cause more timeouts in the pending recv packets. (Add support for packet timeout based on the timeout_timestamp #937)
• we hit again the problem of bad handling when multiple messages are split across one or more Tx-es with some returning errors.
  • We retry them all, even the ones that were in a successful Tx (Evaluate use of Tx simulate to determine the fees #930 )
• i believe @andynog saw an issue with query clients
  • pagination problem, on cosmoshub (only 100 clients were returned instead of ~146) (Query clients only returns 100 entries #946) UX issue, fixed in Include chain-id in hermes query clients output #995

Problem Definition

see above

Proposal

Fix most of these issues before DEX launch

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned