Incorrect account sequence logic upon gas errors

[adizere]

Crate

Summary of Bug

When Hermes encounters an error submitting a transaction, it mistakenly increments the (cached) account sequence number.

See the trace below, provided from Gravity DeX ops:

Sep 30 13:53:44 hznr1 hermes[1800189]: Sep 30 13:53:44.780 DEBUG ThreadId(69) [impacthub-3] send_tx: retrieved account sequence=189 number=214
Sep 30 13:53:44 hznr1 hermes[1800189]: Sep 30 13:53:44.780 DEBUG ThreadId(69) [impacthub-3] send_tx: sending 11 messages using nonce 189
Sep 30 13:53:44 hznr1 hermes[1800189]: Sep 30 13:53:44.795 DEBUG ThreadId(69) [impacthub-3] send_tx: broadcast_tx_sync: Response { code: Err(11), data: Data([]), log: Log("out of gas in location: txSize; gasWanted: 500000, gasUsed: 597058: out of gas"), hash: transaction::Hash(9A3FB110201A811D8B4E469B19E03935040EFE97181E6EA1CA5C115B2C9DB085) }
Sep 30 13:53:46 hznr1 hermes[1800189]: Sep 30 13:53:46.379 DEBUG ThreadId(69) send_messages_and_wait_check_tx with 2 messages
Sep 30 13:53:46 hznr1 hermes[1800189]: Sep 30 13:53:46.379 DEBUG ThreadId(69) [impacthub-3] send_tx: sending 2 messages using nonce 190
Sep 30 13:53:46 hznr1 hermes[1800189]: Sep 30 13:53:46.388 DEBUG ThreadId(69) [impacthub-3] send_tx: broadcast_tx_sync: Response { code: Err(32), data: Data([]), log: Log("account sequence mismatch, expected 189, got 190: incorrect account sequence"), hash: transaction::Hash(ED1C62FCEF2DFDBDE5B86B2AE820441D6028AD164D9CE9760799C4C715B1BCD7) }
Sep 30 13:53:48 hznr1 hermes[1800189]: Sep 30 13:53:48.573 DEBUG ThreadId(69) send_messages_and_wait_check_tx with 2 messages
Sep 30 13:53:48 hznr1 hermes[1800189]: Sep 30 13:53:48.573 DEBUG ThreadId(69) [impacthub-3] send_tx: sending 2 messages using nonce 191
Sep 30 13:53:48 hznr1 hermes[1800189]: Sep 30 13:53:48.580 DEBUG ThreadId(69) [impacthub-3] send_tx: broadcast_tx_sync: Response { code: Err(32), data: Data([]), log: Log("account sequence mismatch, expected 189, got 191: incorrect account sequence"), hash: transaction::Hash(E15A53C54EADB3BF2A4AF0334035B90AF98EE3DBDE9182542B4830506A96376A) }

Steps in the trace above

1. Hermes fetches the acct s.n. from a full node, namely 189
2. Sends tx with s.n. 189
3. Hermes encounters an "out of gas" error
4. Increments the account sequence to 190 -- which it should not do! the tx never entered the mempool as broadcast_tx_sync failed, so the s.n. remained unchanged
5. Hermes then sends a subsequent tx using the wrong s.n. 190

The problematic logic is here
https://github.com/informalsystems/ibc-rs/blob/7bcd7f1da606569681d961016800728815149d3a/relayer/src/chain/cosmos.rs#L277-L281

Version

0.7.2

Acceptance Criteria

• Hermes avoids calling into incr_account_sequence if the call to broadcast_tx_sync is an error code

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned