Poor performance for Tx-es and queries

[ancazamfir]

Crate

relayer

Summary of Bug

Tx-es and queries take a long time to complete. For example getting the account information takes ~1sec:

Jan 07 14:05:31.681  INFO relayer::chain::cosmos:  => getting account info
Jan 07 14:05:32.581  INFO relayer::chain::cosmos:  => got account info

Sending a Tx takes ~2sec:

Jan 07 14:05:32.663  INFO relayer::chain::cosmos:  => sending tx
Jan 07 14:05:34.728  INFO relayer::chain::cosmos:  => received tx response

The problem is here:
https://github.com/informalsystems/ibc-rs/blob/9c5d4f60732db62626cd7163f7e3b87f9f3c289a/relayer/src/chain/cosmos.rs#L132-L134

Version

master

Steps to Reproduce

Run relayer loop or CLIs an notice the slow processing times.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ancazamfir]

After replacing some of the self.block_on with util's block_on:
https://github.com/informalsystems/ibc-rs/blob/5c9739c536ee253baa5bba2b4f2b20968b3489b7/relayer/src/util/block_on.rs#L6-L13

in anca/relayer_profiling branch, the performance improves dramatically:

Old:

Jan 07 14:05:31.681  INFO relayer::chain::cosmos:  => getting account info
Jan 07 14:05:32.581  INFO relayer::chain::cosmos:  => got account info
Jan 07 14:05:32.663  INFO relayer::chain::cosmos:  => sending tx
Jan 07 14:05:34.728  INFO relayer::chain::cosmos:  => received tx response

New:

Jan 07 14:14:30.121  INFO relayer::chain::cosmos:  => getting account info
Jan 07 14:14:30.125  INFO relayer::chain::cosmos:  => got account info
Jan 07 14:14:30.205  INFO relayer::chain::cosmos:  => sending tx
Jan 07 14:14:31.035  INFO relayer::chain::cosmos:  => received tx response

These changes also uncovered a bug (also fixed in same branch) in relayer loop where latest height minus one is used for proofs when building packets. This is a problem because at h-1 and with now fast processing this may become a non-membership proof (e.g a send packet was not present at h-1)