Hermes sometimes fails to retrieve the begin/end block events

[ancazamfir]

Summary of Bug

cc @romac
We discovered an issue with block_search RPC on the incentivized ICS testnet. The query results are not what we expect from hermes for begin/ end block events.
The tendermint issue to clarify this is tendermint/tendermint#9712.
While we are waiting for a resolution on this we will likely need to have a fix in hermes.

When querying for the begin/ end block events for a given event and packet, block_search may return a list of blocks. We need to retrieve the block_results for each block and find the one that includes the one event with the type and attributes matching the query. Currently we only ask for one block in block_search and we attempt to retrieve the results for that one block which may not be the correct one, resulting in packet data not being found and relaying failing.

Version

all

Steps to Reproduce

Currently this is not easy without a custom gaia or using the ICS testnet.
We should write integration tests if possible.

Acceptance Criteria

Be able to retrieve the block and results that include the event with the exact attributes in query.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned