Limit the frequency that queryChannelsOnConnection() is called in the main relayer loop

[jtieri]

The main relayer loop queries all of the channels available for the configured connection, applies a filter rule (allow or deny list), then filters out only the channels that are in OPEN state, spins up a goroutine for each open channel that it is attempting to relay against and blocks until one of these goroutines die; at which point the slice of open channels is updated and the entire loop runs again.

Because the goroutines that are relaying packets/acks can die with some frequency we see the query for all channels on the connection run a lot. This query can fetch a rather large collection of data and puts some strain on the configured RPC endpoint so on busier IBC paths we see queryChannelsOnConnection() fail after the main relayer loop runs a few times and then the relayer process exits.

There is an open issue #685 to not quit if this query fails but to instead keep dialing if we have already run the query once in the same runtime but I propose that we also limit how often this query is even ran.

[boojamya]

Potentially only call this once when the relayer starts up, and cache results on path.

[jtieri]

This is an option as well. My thought process for including the query inside of the loop instead of running it once when the relayer starts is that it allows the user to have a constantly updated list of open channels which can be useful especially in the case of ordered channels, e.g. interchain accounts, where the channel could potentially be closed and then later reopened.

For interchain accounts we will additionally need to be querying txs and checking the events for ChannelOpenInit so that we can properly open channels since the interchain accounts module initiates the channel handshake and the relayer just finishes it. So maybe we actually do just execute queryChannelsOnConnection() once at startup and then inside of the loop we can also query for ChannelOpenConfirm events to pickup any new channels that could have been opened.

maybe @jackzampolin has some thoughts on this as well