Improve performance of health check and only perform it on hermes start

[adizere]

Closes: #1336

Description

This PR also adds a health-check command:

USAGE:
    hermes health-check <OPTIONS>

DESCRIPTION:
    Performs a health check of all chains in the the config

$ hermes health-check
Sep 16 21:54:03.360  INFO using default configuration from '/Users/coromac/.hermes/config.toml'
Sep 16 21:54:03.372  INFO [ibc-0] performing health check...
Sep 16 21:54:03.436  INFO [ibc-0] chain is healthy
Sep 16 21:54:03.437  INFO [ibc-1] performing health check...
Sep 16 21:54:03.443  INFO [ibc-1] chain is healthy
Success: performed health check for all chains in the config

---

For contributor use:

• Added a changelog entry, using unclog.
• If applicable: Unit tests written, added test to CI.
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
• Updated relevant documentation (docs/) and code comments.
• Re-reviewed Files changed in the Github PR explorer.

[andynog]

@romac the code lgtm. Couple of questions:

I've tested it with a config that has to chains (cosmoshub-4 and osmosis-1). In the cosmoshub chain config I deliberately set an invalid value for the max_tx_size but for osmosis is fine. So when running the command I see that the health-check errors out for one of them but not the other. Should this be a condition to fail to start the relayer or it's just for logging and an operator will have to check the log to identify an issue with the config?

hermes -c config.toml health-check
Sep 17 10:56:56.468  INFO [cosmoshub-4] performing health check...
Sep 17 10:56:56.829  WARN Hermes might be misconfigured for chain 'cosmoshub-4'
Sep 17 10:56:56.829  WARN     Reason: 
   0: semantic config validation failed for option `max_tx_size` chain 'cosmoshub-4', reason: `max_tx_size` = 2097152 is greater than 90% of the genesis block param `max_size` = 200000

Location:
   /home/andy/.cargo/registry/src/github.com-1ecc6299db9ec823/flex-error-0.4.2/src/tracer_impl/eyre.rs:10

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Sep 17 10:56:56.829  WARN     Some Hermes features may not work in this mode!
Sep 17 10:56:56.830 ERROR [cosmoshub-4] chain is unhealthy: 
   0: semantic config validation failed for option `max_tx_size` chain 'cosmoshub-4', reason: `max_tx_size` = 2097152 is greater than 90% of the genesis block param `max_size` = 200000

Location:
   /home/andy/.cargo/registry/src/github.com-1ecc6299db9ec823/flex-error-0.4.2/src/tracer_impl/eyre.rs:10

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Sep 17 10:56:56.833  INFO [osmosis-1] performing health check...
Sep 17 10:56:57.261  INFO [osmosis-1] chain is healthy
Success: performed health check for all chains in the config

Another question. Should the updated guide with the new command health-check be part of the PR ?

[romac]

It's just for logging but I agree that we should clean up the log and not display the backtrace, just the error message.

[romac]

Should the updated guide with the new command health-check be part of the PR ?

No, we currently do the guide updates in the release PR, because otherwise they show up before the feature is released. Hopefully once we have versioning we'll be able to do the guide updates in the PR directly.

[andynog]

Looks good to me. The output of the logs can be improved but I think they are not blockers.