Define monitoring / devops strategy for all the components

[gamarin2]

Things to monitor:

• Full-nodes
• Relayers
• Health of our accounts (need to trigger an alert when they go below a certain limit)
• API Server
• CockroachDB instance

[gsora]

Thinking out loud:

Full-nodes

Cosmos SDK exposes Prometheus metrics when configured properly, we can definitely use those.

Relayers

@nodebreaker0-0 figured out some interesting metrics to be included in the Hermes relayer - still Prometheus-based, we are waiting for the Hermes team.

Health of our accounts (need to trigger an alert when they go below a certain limit)

We need to define "account health" first.

After that I think we can build something that pushes our account's balance on Prometheus and instrument alerts from that.

Maybe embed this functionality in CNS?

API Server

There are lots of premade Gin Prometheus exporter we can use to gather interesting metrics.

Some examples:

• https://github.com/penglongli/gin-metrics - this looks interesting but lacks support for less verbose URLs
• https://github.com/zsais/go-gin-prometheus - barebone but looks better, supports less verbose URLs metrics

CockroachDB instance

They natively support Prometheus exporting: https://www.cockroachlabs.com/docs/v20.2/monitor-cockroachdb-with-prometheus.html

Bottom line

[gamarin2]

Update: https://www.notion.so/allinbits/Infrasec-Emeris-MVP-3b85fdeeff0645e3ad52947223516621 (thanks to @gsora)

Will close this issue once we're happy with state of monitoring for MVP