services/horizon/ingest: added 'horizon_ingest_errors_total' metric key

[sreuland]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

added a new metrics counter key 'horizon_ingest_errors_total' which ingest fsm will emit anytime it traps an error response from in ingestion state run.

Why

to configure prometheus alert against the new metric for early visibility on when an ingestion halt starts to form - https://github.com/stellar/prometheus/pull/243

Closes: #5256

Known limitations

[tamirms]

We log ingestion errors here:

https://github.com/stellar/go/blob/master/services/horizon/internal/ingest/main.go#L646

If you search on kibana you can see that these logs occur very rarely. I think it would make sense to have an ingestion error counter metric which is basically triggered every time we emit the log in https://github.com/stellar/go/blob/master/services/horizon/internal/ingest/main.go#L646. The metric definition should resemble the log message: a counter with 2 labels, current_state and next_state.

We can the create the alert using a very simple heuristic: if the ingestion error counter is incremented at least n times in the last m minutes fire the alert

[socket-security]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
golang/github.com/stretchr/testify@v1.9.0	filesystem, network, shell, unsafe	0	616 kB

🚮 Removed packages: golang/github.com/stretchr/testify@v1.8.4

View full report↗︎

[sreuland]

I think it would make sense to have an ingestion error counter metric which is basically triggered every time we emit the log in https://github.com/stellar/go/blob/master/services/horizon/internal/ingest/main.go#L646.

ah, thanks for pointer to that, updated - 3319801

[tamirms]

I left a few minor comments but overall looks good!