Fix bug in usage metrics when multiple service instances are changed in a single transaction

[crhino]

Symptom

A log line like so:

2020-12-18T20:39:41.784Z [WARN]  agent.fsm: DeleteNode failed: error="failed to insert usage entry for "service-names": delta will cause a negative count"

Cause

When multiple service instances are registered/deregistered at once, usage metrics of service names would be computed incorrectly, eventually leading to state store errors like the above. This could happen on DeleteNode RPCs, and also when Consul does a restore.

To fix, we collect all of the memdb changes and then do a pass through the updated memdb state to reconcile, instead of dealing with the memdb changes one at a time.

Unfortunately it might not error out immediately, if you have enough services that the count does not go negative. Instead, the usage metrics will be subtly wrong until another service deletion forces the count negative.

Testing

I have tested out reproduction cases involving both

curl -s -XPUT -H"X-Consul-Token: $CONSUL_HTTP_TOKEN" $CONSUL_HTTP_ADDR/v1/catalog/deregister -d '{"Node": "consul-dc1-client0"}'

and

$ consul snapshot save testing.snap
Saved and verified snapshot to index 161
$ consul snapshot restore testing.snap
Restored snapshot

while watching:

$ curl -s -H"X-Consul-Token: $CONSUL_HTTP_TOKEN" $CONSUL_HTTP_ADDR/v1/agent/metrics | jq '.Gauges[] | select(.Name | contains("state"))'

and this PR does fix the issues and correctly computes usage metrics.

[mkeeler]

What happens if the Txn api is used to delete multiple instances of a service name from two nodes within the same transaction?

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/hashicorp/consul/b4yxpixg4
✅ Preview: Canceled

[hashicorp-ci]

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/309152.

[hashicorp-ci]

🍒✅ Cherry pick of commit 0712e03 onto release/1.9.x succeeded!