Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add shard id to queue processor related metrics #5557

Conversation

taylanisikdemir
Copy link
Member

What changed?
History's queue processors are created per shard by history engine. Logs had shard-id tag but metrics were missing it. So adding shard id tag to relevant metrics.

Why?
To improve overall monitoring of queue processor operations.

How did you test it?

  1. Build and run
./scripts/buildkite/docker-build.sh
docker-compose -f docker/docker-compose.yml up
  1. Visit prometheus UI: http://localhost:9090/graph
  2. Query some queue processor metrics and validate shard_id is there
rate(task_batch_complete_counter{operation="TimerQueueProcessor"}[1m])

Potential risks
We default to 16k shards so cardinality of these queue processor metrics shouldn't be an issue. There's not other high cardinality tags in those metrics.

@taylanisikdemir taylanisikdemir force-pushed the taylan/queue_proc_shardid_metric_tag branch from 6d7e3b3 to 4ccb876 Compare December 28, 2023 17:31
@taylanisikdemir taylanisikdemir force-pushed the taylan/queue_proc_shardid_metric_tag branch from 911313b to 56c58f4 Compare December 28, 2023 18:20
@taylanisikdemir taylanisikdemir merged commit 5a21597 into cadence-workflow:master Dec 28, 2023
16 checks passed
@taylanisikdemir taylanisikdemir deleted the taylan/queue_proc_shardid_metric_tag branch December 28, 2023 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants