feat: More performance metrics

[simster7]

Closes: #4800

Signed-off-by: Simon Behar simbeh7@gmail.com

Checklist:

• My organization is added to USERS.md.

[alexec]

I did something similar to this recently, but I took a slightly different approach. I created a new queue:

metrics.NewWorkQueueWithMetrics(workqueue.NewNamedRateLimitingQueue(&fixedItemIntervalRateLimiter{}, "workflow_queue"), "workflow_queue")

I don't think there's much in it mind you. Just a thought.

[alexec]

@simster7 I've got a few metrics we might want to do at some point:

1. Busy workers.
2. K8S API Metrics
3. Count of workflow pods by phase (similar impl to count of workflows)
4. Count of workflows that actually have running pods (by adding a condition).

I did PoC for these, but 3 and 4 are a bit complex and I don't know if we need both.

[simster7]

Sure, I'll look into this

[simster7]

Busy workers Added

K8S API Metrics: not sure it should be our responsibility to emit K8s metrics. Shouldn't K8s have an endpoint for these already? K8s emits metrics already: https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/

Count of workflow pods by phase (similar impl to count of workflows): currently have only 1 metric for active pods (i.e. phase==Running). I had actually first set the metric for all phases, but that would require changing our Pod informer from only looking at not completed pods to looking at all pods. If we think this is worth it, I can work to ensure that changing the nature of the informer doesn't have any intended consequences.

Count of workflows that actually have running pods (by adding a condition): this seems to be tied more to the refactored mentioned in #4824. This should perhaps be changed with that

[alexec]

I think some tests are probably in order for this PR please.

[alexec]

I think these mutex may create contention in the VM - doesn't the gauge already have these?

[simster7]

Sorry, I'm not too sure what you mean. These mutexes are needed to keep concurrency issues between writing/updating metrics and metrics being scraped. While individual metrics may be concurrent safe, the collection of metrics and the metrics scraper are not without these locks. They are present in all other functions in this file.

[alexec]

Can you take a look at https://github.com/argoproj/argo/pull/4811/files#diff-2721f0228996fd51c5f9a8db168a353b398d051154c82cebb03f03ddd1ee0574?

I think it gives a way to do the workers busy metric very safely.

[alexec]

K8S API Metrics: not sure it should be our responsibility to emit K8s metrics. Shouldn't K8s have an endpoint for these already? K8s emits metrics already: https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/

Interesting. @jessesuen are you aware of this?

Count of workflow pods by phase (similar impl to count of workflows): currently have only 1 metric for active pods (i.e. phase==Running). I had actually first set the metric for all phases, but that would require changing our Pod informer from only looking at not completed pods to looking at all pods. If we think this is worth it, I can work to ensure that changing the nature of the informer doesn't have any intended consequences.

I think pods that are pending are useful. I think you can do all in-complete pods.

Count of workflows that actually have running pods (by adding a condition): this seems to be tied more to the refactored mentioned in #4824. This should perhaps be changed with that

This is harder and I'm not 100% sure this is a valuable metric. Lets hold on it.

[alexec]

some minor comments - maybe when you merge give a more specific commit message than the PR currently has?

[alexec]

delete comments?