[Response Ops][Alerting] Exposing background worker utilization load metric

[ymao1]

Resolves #155762, #155761

Summary

This PR exposes a metric that represents the background task worker utilization load at the end of each polling cycle. This is calculated as (# of workers already busy + claimed tasks) / max workers, which comes out to the number of workers in use at the end of each claim cycle. This metric is then averaged over the previous 15 seconds (or 5 polling cycles). This window size is configurable using xpack.task_manager.worker_utilization_running_average_window

This PR exposes this metric in the existing /internal/task_manager/_background_task_utilization API but also adds a public version of this API (/api/task_manager/_background_task_utilization) that only exposes this metric. We need the public API for serverless but I thought we could keep the private route as well to expose experimental metrics without the overhead of supporting them long term.

To Verify

1. Run ES and Kibana with this branch
2. Navigate to /internal/task_manager/_background_task_utilization and see the new metric exposed as stats.value.load along with the existing adhoc and recurring metrics
3. Navigate to /api/task_manager/_background_task_utilization and see only the load metric returned from the public API
4. You can also create some rules to see the load metric increase.

[ymao1]

@elasticmachine merge upstream

[ymao1]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

@elasticmachine merge upstream

[ymao1]

@kobelb I made an /api and an /internal endpoint so we could have experimental metrics without the overhead of having to support them but if you think that's overkill, I can remove the /internal endpoint and remove the adhoc and recurring counters.

[kobelb]

@kobelb I made an /api and an /internal endpoint so we could have experimental metrics without the overhead of having to support them but if you think that's overkill, I can remove the /internal endpoint and remove the adhoc and recurring counters.

Works for me!

[ymao1]

@elasticmachine merge upstream

[ymao1]

@elasticmachine merge upstream

[mikecote]

Tested locally and changes LGTM! Saw the load go up to 100 when overloaded and back down to ~6 when idle from alerting rules.

[jbudz]

kibana-docker

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 54fd3c3

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	398	401	+3
taskManager	24	23	-1
total			+4

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	478	481	+3
taskManager	24	23	-1
total			+4

History

• 💔 Build #124755 failed 111c7ea
• 💛 Build #124567 was flaky 1fd04ef
• 💔 Build #124486 failed 75324dd
• 💛 Build #123043 was flaky 1c9971f
• 💚 Build #122723 succeeded bd34bcb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1