Expose number of threads per search in metrics

[dainiusjocas]

Is your feature request related to a problem? Please describe.

We are woking on query performance and estimating how much throughput can be handled per content node. We've made a nice Grafana dashboard with these estimates (based on docs and the Little's Law) but for now it is needed to manually add constants for how many threads are used per search per rank profile to make the numbers match.

The main pain point is that some queries require many threads (e.g. 4 or 8) while some are good with 1 or 2. And thread per search varies per rank-profile. We have about ~15 rank profiles.

The threads per search count per rank profile is mostly static in production except when full scale experiments are being run.

Having the default thread count as set in services.xml or in rank profile would also be useful.

Describe the solution you'd like
I'd like that Prometheus metrics for rank_profile type, e.g. content.proton.documentdb.matching.rank_profile.query_latency would contain a label for how many threads were actually used.

Describe alternatives you've considered
I've created promql with hardcoded thread counts per rank profile as multipliers in the price calculations.

Additional context
Vespa Slack thread.