Add Grafana dashboard for monitoring OPEA application scaling in k8s

[eero-t]

Description

Adds Grafana dashboard for monitoring OPEA application scaling:

• How many of the application and its TGI + TEI pods are created, ready and in use
• How many requests they are processing (min and max across all replicats)
• How many failures they are reporting (sum across replicas)

And a helper script for installing dashboard k8s configMaps for Grafana.

Unlike earlier ChatQnA dashboard, this handles multiple OPEA application having same names but being in separate namespaces. User selects namespace and then the OPEA application from that. If cluster has only one running, Dashboard will default to that.

(Therefore it does not make sense to install dashboard with application specific Helm charts, as it can cover all apps that use TGI for LLM, i.e. most of them.)

Issues

n/a.

Type of change

• New feature (non-breaking change which adds new functionality)

Dependencies

n/a.

Tests

Manual testing of the script and dashboard working.

[eero-t]

Currently dashboard relies on HTTP inprogress metric for how many pending requests application has: opea-project/GenAIComps#845

But depending on whether following PR is merged for v1.1, that particular metric may need to be changed before v1.1: opea-project/GenAIComps#864

[eero-t]

I can add blurb about this also to README, but scaling is currently a bit of corner case, so IMHO it could come also in next release.

Larger question about Observability README, and things it refers to, is what to do with chatqna/ sub-directory content here, now that Helm charts have more generic monitoring support for OPEA applications.

Regarding the dashboards under that:

• queue_size_embedding_rerank_tgi.json: some queries in that do not have any selectors, some use service selector
• tgi_grafana.json: queries use container selector (container="$service")

I.e. neither handles properly cases when cluster is running multiple OPEA applications with TGI instances. The new dashboard covers first one to some extent. TGI details dashboard could be updated to have similar selectors as this new dashboard.

[eero-t]

FYI: I'm going to change dashboard "Failures" heading to "Incomplete requests". I do not think half of TGI requests are failures, but that frontend needs to request rest of reply with another query before TGI deems it "complete" (successful).

[eero-t]

Dashboard changes:

• Changed HTTP metric to one that will come with: Replace HTTP "inprogress" gauge with megaservice "request_pending" one GenAIComps#864
• Lowered scaling & failures rows so that dashboard first better to screen
• "Failures" -> "Incomplete requests" header change