Add premade dashboard to monitor Traefik [enhancement]

[Adam-D-Lewis]

I'm looking into adding https://github.com/tcheronneau/grafana_dashboard to the qhub deployment for Traefik. The dashboard doesn't work off the bat. I'm still working on making the changes to get it working.

[costrouc]

Thanks for the update @Adam-D-Lewis . You may be able to roughly use:

• https://github.com/Quansight/qhub-hpc/blob/main/templates/prometheus.yaml.j2#L38
• https://github.com/Quansight/qhub-hpc/blob/main/files/dashboards/traefik.json

[Adam-D-Lewis]

The following links are useful resources:

• https://www.metricfire.com/blog/traefik-and-prometheus-for-sites-monitoring/#Configure-Prometheus
• https://doc.traefik.io/traefik/observability/metrics/prometheus/
• [prometheus-kube-stack] additionalScrapeConfigs never updates prometheus-community/helm-charts#229

[Adam-D-Lewis]

The steps as I understand them now:

1. Configure traefik to output metrics for prometheus
2. Get prometheus to talk to traefik
3. Create or import dashboard showing Traefik stats

I've made a start on step 1 at https://github.com/Quansight/qhub/tree/prometheus_grafana.
I'm trying step 2 (manually) at https://github.com/Quansight/grafana-prometheus-qhub. It seems that prometheus is not able to get the metrics from traefik. I'm not sure if I need to add a service that routes prometheus to the traefik pod. Trying a few different things. After port forwarding port 9090 on the prometheus pod, I can see the Traefik target that I'm trying to add, but it's not working yet.

Also, I'd assume we don't want the metrics available publicly since then anyone could set up their own prometheus to poll metrics from your traefik instance, but how do I route the prometheus traffic to the Traefik pod itself when both prometheus and Traefik are within the cluster? Just use a ClusterIP k8s service, I would think, but I'll see if that works.

[Adam-D-Lewis]

From stackoverflow, For automatic discovery of a pod, this is needed in the kubernetes configuration:

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__

[Adam-D-Lewis]

I was able to get this working using a helm chart (see prom_values.yaml in https://github.com/Quansight/grafana-prometheus-qhub). Next, I just need to transfer the prom_values.yaml file into the terraform format in qhub/qhub/template/{{ cookiecutter.repo_directory }}/infrastructure/modules/kubernetes/services/monitoring/main.tf.

[Adam-D-Lewis]

Alright so the prometheus_grafana branch sets up traefik as a prometheus target correctly now. The dashboard still needs to be added.

[Adam-D-Lewis]

We can add a grafana dashboard via a configmap or a secret. Now I need to determine which dashboard we want added or make a sensible dashboard then define the configmap in terraform. I've committed a sample configmap to the prometheus_grafana branch to document the work so far.

[costrouc]

Thanks @Adam-D-Lewis. Would you mind turning this into a PR? I'd like to review it and be able to merge it before you get too busy with other projects.

[costrouc]

@tylerpotts schedule a time to sync up with @Adam-D-Lewis to pick up this issue.