[ENH] - Adding Conda-Store, JupyterHub, Traefik, Keycloak Grafana Dashboards

[costrouc]

Feature description

Currently the Grafana dashboards for these services are already added but the metrics are not being fetched. In order to fix this we need to add the prometheus scraping operators to get the configuration. Simlar to the example here. The catch is that this requires a custom resource definition. CRDs cannot be used and defined within the same step hashicorp/terraform-provider-kubernetes#1367. Unless this issue is resolved this is complex to solve.

The most promising plan would be to use the kubectl provider

Value and/or benefit

Users can see monitoring from all their services.

Anything else?

No response

[Adam-D-Lewis]

Adding a note in case it's useful. Though a CRD sounds like a potentially better solution, we did have a workaround in the Prometheus config in the past (https://github.com/Quansight/qhub/pull/797/files) (see monitoring/main.tf)

[Adam-D-Lewis]

@costrouc When adding kubectl provider, we'll need to add credentials, right? e.g.

provider "kubectl" {
  host                   = var.eks_cluster_endpoint
  cluster_ca_certificate = base64decode(var.eks_cluster_ca)
  token                  = data.aws_eks_cluster_auth.main.token
  load_config_file       = false
}

I see these outputs are output in stage 2, but we don't have a way to get them from stage 2 to stage 7, do we?

[costrouc]

@Adam-D-Lewis this is done via https://github.com/Quansight/qhub/blob/main/qhub/stages/tf_objects.py#L27-L64 and dynamically generated and added to each stage in https://github.com/Quansight/qhub/blob/main/qhub/stages/tf_objects.py#L183-L250. I would like to add this support though.

If you were to develop this I'd like to have it in two PRs:

• one for adding the kubectl provider to each stage which uses kubernetes
• one for adding the dashboards via kubectl