Expose prometheus with basic auth

[yuvipanda]

This is the beginning of implementing idea 1 from the
list @georiganaelena made in
#328 (comment).

We have one prometheus running per cluster, but manage many clusters.
A single grafana that can connect to all such prometheus clusters
will help with monitoring as well as reporting. So we need to expose
it as securely as possible to the external world, as it can contain
private information.

In this case, we're using https + basic auth provided by
nginx-ingress
(https://kubernetes.github.io/ingress-nginx/examples/auth/basic/)
to safely expose prometheus to the outside world. We can then
use a grafana that knows these username / passwords to access this
prometheus instance. Each cluster needs its own username / password
(generated with pwgen 64 1), so users in one cluster can not access
prometheus for another cluster.

Ref #328

[yuvipanda]

I'm manually generating and check-ing in the username / password for each prometheus instead of autogenerating it in the deployer, as I sense that's the preferred approach now based on conversations with @consideRatio and @sgibson91. Keeps config explicit.

TODO:

• Do this for all clusters
• Document this as part of setting up the cluster

/cc @GeorgianaElena as well.

[yuvipanda]

I used a tiny helper script to generate these encrypted secret files:

#!/bin/bash
set -euo pipefail

F="${1}/enc-support.secret.yaml"
echo "prometheusAuth:" > $F
echo "  username: $(pwgen 64 1)" >> $F
echo "  password: $(pwgen 64 1)" >> $F
sops -i -e $F
echo "Done $F"%   

Then ran find . -type d | xargs -L1 ./gen.bash to run it for all dirs inside config/clusters.

However, meom-ige and farallon still don't have a support cluster, so this can't be used there yet.

[yuvipanda]

Note - I've already run deploy-support for all these :)

[consideRatio]

Hmmm, the support chart could also do the trick jupyterhub does and generate secrets in a Secret file - if prometheus could accept accessing the content from there.

It may not be what makes sense, just floating the idea to ensure it is considered as an option we rule out. Someplace, the grafana server would need to have access to those secrets in other clusters so it would probably be relevant to have access to those centrally still without probing multiple clusters whenever they are needed.

[yuvipanda]

@consideRatio yeah, that's the other option - but that would require giving kubernetes API access to every single cluster to the centralized grafana, which I would very much prefer to not do.

[consideRatio]

This is a nice beginning!

My understanding summarized

1. We assume we have prometheus subdomains for each cluster point to an nginx-ingress controllers in each cluster.
2. Letting the support chart create a k8s Secret to hold username/password for prometheus access via support chart values.
3. Letting the support chart's dependent prometheus Helm chart create an k8s Ingress resource that references the created k8s Secret with username/password. This reference is a annotation that an nginx-ingress controller understands: we wish to require basic authentication using the provided username/password in the k8s Secret
4. Later we manually (?) configure a central Grafana instance through its web UI to have many data sources, one for each cluster. Following this, we can allow dashboards there to choose to inspect any given cluster or have some dashboards that summarize the state of all clusters.

A few action points

• Add a k8s Secret template in the support chat
• Do work to ensure we get the relevant new sub-domain names get CNAME entries in the DNS servers we control, I figure they should point to the primary domain name we use which also points to the ingress controllers IP.
• Point out a specific Grafana instance to be the central instance we use (the one in the 2i2c cluster right?)

[consideRatio]

This LGTM, if needed, lets iterate from here to not slow down too much!

I think the key points now with relation to this PR is that this goes with the creation of additional domain names etc and those aspects may need documentation updates.

[yuvipanda]

I've added docs on what needs to happen with DNS, and I think it's already happened for all the clusters mentioned here.

[consideRatio]

With these comments resolved, this LGTM! Nice work on this @yuvipanda!

[yuvipanda]

I think what's left here is to automate adding all these as a data source to one centralized grafana.

[yuvipanda]

I think this is ready to go!

[yuvipanda]

Still need to write code that'll populate central grafana with datasources, but let's get this merged until then.

[damianavila]

Still need to write code that'll populate central grafana with datasources

I think the upcoming work is actually described here: #328 (comment), let me know if you disagree.

Btw, thanks for all this work, @yuvipanda!!