TLS part 1: Enable TLS if cert is available

[sed-i]

Tracking issue: canonical/cos-lite-bundle#75

This PR enables TLS for alertmanager when a certificate is available.

This involves:

• adding tls-certificates to metadata
• writing private key and certificate to disk
• writing a web config file
• updating the pebble command with the web config file

This PR goes in tandem with canonical/observability-libs#49.

TODO:

• Scale to two units and make sure both units' TLS endpoints are reachable
• Remove relation and:
  • make sure TLS endpoint is NOT reachable
  • re-relate, and make sure TLS endpoint is reachable
• Confirm TLS endpoint still reachable after an upgrade sequence
• Think what to do with insecure_skip_verify in this line in alertmanager charm code: "global": {"http_config": {"tls_config": {"insecure_skip_verify": True}}},

[lucabello]

I've addressed some of the missing pieces:

• about the certificate files deletion, I've added it conditionally since container.push() overwrites the file anyway; this way we don't re-create certificates every time the configuration is updated
• the reload logic has been moved to a separate method so it can be called in the order you mentioned was correct

[lucabello]

CertManager has been changed to provide certificate-per-app (i.e., only the leader can generate a CSR); however, the previous scope was certificate-per-unit.

Are we saying that all units should share the same certificate?

[sed-i]

Tested successfully with:

# # Deploy bundle
# juju deploy --trust ./bundle.yaml
#
# # Obtain alertmanager IP address
# IPADDR=$(juju status --format json am | jq -r '.applications.am.units."am/0".address')
#
# # Make sure alertmanager web-external-hostname is locally routable
# echo "$IPADDR alertmanager.local" | sudo tee -a /etc/hosts
#
# # Make sure charm code created web-config, cert and key files
# juju ssh --container alertmanager am/0 ls /etc/alertmanager/
#
# # Inspect server cert and confirm `X509v3 Subject Alternative Name` field is as expected
# echo | openssl s_client -showcerts -servername $IPADDR:9093 -connect $IPADDR:9093 2>/dev/null | openssl x509 -inform pem -noout -text
#
# # Save CA cert locally
# juju show-unit am/0 --format yaml | yq '.am/0."relation-info"[0]."local-unit".data.ca' > /tmp/cacert.pem
#
# # Confirm alertmanager TLS endpoint reachable
# curl --fail-with-body --capath /tmp --cacert /tmp/cacert.pem https://alertmanager.local:9093/-/ready

---
bundle: kubernetes
applications:
  am:
    charm: ./alertmanager-k8s_ubuntu-20.04-amd64.charm
    series: focal
    scale: 1
    trust: true
    resources:
      alertmanager-image: ubuntu/prometheus-alertmanager:0.23-22.04_beta
    options:
      web_external_url: https://alertmanager.local
  ca:
    charm: self-signed-certificates
    channel: edge
    scale: 1
relations:
- [am:certificates, ca:certificates]

[sed-i]

Now testing with this bundle (with traefik):

# # Deploy bundle
# juju deploy --trust ./bundle.yaml
#
# # Obtain IP addresses
# IPADDR=$(juju status --format json trfk | jq -r '.applications.trfk.address')
# IPADDR0=$(juju status --format json am | jq -r '.applications.am.units."am/0".address')
# IPADDR1=$(juju status --format json am | jq -r '.applications.am.units."am/1".address')
# IPADDR2=$(juju status --format json am | jq -r '.applications.am.units."am/2".address')
#
# # Make sure traefik external-hostname is locally routable
# echo "$IPADDR cluster.local" | sudo tee -a /etc/hosts
#
# # Make sure charm code created web-config, cert and key files
# juju ssh --container alertmanager am/0 ls /etc/alertmanager/
#
# # Inspect server cert and confirm `X509v3 Subject Alternative Name` field is as expected
# echo | openssl s_client -showcerts -servername cluster.local -connect cluster.local 2>/dev/null | openssl x509 -inform pem -noout -text
#
# # Save CA cert locally
# juju show-unit am/0 --format yaml | yq '.am/0."relation-info"[0]."local-unit".data.ca' > /tmp/cacert.pem
#
# # Confirm traefik ingress has `https` for alertmanager's server url
# juju ssh --container traefik trfk/0 cat /opt/traefik/juju/juju_ingress_ingress_6_am.yaml
#
# # Confirm alertmanager TLS endpoint reachable
# curl --fail-with-body --capath /tmp --cacert /tmp/cacert.pem https://cluster.local/tlstest-am-0/-/ready

---
bundle: kubernetes
applications:
  am:
    charm: ./alertmanager-k8s_ubuntu-20.04-amd64.charm
    series: focal
    scale: 3
    trust: true
    resources:
      alertmanager-image: ubuntu/prometheus-alertmanager:0.23-22.04_beta
    options:
      web_external_url: https://alertmanager.local
  ca:
    charm: self-signed-certificates
    channel: edge
    scale: 1
  trfk:
    charm: traefik-k8s
    channel: edge
    scale: 1
    options:
      external_hostname: cluster.local
relations:
- [am:certificates, ca:certificates]
- [trfk:certificates, ca:certificates]
- [am:ingress, trfk]

[PietroPasotti]

It's too large to really pass with a fine comb, but what I see is good and ITWT (in testing we trust)