No alert even when certificate is about to expire nor a target is down

[nobuto-m]

Bug Description

There is no alert defined by the charm for some important or critical events.

• certification is about to expire (e.g. warning when < 30 days, critical when < 7 days)
• a target is down

Alerts are always happy even with the events above.

For the certificate expiration part, we need something like this:
https://grafana.com/blog/2020/11/25/how-we-eliminated-service-outages-from-certificate-expired-by-setting-up-alerts-with-grafana-and-prometheus/#alerting

To Reproduce

$ juju deploy blackbox-exporter-k8s blackbox-exporter --channel latest/edge

$ juju config blackbox-exporter probes_file="
scrape_configs:
  - job_name: 'blackbox_k8s_endpoints'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://ubuntu.com
        - https://ubuntu.com
        - https://releases.ubuntu.com
        - https://non-existent.canonical.com/
"

$ juju relate prometheus:metrics-endpoint blackbox-exporter:self-metrics-endpoint
$ juju relate grafana:grafana-dashboard blackbox-exporter:grafana-dashboard

Environment

blackbox-exporter-k8s latest/edge 3

Relevant log output

N/A

Additional context

No response

[nobuto-m]

The context:
https://bugs.launchpad.net/charm-kubernetes-master/+bug/2045009

[nobuto-m]

This should be fixed in this blackbox exporter charm rather than other charms.
ref: charmed-kubernetes/charm-kubernetes-control-plane#320

[lucabello]

Just to clarify after reading all the context, this issue is about adding one alert rule for certificate expiration to the ones bundled with the Blackbox exporter charm, along the lines of this one.

As a sidenote clarification, by default when the alert fires the only way you'll be able to identify which endpoint it refers to, is through the probe_target label, which will show the endpoint that has been probed.

[nobuto-m]

@lucabello You implemented an alert for cert expiration, which is good.

There is no alert defined by the charm for some important or critical events.

• certification is about to expire (e.g. warning when < 30 days, critical when < 7 days)
• a target is down

Would you mind adding the second alert for when a target is down or do you need a separate bug report?