Certificate expiry monitoring

[neilisfragile]

Sygnal should check the expiry date of its certificates and emit them as a metric which can trigger alerts.

[clokep]

This should be pretty straight forward, I've done this previously.

@neilisfragile: Was there any discussions about how we want the metric to be exported? An integer number of seconds is what comes to mind to me...? We could also export the raw time of expiration, but that seems less ideal.

[clokep]

Also, just to make sure I'm not missing anything -- it seems this is only applicable when using APNs with a certfile (not a keyfile).

[neilisfragile]

Sounds sane but @michaelkaye is the best person to ask.

[michaelkaye]

Raw time of expiration will be independent of delays / clock skew on the target machine, and is what (eg) blackbox exporter reports currently, as the metric probe_ssl_earliest_cert_expiry. You can probably also optimize somewhat by setting the value once on certificate read (as it won't change), and avoiding a background task that needs to wake up and update it every 60s.

We alert on a derived metric that is like so:

(max(probe_ssl_earliest_cert_expiry{ job="blackbox_http"}) by (instance) - time())/(60*60*24)

to get 'days to expiry' for our https used certificates

[michaelkaye]

i know it's a tiny task, but we do run it about 512,000 times between cert expiries ^^

It will still be useful the other way, but this is just a bit of context on other situations where we have this type of metric.

[richvdh]

so for clarity: you're suggesting we report expiry date, in seconds since the unix epoch?

[michaelkaye]

yes, to keep it in sync with similar things we have reporting.