Concourse-Up deployments will stop working after 1 year

[danyoung]

EngineerBetter's own Concourse deployment stopped working over the weekend as the NATS TLS certificate that was created when the Director was first deployed expired on the 3rd of January 2019.

This didn't present a problem until the next time a NATS client (ie the BOSH agents on VMs) established a connection, at which point they couldn't connect. As the Health Manager couldn't communicate with the agents on the VMs via NATS, it decided to recreate them. Unfortunately it would recreate them with the same expired certificate, causing them to time out after 10 minutes, be destroyed, and then rescheduled for creation.

Unfortunately this will happen to every Concourse-Up deployment after a year, since BOSH-generated certs are hardcoded to this. We're currently looking at the problem and thank you for your patience while we implement a solution.

In the meantime, please check the age of your deployment using the info command and take steps to rotate the certs as described here: https://bosh.io/docs/nats-ca-rotation/. Please join us on Concourse-Up Slack to discuss further.

[crsimmons]

As of v0.18.1 there is now a built-in Concourse-Up command for renewing these NATS certificates.

concourse-up maintain --renew-nats-cert

Read more in the readme

[danyoung]

Closing this for now, since there is a preventative action available. Full automation of NATs cert renewal may be a future story.