cmd: Allow SYSTEM_SECRET key rotation

[aeneasr]

Key rotation is tricky because we either need a table lock for JWK or an overlord. Having an overlord who is attached to rethinkdb might be the smartest thing to do. There should never be more than one overlord running (how to ensure this? table row? ttl?).

The system secret key rotation definitely needs an overlord. Key exchange could done through HTTP:

• Poll last key creation and check TTL
• Generate new secret
• Gather a list of all running instances
• Authorize via access token
• Provide old secret
• Provide new secret

Alternatively the new secret could be encrypted with the old key and stored in the database. Everyone who has the old key, can read the new one as well. This would save the trouble of instance discovery as all instances are connected to rethinkdb anyways.

Using policies and IP ranges, access to the endpoint could be allowed to trusted addresses only.

On system key rotation, all JWKs must be re-encrypted using the new key. All tokens become invalid when the key changes which is why sysem key ttl must be larger than token ttl. This way we can be sure that all instances can validate tokens using either the old or the new key

[aeneasr]

Let me know if you have any ideas how to improve this. Best practices, specs and ideas alike :)

[aeneasr]

Instead of a flag, JWKs should have an exp claim.

[waynerobinson]

For standard, regular key rotation that are meant to prevent key abuse, I would recommend a period of time where the old and new keys could be used. Basically the maximum expiry period for anything that would've been encrypted or signed with the old key.

If an existing key has been compromised, it should be replaced instantly and anything encrypted with the old key should be considered unsafe anyway and discarded. Locking during the replacement process would be nice, but ultimately there is going to be failing messages at some point if you're not respecting the previous keys anyway.

[aeneasr]

JWK Rotation is now implemented by adding another key (pair) to the existing set.