Add support for credential rotation for Zero Downtime Deployment

[thomasdarimont]

Currently when a JGroups stack uses AUTH or Encrypt stack elements, then the configured password or keystores are the sole credentials that are used to authenticate or encrypt jgroups traffic.

• AUTH.java
• Encrypt.java

If credentials need to be rotated, then old cluster members cannot communicate with the new members - because the old members don't know the new credentials. This my cause service interruptions and makes it hard to implement zero-downtime requirements.

Idea:
In order to support credential rotation, jgroups could support a "backup" Keystore or credentials / key / password that could be used to provide the "old" password / keystore to new jgroups members, for the time of the migration, such that new members could catch up with old members. Once the new credentails are available to all new members, the backup keystore / auth key / passwords could be removed again from the configuration.

[belaban]

First off, AUTH is not recommended; see [1] ("Problems with AUTH").

Secondly, SYM_ENCRYPT relies on a static keystore. If keys are changed (in all keystores), then members need to be restarted.
If you want to be able to provide dynamic key dissemination in a cluster, including changing the shared group key when a member leaves, then use the ASYM_ENCRYPT / SS_KEY_EXCHANGE combo.

If this doesn't work for you, would adding a function to reload keys to SYM_ENCRYPT work for you? This would allow for a keystore to be changed, and then the members would be signalled (e.g. via probe) to reload the keys from the keystore.

[1] http://www.jgroups.org/manual5/index.html#AUTH
[2] https://issues.redhat.com/browse/JGRP-2667