Add ADR for support for etcd s3 config secret

[brandond]

Proposed Changes

• Add ADR for etcd s3 config secret
• Add implementation

Types of Changes

new feature

Verification

See ADR

Testing

Linked Issues

• Parent issue for integration with Rancher - should switch to provisioning secret and passing env vars when restoring, instead of writing config to disk:
  RFE: Airgapped RKE2 etcd to AWS S3 via proxy rancher/rancher#44256
• Etcd s3 config secret support #10291

User-Facing Change

Further Comments

[cwayne18]

@jakefhyde

[jakefhyde]

I feel like this applies to both the standalone case and Rancher, but if the s3 credential is rotated and you restore using a snapshot with an outdated s3 credential, are we planning to do anything on that front? I'm mostly worried about this in the case where a restore is performed and everything looks fine, but snapshots aren't being uploaded correctly to s3 which isn't going to be immediately visible to a user until there is a problem (I'm expecting on the Rancher side we'll create some controller to sync the secret with the downstream cluster, so it may be a moot point, but just wanted to pick your brain on this). Stale credentials may get left over post-restore, but that's a much smaller concern IMO. Also curious on which form will have priority if both mechanisms are provided during the migratory period.

[brandond]

if the s3 credential is rotated and you restore using a snapshot with an outdated s3 credential, are we planning to do anything on that front?

I'm not sure that we have a good way to take care of that for the user. I think the user, or an external controller (ie rancher), would need to be responsible for updating anything that has drifted since the snapshot was taken - that would include auth secrets. Other things like ImagePullSecrets are probably affected by this as well.

which form will have priority if both mechanisms are provided during the migratory period.

in the ADR I suggested that cli/config should always take precedence over the Secret. I'll update the ADR to make this more clear, calling them "defaults" probably isn't the correct language.

• The Secret will provide default values; if S3 configuration is passed via CLI flags or configuration file, ALL fields set by the Secret will be ignored. Secret and CLI/config values will NOT be merged.

[brandond]

ADR has been updated to clarify intent.

[Oats87]

At a high level this seems fine -- I don't see any issues with trying to implement support for this with the planner; however, that is not to say that there might be implementation details we discover down the line after initial implementation.

If possible, if the K3s/RKE2 implementation could be shipped in a pre-release that we could test on the planner side to see how well it can integrate, I think that would be best to ensure we don't end up shipping a feature that we'll need to use workarounds in Rancher with.

[Oats87]

Still needs review by @jakefhyde

[brandond]

@snasovich I can't add you as a reviewer; I don't think you're in the correct groups, but would you mind taking a look as well?

[brandond]

Will merge this with just ADR, and add implementation in a future release cycle.