Add ADR for etcd s3 config secret

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

docs/adrs/etcd-s3-secret.md

@@ -0,0 +1,73 @@
+# Support etcd Snapshot Configuration via Kubernetes Secret
+
+Date: 2024-02-06
+
+## Status
+
+Accepted
+
+## Context
+
+### Current State
+
+K3s currently reads configuration for S3 storage of etcd snapshots from CLI flags and/or configuration files.
+
+Security-conscious users have raised issue with the current state. They want to store snapshots on S3, but do not want
+to have credentials visible in config files or systemd units. Users operating in highly secure environments have also
+asked for the ability to configure a proxy server to be used when creating/restoring snapshots stored on S3, without
+managing complicated `NO_PROXY` settings or affecting the rest of the K3s process environment.
+
+### Security Considerations
+
+Storing credentials on-disk is generally considered a bad idea, and is not allowed by security practices in many
+organizations. Use of static credentials in the config file also makes them difficult to rotate, as K3s only reloads the
+configuration on startup.
+
+### Existing Work
+
+Cloud-providers and other tools that need to auth to external systems frequently can be configured to retrieve secrets
+from an existing credential secret that is provisioned via an external process, such as a secrets management tool. This
+avoids embedding the credentials directly in the system configuration, chart values, and so on.
+
+## Decision
+
+* We will add support for reading etcd snapshot S3 configuration from a Secret named `k3s-etcd-snapshot-s3` in the
+  `kube-system` namespace.
+* The secret will ONLY be used for on-demand and scheduled snapshot operations.
+* The Secret will provide default values; if S3 configuration is passed via CLI flags or configuration file, ALL fields
+* set by the Secret will be ignored. Secret and CLI/config values will NOT be merged.
+* Snapshot restore operations that want to retrieve a snapshot from S3 will need to pass the appropriate configuration
+  via environment variables or CLI flags, as the Secret is not available during the restore process.
+* We will add a `--etcd-s3-proxy` flag that can be used to set the proxy used by the S3 client. This will override the
+  settings that golang's default HTTP client reads from the `HTTP_PROXY/HTTPS_PROXY/NO_PROXY` environment varibles.
+
+Fields within the Secret will match CLI flags / config file keys, with the `etcd-s3` prefix dropped.
+
+For the Endpoint Certificate Authority field (`etcd-s3-endpoint-ca` flag), which normally contains the path of a file on
+disk, the `endpoint-ca` field can specify an inline PEM-encoded CA bundle, or the `endpoint-ca-name` can be used to
+specify the name of a ConfigMap in the same namespace containing one or more CA bundles. All valid CA bundles found in
+either field are loaded.
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: k3s-etcd-snapshot-s3
+  namespace: kube-system
+stringData:
+  endpoint: ""
+  endpoint-ca: ""
+  endpoint-ca-name: ""
+  skip-ssl-verify: "false"
+  access-key: "AWS_ACCESS_KEY_ID"
+  secret-key: "AWS_SECRET_ACCESS_KEY"
+  bucket: "bucket"
+  region: "us-east-1"
+  insecure: "false"
+  timeout: "5m"
+  proxy: ""
+```
+
+## Consequences
+
+This will require additional documentation, tests, and QA work to validate use of secrets for s3 snapshot configuration.