etcd snapshot does x509 host name verification even when skip verify is true

[braunsonm]

Environmental Info:
RKE2 Version: v1.21.4 rke2r3

Describe the bug:
Even when passing --s3-skip-ssl-verify to etcd-snapshot the CLI will attempt to do host name verification on the certificate presented. In our case this is not matching so the verification fails.

Steps To Reproduce:

• Have an S3 with a self signed cert which does not have a valid host name
• Attempt to do a snapshot with --s3-skip-ssl-verify

Expected behavior:
The backup should complete since we intentionally skip SSL verification.

Actual behavior:
The backup fails with the error:

x509: certificate is not valid for any names, but wanted to match ...

[brandond]

Hey @briandowns it looks like we only pass through the EtcdS3SkipSSLVerify flag if EtcdS3EndpointCA is not empty, which seems like a bug - you probably wouldn't configure a custom CA if you wanted to disable certificate verification.

https://github.com/k3s-io/k3s/blob/bfb3d9b19d174fdd3feaf106fe23a9e8e6c4c86b/pkg/etcd/s3.go#L39-L40

[braunsonm]

Passing in a random CA file works around the issue and allows skip verify to work. Thanks.

[rancher-max]

Validated on master branch commit 8821dc49569c383fd158ae3990eebfe91285acd8

I saw this fail on the mentioned version and see it now working. It should be working on the backport issues as well, but I will validate those on their own issues that have already been linked above.

$ mc ls --insecure main/bucket1/newsnap
[2021-10-12 19:25:39 UTC] 4.6MiB on-demand-ip-172-31-26-137-1634066738

My steps:
On a VM within the network:

1. Generate private certs:

$ wget -O certgen https://github.com/minio/certgen/releases/download/v0.0.2/certgen-linux-amd64
$ chmod +x certgen 
$ ./certgen  -ca -host "<my ip or anything>"

2. Create minio instance:

$ mkdir -p /tmp/.minio/certs/ && cp public.crt /tmp/.minio/certs/ && cp private.key /tmp/.minio/certs/
$ mkdir -p /tmp/minio/data

$ cat <<EOF > docker-compose.yaml
version: '3'

services:
  minio:
    image: minio/minio
    command: server --address ":443" /data
    ports:
      - "443:443"
    environment:
      MINIO_ACCESS_KEY: "adminexample"
      MINIO_SECRET_KEY: "adminexample"
    volumes:
      - /tmp/minio/data:/data
      - /tmp/.minio:/root/.minio
EOF

$ docker-compose up -d

3. Using the mc tool, add a bucket to the minio s3 instance:

$ curl https://dl.min.io/client/mc/release/linux-amd64/mc   --create-dirs   -o $HOME/minio-binaries/mc
$ chmod +x $HOME/minio-binaries/mc
$ export PATH=$PATH:$HOME/minio-binaries/

$ mc alias set main <vm ip> adminexample adminexample
$ mc admin info --insecure main
$ mc mb --insecure main/bucket1
$ mc ls --insecure main

On another VM:

1. Install rke2
2. Take a snapshot:

# On previous version, saw this:
$ sudo rke2 etcd-snapshot --s3 --s3-skip-ssl-verify --s3-endpoint=<vm ip from above> --s3-bucket=bucket1 --s3-folder=newsnap --s3-access-key=adminexample --s3-secret-key=adminexample
...
INFO[0000] Checking if S3 bucket bucket1 exists        
FATA[0004] Head "https://<vm ip from above>/bucket1/": x509: certificate signed by unknown authority 

# Now, see this:
$ sudo rke2 etcd-snapshot --s3 --s3-skip-ssl-verify --s3-endpoint=<vm ip from above> --s3-bucket=bucket1 --s3-folder=newsnap --s3-access-key=adminexample --s3-secret-key=adminexample
...
INFO[0000] Checking if S3 bucket bucket1 exists         
INFO[0000] S3 bucket bucket1 exists                     
INFO[0000] S3 upload complete for on-demand-ip-<redacted>-1634066738 
INFO[0000] Saving current etcd snapshot set to rke2-etcd-snapshots ConfigMap