health check with client-cert-auth enabled #13849

moonovo · 2022-03-29T11:12:09Z

What happened?

when client-cert-auth enabled, heath check failed.
logs:

{"level":"warn","ts":"2022-03-29T18:54:08.511Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:49502","server-name":"","error":"tls: client didn't provide a certificate"}

What did you expect to happen?

health check is successful.

How can we reproduce it (as minimally and precisely as possible)?

Set the ETCD_CLIENT_CERT_AUTH environment variable to true.

k8s livessnessProbe:

    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /health
        port: 61111
        scheme: HTTPS

      initialDelaySeconds: 10
      timeoutSeconds: 11
      periodSeconds: 10
      failureThreshold: 3
      successThreshold: 1

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here
etcd Version: 3.5.2
Git SHA: 99018a77b
Go Version: go1.16.3
Go OS/Arch: linux/arm64

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

paste your configuration here


    - name: ETCD_LOG_LEVEL
      value: "debug"
    - name: ETCD_QUOTA_BACKEND_BYTES
      value: "8589934592"
    - name: "ETCD_LISTEN_METRICS_URLS"
      value: "http://192.168.2.2:61113"
    - name: ETCD_TYPE
      value: "etcd-test"
    - name: ETCD_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: "v1"
          fieldPath: "metadata.name"
    - name: ETCD_CLUSTER_NAME
      value: "k8s-etcd-test-server"
    - name: ETCD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: "v1"
          fieldPath: "metadata.namespace"
    - name: ETCD_INITIAL_ADVERTISE_PEER_URLS
      value: "https://192.168.2.2:61112"
    - name: ETCD_LISTEN_PEER_URLS
      value: "https://192.168.2.2:61112"
    - name: ETCD_LISTEN_CLIENT_URLS
      value: "https://192.168.2.2:61111,https://127.0.0.1:61111"
    - name: ETCD_ADVERTISE_CLIENT_URLS
      value: "https://192.168.2.2:61111"
    - name: ETCD_INITIAL_CLUSTER_TOKEN
      value: "etcd-test-cluster-1"
    - name: ETCD_HEARTBEAT_INTERVAL
      value: "500"
    - name: ETCD_ELECTION_TIMEOUT
      value: "5000"
    - name: ETCD_TRUSTED_CA_FILE
      value: "/srv/kubernetes/ca.cer"
    - name: ETCD_CERT_FILE
      value: "/srv/kubernetes/common_server.cer"
    - name: ETCD_KEY_FILE
      value: "/srv/kubernetes/common_server_key_de.pem"
    - name: ETCD_CLIENT_CERT_AUTH
      value: "true"
    - name: ETCD_PEER_CERT_FILE
      value: "/srv/kubernetes/common_server.cer"
    - name: ETCD_PEER_KEY_FILE
      value: "/srv/kubernetes/common_server_key_de.pem"
    - name: ETCD_PEER_CLIENT_CERT_AUTH
      value: "true"
    - name: ETCD_PEER_TRUSTED_CA_FILE
      value: "/srv/kubernetes/ca.cer"
    - name: ETCD_PEER_CRL_FILE
      value: "/srv/kubernetes/paas.crl"
    - name: ETCD_CLIENT_CRL_FILE
      value: "/srv/kubernetes/paas.crl"
    - name: ETCD_ENABLE_PPROF
      value: "true"
    - name: ETCD_ENABLE_V2
      value: "false"
    - name: ETCD_LOGGER
      value: "zap"
    - name: ETCD_BACKEND_BATCH_INTERVAL 
      value: "1s"
    - name: ETCD_EXPERIMENTAL_BACKEND_BBOLT_FREELIST_TYPE
      value: "map"
    - name: ETCD_PRE_VOTE
      value: "true"
    - name: ETCD_EXPERIMENTAL_WARNING_APPLY_DURATION
      value: "300ms"

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] parsed scheme: \"\""}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] scheme \"\" not registered, fallback to default scheme"}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] ccResolverWrapper: sending update to cc: {[{127.0.0.1:61111  <nil> 0 <nil>}] <nil> <nil>}"}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] ClientConn switching balancer to \"pick_first\""}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] parsed scheme: \"\""}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] scheme \"\" not registered, fallback to default scheme"}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] ccResolverWrapper: sending update to cc: {[{192.168.2.2:61111  <nil> 0 <nil>}] <nil> <nil>}"}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] ClientConn switching balancer to \"pick_first\""}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Channel switches to new LB policy \"pick_first\""}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Subchannel Connectivity change to CONNECTING"}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Channel switches to new LB policy \"pick_first\""}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Subchannel Connectivity change to CONNECTING"}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] pickfirstBalancer: UpdateSubConnState: 0x40001ad750, {CONNECTING <nil>}"}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Channel Connectivity change to CONNECTING"}
{"level":"info","ts":"2022-03-29T19:03:12.789Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Subchannel picks a new address \"192.168.2.2:61111\" to connect"}
{"level":"info","ts":"2022-03-29T19:03:12.790Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] pickfirstBalancer: UpdateSubConnState: 0x40005a40f0, {CONNECTING <nil>}"}
{"level":"info","ts":"2022-03-29T19:03:12.790Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Channel Connectivity change to CONNECTING"}
{"level":"info","ts":"2022-03-29T19:03:12.790Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Subchannel picks a new address \"127.0.0.1:61111\" to connect"}
{"level":"info","ts":"2022-03-29T19:03:12.790Z","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"127.0.0.1:61111"}
{"level":"info","ts":"2022-03-29T19:03:12.790Z","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"192.168.2.2:61111"}
{"level":"info","ts":"2022-03-29T19:03:12.824Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Subchannel Connectivity change to READY"}
{"level":"info","ts":"2022-03-29T19:03:12.824Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Subchannel Connectivity change to READY"}
{"level":"info","ts":"2022-03-29T19:03:12.824Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] pickfirstBalancer: UpdateSubConnState: 0x40001ad750, {READY <nil>}"}
{"level":"info","ts":"2022-03-29T19:03:12.824Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] pickfirstBalancer: UpdateSubConnState: 0x40005a40f0, {READY <nil>}"}
{"level":"info","ts":"2022-03-29T19:03:12.824Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Channel Connectivity change to READY"}
{"level":"info","ts":"2022-03-29T19:03:12.824Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Channel Connectivity change to READY"}
{"level":"warn","ts":"2022-03-29T19:03:19.864Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:38006","server-name":"","error":"tls: client didn't provide a certificate"}
{"level":"warn","ts":"2022-03-29T19:03:20.699Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.2.2:60734","server-name":"","error":"EOF"}
{"level":"warn","ts":"2022-03-29T19:03:20.699Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.2.2:60736","server-name":"","error":"EOF"}
{"level":"warn","ts":"2022-03-29T19:03:29.865Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:44964","server-name":"","error":"tls: client didn't provide a certificate"}
{"level":"warn","ts":"2022-03-29T19:03:39.865Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:45254","server-name":"","error":"tls: client didn't provide a certificate"}
{"level":"warn","ts":"2022-03-29T19:03:40.699Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.2.2:39744","server-name":"","error":"EOF"}

The text was updated successfully, but these errors were encountered:

serathius · 2022-03-29T11:49:18Z

As I mentioned in #13706 (comment), the goal of client-cert-auth is to require client certificates.

moonovo · 2022-03-29T12:04:20Z

Yes, want to use

    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /health
        port: 61111
        scheme: HTTPS

instead

    livenessProbe:
      exec:
        command:
        - "/bin/sh"
        - "-ec"
        - "ETCDCTL_API=3 etcdctl --cacert /srv/kubernetes/ca.cer --cert /srv/kubernetes/server.cer --key /srv/kubernetes/server.key --endpoints https://127.0.0.1:61111--command-timeout=10s get --keys-only=true --consistency=s / "

stale · 2022-07-10T13:01:14Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

moonovo added the type/bug label Mar 29, 2022

moonovo changed the title ~~health check with client-cert-auth enabled~~ health check failed with client-cert-auth enabled Mar 29, 2022

moonovo mentioned this issue Mar 29, 2022

[3.5] enhance health check endpoint to support serializable request #13706

Merged

serathius added type/feature and removed type/bug labels Mar 29, 2022

serathius changed the title ~~health check failed with client-cert-auth enabled~~ health check with client-cert-auth enabled Mar 29, 2022

stale bot added the stale label Jul 10, 2022

stale bot closed this as completed Oct 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

health check with client-cert-auth enabled #13849

health check with client-cert-auth enabled #13849

moonovo commented Mar 29, 2022 •

edited

Loading

paste your configuration here

serathius commented Mar 29, 2022

moonovo commented Mar 29, 2022 •

edited

Loading

stale bot commented Jul 10, 2022

health check with client-cert-auth enabled #13849

health check with client-cert-auth enabled #13849

Comments

moonovo commented Mar 29, 2022 • edited Loading

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Etcd version (please run commands below)

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

Relevant log output

serathius commented Mar 29, 2022

moonovo commented Mar 29, 2022 • edited Loading

stale bot commented Jul 10, 2022

moonovo commented Mar 29, 2022 •

edited

Loading

moonovo commented Mar 29, 2022 •

edited

Loading