"error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid"

[Tejaswini5327]

What happened?

We have 3 member etcd cluster. All of the 3 pods have expired certificates then we are getting certficate expired logs which is expected because ETCD has expired certificates. After sometime we are placing correct valid certificates in all of the pods of ETCD. But in all etcd pods , we still see the following error messages getting printed in logs.

{"level":"warn","ts":"2023-04-18T07:04:23.473Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.22.126:37830","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:04:23Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:04:52.552Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:40310","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:04:52Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:04:53.558Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:40376","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:04:53Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:07:28.609Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:17702","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:07:28Z is after 2023-04-18T07:04:06Z"}

Now we want to ask one question is, even when we place valid certificates in etcd pods still why certificate expiry error messages are coming in all of the etcd pods.

What did you expect to happen?

No certificate expiry log should be seen in any of the pods after renewal of certificates.

How can we reproduce it (as minimally and precisely as possible)?

The certificate error logs are printed even after the renewal of certificates.

Anything else we need to know?

No response

Etcd version (please run commands below)

bash-4.4$ etcd --version
etcd Version: 3.5.5
Git SHA: 19002cf
Go Version: go1.16.15
Go OS/Arch: linux/amd64
bash-4.4$ etcdctl version
etcdctl version: 3.5.5
API version: 3.5

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

{"level":"warn","ts":"2023-04-18T07:04:23.473Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.22.126:37830","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:04:23Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:04:52.552Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:40310","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:04:52Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:04:53.558Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:40376","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:04:53Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:07:28.609Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:17702","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:07:28Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:07:33.592Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00017c000/etcd.zragsai:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: remote error: keys: bad certificate\""}
Error: context deadline exceeded
{"level":"warn","ts":"2023-04-18T07:09:33.464Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.22.126:55086","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:09:33Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:09:58.606Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:45858","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:09:58Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:10:00.276Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:45886","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:10:00Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:12:34.638Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:7041","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:12:34Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:12:36.315Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:3746","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:12:36Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:12:38.620Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004d4000/etcd.zragsai:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: remote error: keys: bad certificate\""}
Error: context deadline exceeded
{"level":"warn","ts":"2023-04-18T07:14:27.885Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:22103","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:14:27Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:14:33.532Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.22.126:43640","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:14:33Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:15:01.408Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:41863","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:15:01Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:15:02.619Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:49646","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:15:02Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:15:03.624Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:49666","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:15:03Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:15:05.134Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:49680","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:15:05Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:15:21.173Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:41684","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:15:21Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:17:39.666Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:35661","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:17:39Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:17:40.976Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:16254","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:17:40Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:17:43.648Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000448540/etcd.zragsai:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: remote error: keys: bad certificate\""}
Error: context deadline exceeded
{"level":"warn","ts":"2023-04-18T07:19:43.528Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.22.126:32912","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:19:43Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:20:07.659Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:53268","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:20:07Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:20:10.242Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:53308","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:20:10Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:20:12.420Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:53322","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:20:12Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:22:48.675Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0003b0c40/etcd.zragsai:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: remote error: keys: bad certificate\""}
Error: context deadline exceeded
{"level":"info","ts":"2023-04-18T07:24:18.853Z","caller":"v3compactor/revision.go:86","msg":"starting auto revision compaction","revision":53,"revision-compaction-retention":100}
{"level":"info","ts":"2023-04-18T07:24:18.862Z","caller":"v3compactor/revision.go:94","msg":"completed auto revision compaction","revision":53,"revision-compaction-retention":100,"took":"9.075164ms"}
{"level":"info","ts":"2023-04-18T07:24:18.862Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":53}
{"level":"info","ts":"2023-04-18T07:24:18.862Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":53,"took":"166.172µs","hash":4251293922}
{"level":"info","ts":"2023-04-18T07:24:18.862Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":4251293922,"revision":53,"compact-revision":47}
{"level":"warn","ts":"2023-04-18T07:24:44.612Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.22.126:50586","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:24:44Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:24:46.137Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.22.126:50694","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:24:46Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:25:12.692Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.211.110:57364","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:25:12Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:27:10.000Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:12556","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:27:10Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:27:25.178Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:41664","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:27:25Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:27:40.259Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:1887","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:27:40Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:27:47.314Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:58811","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:27:47Z is after 2023-04-18T07:04:06Z"}
{"level":"warn","ts":"2023-04-18T07:27:48.788Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.117.148.205:29326","server-name":"etcd.zragsai","error":"keys: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2023-04-18T07:27:48Z is after 2023-04-18T07:04:06Z"}

[jmhbnz]

Hey @Tejaswini5327, thanks for raising this issue. To help us look into this are you able to provide the configuration you are using for etcd in relation to certificates and please briefly outline the procedure you used to copy the new certificates to the pods?

[prometheus-tao]

Hey @jmhbnz, I met other issue when kubeadm alpha certs renew all, Then I had copy the new certificates to the other two etcd node, See log had many tls connect error as fellow.

CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf May 23, 2024 01:45 UTC 364d no
apiserver May 23, 2024 01:45 UTC 364d ca no
apiserver-etcd-client May 23, 2024 01:45 UTC 364d etcd-ca no
apiserver-kubelet-client May 23, 2024 01:45 UTC 364d ca no
controller-manager.conf May 23, 2024 01:45 UTC 364d no
etcd-healthcheck-client May 23, 2024 01:45 UTC 364d etcd-ca no
etcd-peer May 23, 2024 01:45 UTC 364d etcd-ca no
etcd-server May 23, 2024 01:45 UTC 364d etcd-ca no
front-proxy-client May 23, 2024 01:45 UTC 364d front-proxy-ca no
scheduler.conf May 23, 2024 01:45 UTC 364d no

CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca May 20, 2032 10:20 UTC 8y no
etcd-ca May 20, 2032 10:20 UTC 8y no
front-proxy-ca May 20, 2032 10:20 UTC 8y no

[master node] 192.16.0.1 etcd log
2023-05-24 02:17:33.760448 I | embed: rejected connection from "192.16.0.3:41464" (error "tls: "192.16.0.3" does not match any of DNSNames ["host1" "localhost"]", ServerName "", IPAddresses ["192.16.0.1" "127.0.0.1" "::1"], DNSNames ["host1" "localhost"])
2023-05-24 02:17:37.498451 I | embed: rejected connection from "192.16.0.2:36448" (error "tls: "192.16.0.2" does not match any of DNSNames ["host1" "localhost"]", ServerName "", IPAddresses ["192.16.0.1" "127.0.0.1" "::1"], DNSNames ["host1" "localhost"])
2023-05-24 02:17:36.357622 W | rafthttp: health check for peer c6198c3c2a184417 could not connect: x509: certificate is valid for 192.16.0.1, 127.0.0.1, ::1, not 192.16.0.3
2023-05-24 02:17:36.359115 W | rafthttp: health check for peer cde1c9316d25ba89 could not connect: x509: certificate is valid for 192.16.0.1, 127.0.0.1, ::1, not 192.16.0.2
2023-05-24 02:17:51.359023 W | rafthttp: health check for peer c6198c3c2a184417 could not connect: dial tcp 192.16.0.3:2380: connect: connection refused
2023-05-24 02:17:51.359663 W | rafthttp: health check for peer cde1c9316d25ba89 could not connect: dial tcp 192.16.0.2:2380: connect: connection refused

[jmhbnz]

Closing this support issue as information requested above has not been provided, and there has been no updates for approximately a month.

@Tejaswini5327 for any future support requests please create a github discussion instead of a bug issue. Refer https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/triage_issues.md#support-requests

@prometheus-tao please create a separate github discussion for support, or a new bug report filling in the template if you believe you have found a bug, along with clear steps to reproduce it.