etcd clientv3 doesn't renew the auth token

[szmgfcsp]

The etcd clientv3 doesn't renew the auth token when it expires, it just fails with invalid auth token. It happens with both simple and jwt token types.

Steps to reproduce:

Start etcd with etcd --auth-token simple --auth-token-ttl 5
Enable authentication
etcdctl user add root
etcdctl user grant-role root root
etcdctl auth enable

Verify that it works

etcdctl --user root:root put test test
etcdctl --user root:root get test
Run the following Go program

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"
	"google.golang.org/grpc"
)

func main() {
	client, err := clientv3.New(clientv3.Config{
		Context:   context.Background(),
		Endpoints: []string{"127.0.0.1:2379"},
		Username:  "root",
		Password:  "root",
		DialOptions: []grpc.DialOption{
			grpc.WithBlock(),
		},
	})
	if err != nil {
		log.Fatal(err)
	}

	resp, err := client.Get(context.Background(), "test")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(resp.Kvs)

	time.Sleep(6 * time.Second)

	_, err = client.Get(context.Background(), "test")
	if err != nil {
		log.Fatal(err)
	}
}

output:

[key:"test" create_revision:2 mod_revision:2 version:1 value:"test" ]
{"level":"warn","ts":"2021-10-11T11:38:59.474+0200","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001ef180/#initially=[127.0.0.1:2379]","attempt":0,"error":"rpc error: code = Unauthenticated desc = **etcdserver: invalid auth token**"}

As you can see, the first client.Get succeeded in fetching the test key, but the second one fails with invalid auth token.

Am I missing something simple here?

EDIT: If you make 10 requests while sleeping for 1 second in-between, the error does not show up.

for i := 1; i <= 10; i++ {
		time.Sleep(1 * time.Second)
                // no error
		resp, err := client.Get(context.Background(), "test")
		if err != nil {
			log.Fatal(err)
		}

		fmt.Println(resp.Kvs)
	}

	time.Sleep(6 * time.Second)
        // return invalid auth token error
	_, err = client.Get(context.Background(), "test")
	if err != nil {
		log.Fatal(err)
	}

[szmgfcsp]

问题好像出现在go-zero内部定时轮询ETCD时，token过期了服务器上不存在token了，但还是使用之前的token，从而导致服务瘫痪

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The problem seems to occur when go-zero polls ETCD regularly, the token expires and there is no token on the server, but the previous token is still used, resulting in service paralysis

[kevwan]

I'm wondering if your code has anything related to go-zero?

[re-dylan]

It's etcd bug， please see issues

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.