Generating identitcal JWT token and authenticating results in server go panic

[Shambuwu]

What happened?

Authenticating with a JWT token that has been generated with the same payload en keypair as the server results in the server getting a go panic and shutting down.

panic: interface conversion: interface {} is nil, not float64

Authenticating with a JWT token that the server generated does work however, even though the signatures of both tokens get verified with the same public key.

What did you expect to happen?

I expect an identitcal JWT token to properly authenticate the client, this does not happen however and completely shuts down the ETCD server.

How can we reproduce it (as minimally and precisely as possible)?

Generate and identical JWT token to one the server generates with a given keypair. Use this to authenticate a client.

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --versionetcd 
Version: 3.5.5
Git SHA: 19002cfc6
Go Version: go1.16.15
Go OS/Arch: linux/amd64


$ etcdctl version
etcdctl version: 3.5.5
API version: 3.5

Etcd configuration (command line flags or environment variables)

etcd --auth-token=jwt,pub-key=/jwt/public.pem,priv-key=/jwt/private.pem,sign-method=RS256

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
+------------------+---------+---------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME   |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+---------+-----------------------+-----------------------+------------+
| 8e9e05c52164694d | started | default | http://localhost:2380 | http://127.0.0.1:2379 |      false |
+------------------+---------+---------+-----------------------+-----------------------+------------+


$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

panic: interface conversion: interface {} is nil, not float64

goroutine 79 [running]:
go.etcd.io/etcd/server/v3/auth.(*tokenJWT).info(0xc000576740, 0x13ba798, 0xc000013170, 0xc000736000, 0x3f9, 0x5, 0x0, 0xc00050cf10)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/auth/jwt.go:78 +0x6aa
go.etcd.io/etcd/server/v3/auth.(*authStore).authInfoFromToken(0xc00062a100, 0x13ba798, 0xc000013170, 0xc000736000, 0x3f9, 0x1000000000001, 0x0)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/auth/store.go:772 +0x64
go.etcd.io/etcd/server/v3/auth.(*authStore).AuthInfoFromCtx(0xc00062a100, 0x13ba798, 0xc000013170, 0x1388ae0, 0x11c56a0, 0xc000575180)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/auth/store.go:1173 +0xf5
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).AuthInfoFromCtx(0xc000571100, 0x13ba798, 0xc000013170, 0x0, 0xc0006c0000, 0x0)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/v3_server.go:915 +0x52
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).doSerialize(0xc000571100, 0x13ba798, 0xc000013170, 0xc00055b5e0, 0xc00055b5f8, 0xc000667e00, 0x0)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/v3_server.go:646 +0x85
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).Range(0xc000571100, 0x13ba798, 0xc000013170, 0xc00067a6c0, 0x0, 0x0, 0x0)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/v3_server.go:132 +0x445
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*kvServer).Range(0xc000026300, 0x13ba798, 0xc000013020, 0xc00067a6c0, 0x110fe20, 0x0, 0x13bd2b8)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/api/v3rpc/key.go:46 +0x79
go.etcd.io/etcd/api/v3/etcdserverpb._KV_Range_Handler.func1(0x13ba798, 0xc000013020, 0x11efaa0, 0xc00067a6c0, 0x16, 0xc000667c20, 0x2, 0xc0004b3818)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/api/etcdserverpb/rpc.pb.go:6563 +0x89
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1(0x13ba798, 0xc000013020, 0x11efaa0, 0xc00067a6c0, 0xc00003a960, 0xc000500b28, 0x0, 0x0, 0x0, 0xc00050cc50)
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0xb0
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x13ba798, 0xc000013020, 0x11efaa0, 0xc00067a6c0, 0x1ae3e40, 0xc0001c9b00, 0x8b2fac, 0x13a07a0)
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x63
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.newUnaryInterceptor.func1(0x13ba798, 0xc000013020, 0x11efaa0, 0xc00067a6c0, 0xc00003a960, 0xc00003a9e0, 0x493be6, 0x6396f5b0, 0x1cc74c05, 0x32f5b6dfa0a)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/api/v3rpc/interceptor.go:71 +0x139
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x13ba798, 0xc000013020, 0x11efaa0, 0xc00067a6c0, 0x0, 0x203000, 0x13ca7b0, 0xc00007de00)
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x63
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1(0x13ba798, 0xc000013020, 0x11efaa0, 0xc00067a6c0, 0xc00003a960, 0xc00003aa00, 0x0, 0x0, 0x0, 0x0)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/api/v3rpc/interceptor.go:78 +0xd9
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x13ba798, 0xc000013020, 0x11efaa0, 0xc00067a6c0, 0xc0006c0000, 0x0, 0xc0004b3b30, 0x40e318)
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x63
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x13ba798, 0xc000013020, 0x11efaa0, 0xc00067a6c0, 0xc00003a960, 0xc000500b28, 0xc0004b3ba0, 0x573f66, 0x116d020, 0xc000013020)
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xd7
go.etcd.io/etcd/api/v3/etcdserverpb._KV_Range_Handler(0x110fe20, 0xc000026340, 0x13ba798, 0xc000013020, 0xc0001c9aa0, 0xc00002a1e0, 0x13ba798, 0xc000013020, 0xc00065e348, 0x15)
   /tmp/etcd-release-3.5.5/etcd/release/etcd/api/etcdserverpb/rpc.pb.go:6565 +0x150
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0005a2e00, 0x13c6c38, 0xc0006c2f00, 0xc000016480, 0xc00002a330, 0x1a9dbe0, 0x0, 0x0, 0x0)
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0005a2e00, 0x13c6c38, 0xc0006c2f00, 0xc000016480, 0x0)
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0005f60f0, 0xc0005a2e00, 0x13c6c38, 0xc0006c2f00, 0xc000016480)
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
   /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:921 +0x1fd

[mitake]

@Shambuwu hi, could you provide a way to reproduce the issue? I think you attached the special token to a request in an adhoc way?

[Shambuwu]

Start an ETCD server with a given keypair:

etcd --auth-token=jwt,pub-key=/jwt/public.pem,priv-key=/jwt/private.pem,sign-method=RS256

Using this same keypair, generate a JWT token with an identitical payload to the one the server returns when querying the auth/authenticate endpoint with a username/password.

Add this JWT token to your request headers. In my case this ends up with the server having a go panic.

[Shambuwu]

Also, the server actually succeeds in authenticating the user, the panic occurs after.
{"level":"debug","ts":"2022-12-15T10:18:05.357Z","caller":"auth/store.go:317","msg":"authenticated a user","user-name":"root","token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NzExMDAwMjcsInJldmlzaW9uIjo2LCJ1c2VybmFtZSI6InJvb3QifQ.MVHHN0GXfiibuUfO7rR3W73MwOSFt-1Ji9ZHEfGHSXkPeI_U_NiSywm1QNjrHNHgcyoUIX3XIDcOJ9L568imn2viie6dQEibzWL_akONrmQb7Ly4Q_1feZDK2w9BhZB3HyjyqgsLj-3fTJZVTu_Eu_-EgX8biquJxV8yK1h7kv7h386S33ohVvWyBqQFSsYS4iBBMkqmFVMWQfLxxkjag2GtCqbApjbCOxowFJDZs2axGveOIH26OY5qxFNvrW_epNXXy8bSqVz3GqDw2HPJra9yzYoKOzUhw_OIuY38ZJrQtUAF9ndx74HoVFbdUgXjc-TmBKoli9_XGJpU1l8H5kgFdqsI4lejPzRSyNBhTT8m5wc9WC27bQMouQqbUaMhe7fv4zFumHGpD787c-xnPajE-ifYwpNSpCWA5_Iu6D_SXWB78or79PMo8kViKdJMUK7P8UolinbtL8SWbn4E5lUAiFzq6YsqJNxDuH57rDOwnZXtnM9P_iOj2JMkG-CDydozAUSpV9jMQaARUy6Q7Og6UcZzmcd_nkq8yM6eJZdCs0oOpfFYIp9QHk-Gqw-jYcShptlMGdfSJVWRXHEOOrqoeEiRk7-XKQH17IAcVYTkovlsRgSDv6RXcTWCDCqm5Xtwr16lGGDKIizT9aOLWYs3brK28mLZrZR2WX7zuOw"}

panic: interface conversion: interface {} is string, not float64

[ahrtr]

Using this same keypair, generate a JWT token with an identitical payload

Please decode the token using an online tool, such as https://jwt.io/. Based on the error, there should be no field "revsion".

But of course, we should add protection to prevent etcdserver from panicking. So please anyone feel free to deliver a PR on this.

[mitake]

@Shambuwu I don’t think this is a realistic issue. If you own a key pair (and etcd cluster itself), you can do it but I don’t recommend to generate tokens outside of etcd… Do you have any reasons to do it? Anyway preventing panic is better though.

[serathius]

Moving backport labels to PR instead of issue.