Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd can't serve serializable read request when the quorum isn't satisfied #13524

Closed
ahrtr opened this issue Dec 8, 2021 · 0 comments · Fixed by #13525
Closed

etcd can't serve serializable read request when the quorum isn't satisfied #13524

ahrtr opened this issue Dec 8, 2021 · 0 comments · Fixed by #13525
Milestone

Comments

@ahrtr
Copy link
Member

ahrtr commented Dec 8, 2021

This issue applies to both 3.5 and the 3.6 (the main branch). I do not get time to verify previous versions.

Steps to reproduce this issue:

  1. Start an etcd cluster with 3 members, and add some K/V data;
  2. Stop all the members;
  3. Start only one member;
  4. Try to issue a serializable request using command something like etcdctl --endpoints http://127.0.0.1:32379 get k1 --consistency="s", then you will get the error like below,
$ etcdctl --endpoints http://127.0.0.1:32379 get k1 --consistency="s"
{"level":"warn","ts":"2021-12-09T06:18:40.477+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00000a3c0/127.0.0.1:32379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: context deadline exceeded

The root cause of this issue is that etcd waits for the ready notification infinitely on startup, please see serve.go#L96. But when the quorum isn't satisfied, the etcd instance will never get the notification.

The etcd instance should NOT wait for the ready notification infinitely; otherwise, it will never be able to serve serializable read request when the quorum isn't satisfied.

I will submit a PR soon, and any feedback is appreciated.

@serathius @xiang90 @gyuho @ptabor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

2 participants