Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDC print lots of "etcd client outCh blocking too long, the etcdWorker may be stuck" and checkpoint lag metric abnormal #11884

Closed
fubinzh opened this issue Dec 16, 2024 · 4 comments · Fixed by #11886
Assignees
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug. type/regression

Comments

@fubinzh
Copy link

fubinzh commented Dec 16, 2024

What did you do?

  1. create cdc changefeed
  2. run workload (use lighting tidb backend to import 10+ tables )

What did you expect to see?

CDC should be normal.

What did you see instead?

CDC print lots of WARN :
{"container":"ticdc","namespace":"cdc-testbed-tps-7653796-1-407","pod":"upstream-ticdc-0","log":"[client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=5m32.997091037s] [role=owner]","level":"WARN"}

CDC checkpoint lag abnormal:
Image

Versions of the cluster

cdc version: master, git-hash: 4538839

@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Dec 16, 2024
@flowbehappy
Copy link
Collaborator

Introduces by #11830

@fubinzh
Copy link
Author

fubinzh commented Dec 16, 2024

/severity major

@fubinzh
Copy link
Author

fubinzh commented Dec 16, 2024

/type regression

@3AceShowHand 3AceShowHand added affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. and removed may-affects-7.5 may-affects-8.1 may-affects-8.5 may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Dec 16, 2024
@ti-chi-bot ti-chi-bot bot closed this as completed in 0bb4977 Dec 16, 2024
@3AceShowHand 3AceShowHand removed the affects-7.1 This bug affects the 7.1.x(LTS) versions. label Dec 16, 2024
@wlwilliamx
Copy link
Contributor

/affects 6.5 7.1 7.5 8.1 8.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug. type/regression
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants