Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

huge logs for "event feed receives a region error" and "region worker get a region error" during initial scan #10177

Closed
fubinzh opened this issue Nov 29, 2023 · 4 comments

Comments

@fubinzh
Copy link

fubinzh commented Nov 29, 2023

What did you do?

  1. What did you do?
  2. Deploy TiDB cluster with 6 TiKV and 2 CDC
  3. Stop GC
  4. Run workload for 1 days to generate hube incremetal data for cdc to sync
  5. Resume cdc changefeed with start-ts earlier to initial scan for workload events.

What did you expect to see?

  1. There should not be too much

What did you see instead?

  1. There are too much logs about "event feed receives a region error" and "region worker get a region error", which are introduced by kv-client(cdc): fix the bug that region errors can be lost (#9963) #9977

image
image
image
image
image
image

Versions of the cluster

Upstream TiKV version (execute tikv-server --version):

[root@tc-tikv-0 /]# /tikv-server -V
TiKV
Release Version:   7.6.0-alpha
Edition:           Community
Git Commit Hash:   04ab96c248408ef4c71807b527f9cc925ce8fb42
Git Commit Branch: heads/refs/tags/v7.5.0-pr16048-04ab96
UTC Build Time:    2023-11-23 07:55:18
Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Enable Features:   pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored
Profile:           dist_release

TiCDC version (execute cdc version):

[root@tc-ticdc-0 /]# /cdc version
Release Version: v7.5.0
Git Commit Hash: 8c006b38bdc304201d646c35c993156abcc2b845
Git Branch: heads/refs/tags/v7.5.0
UTC Build Time: 2023-11-20 05:31:19
Go Version: go version go1.21.3 linux/amd64
Failpoint Build: false
@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Nov 29, 2023
@fubinzh
Copy link
Author

fubinzh commented Nov 29, 2023

/severity major

@fubinzh fubinzh changed the title CDC throughput are 0 for most of the time and huge logs for "event feed receives a region error" and "region worker get a region error" during initial scan huge logs for "event feed receives a region error" and "region worker get a region error" during initial scan Dec 7, 2023
@fubinzh
Copy link
Author

fubinzh commented Dec 8, 2023

/label affects-6.5
/label affects-7.1
/label affects-7.5

@hicqu
Copy link
Contributor

hicqu commented Dec 8, 2023

Closed by #10268.

@hicqu hicqu closed this as completed Dec 8, 2023
@zimulala
Copy link
Contributor

zimulala commented Dec 8, 2023

Closed by #10241.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

3 participants