Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

changefeed LAG increased when a tikv is killed #9741

Closed
sdojjy opened this issue Sep 13, 2023 · 4 comments
Closed

changefeed LAG increased when a tikv is killed #9741

sdojjy opened this issue Sep 13, 2023 · 4 comments
Assignees
@sdojjy
Labels
affects-6.5 affects-7.1 area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.

Comments

@sdojjy
Copy link
Member

sdojjy commented Sep 13, 2023

Every region belongs to the killed tikv store will try to connect that tikv again, and the retry backoff time is 100ms
If there are thousands of regions, that will cause big changefeed LAG (100ms * region count)

[2023/09/13 17:09:03.713 +08:00] [WARN] [client.go:636] ["get grpc stream client failed"] [namespace=default] [changefeed=test] [regionID=1398592] [requestID=102976] [storeID=6] [error="[CDC:ErrTiKVEventFeed]tikv event feed failed: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.2.6.210:20160: connect: connection refused\""] [errorVerbose="[CDC:ErrTiKVEventFeed]tikv event feed failed: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.2.6.210:20160: connect: connection refused\"\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/kv.(*CDCClient).newStream\n\tgithub.com/pingcap/tiflow/cdc/kv/client.go:239\ngithub.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).requestRegionToStore\n\tgithub.com/pingcap/tiflow/cdc/kv/client.go:632\ngithub.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed.func3\n\tgithub.com/pingcap/tiflow/cdc/kv/client.go:425\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
@sdojjy sdojjy added area/ticdc Issues or PRs related to TiCDC. type/enhancement The issue or PR belongs to an enhancement. labels Sep 13, 2023
@sdojjy sdojjy mentioned this issue Sep 13, 2023
Merged
@nongfushanquan
Copy link
Contributor

/label affects-6.5

@sdojjy sdojjy added type/bug The issue is confirmed as a bug. and removed type/enhancement The issue or PR belongs to an enhancement. labels Sep 14, 2023
ti-chi-bot bot pushed a commit that referenced this issue Sep 14, 2023
@sdojjy
kv(ticdc): Remove store connect backoff (#9740)
41c8fdf 
close #9741
@nongfushanquan nongfushanquan mentioned this issue Sep 15, 2023
Merged
18 tasks
@nongfushanquan
Copy link
Contributor

/assign @sdojjy

@sdojjy sdojjy mentioned this issue Sep 19, 2023
Merged
@nongfushanquan nongfushanquan mentioned this issue Sep 21, 2023
Closed
@lijie0123 lijie0123 mentioned this issue Sep 21, 2023
Closed
11 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 26, 2023
@sdojjy
kv(ticdc): remove backoff from newStream func (#9771)
43848f2 
ref #9741
This was referenced Sep 26, 2023
Closed
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 27, 2023
@ti-chi-bot
kv(ticdc): remove backoff from newStream func (#9771) (#9810)
b7e97aa 
ref #9741
@nongfushanquan
Copy link
Contributor

/close

@ti-chi-bot ti-chi-bot bot closed this as completed Sep 28, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Sep 28, 2023

@nongfushanquan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@nongfushanquan nongfushanquan mentioned this issue Oct 18, 2023
Merged
18 tasks
hicqu pushed a commit to hicqu/ticdc that referenced this issue Nov 22, 2023
@ti-chi-bot @hicqu
kv(ticdc): remove backoff from newStream func (pingcap#9771) (pingcap…
249f520 
…#9810)

ref pingcap#9741
@nongfushanquan nongfushanquan mentioned this issue Nov 30, 2023
Merged
16 tasks
sdojjy added a commit to sdojjy/tiflow that referenced this issue Dec 29, 2023
@sdojjy
kv(ticdc): Remove store connect backoff (pingcap#9740)
329b0ad 
close pingcap#9741
sdojjy added a commit to sdojjy/tiflow that referenced this issue Dec 29, 2023
@sdojjy
kv(ticdc): Remove store connect backoff (pingcap#9740)
b8be2e5 
close pingcap#9741
sdojjy added a commit to sdojjy/tiflow that referenced this issue Dec 29, 2023
@sdojjy
kv(ticdc): Remove store connect backoff (pingcap#9740)
9e0b4cc 
close pingcap#9741
@sdojjy sdojjy mentioned this issue Dec 29, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sdojjy sdojjy

Labels
affects-6.5 affects-7.1 area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sdojjy @nongfushanquan