Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ticdc lag is more than 10m after inject pdleader network partition to all #8808

Closed
Lily2025 opened this issue Apr 19, 2023 · 8 comments · Fixed by #9106
Closed

ticdc lag is more than 10m after inject pdleader network partition to all #8808

Lily2025 opened this issue Apr 19, 2023 · 8 comments · Fixed by #9106
Assignees
@asddongmen
Labels
affects-6.1 affects-6.5 affects-7.1 area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@Lily2025
Copy link

Lily2025 commented Apr 19, 2023 •
edited
Loading

What did you do?

inject pdleader network partition
http://perf.pingcap.net:3000/d/WOw1JWmVk/ha-test-case-detail?orgId=1&refresh=10s&var-case_id=2731539

What did you expect to see?

ticdc lag is less than 30s
34cb2477-6bc3-4f10-8e1a-8992ea76046e

What did you see instead?

ticdc lag is more than 10m after inject pdleader network partition

Versions of the cluster

git hash：5252420517ce5497ed053aa6322fdb2a33f898e1

current status of DM cluster (execute query-status <task-name> in dmctl)

No response
@Lily2025 Lily2025 added area/dm Issues or PRs related to DM. type/bug The issue is confirmed as a bug. labels Apr 19, 2023
@Lily2025
Copy link
Author

/remove-area dm
/area ticdc
/assign overvenus

@ti-chi-bot ti-chi-bot added area/ticdc Issues or PRs related to TiCDC. and removed area/dm Issues or PRs related to DM. labels Apr 19, 2023
@fubinzh
Copy link

fubinzh commented Apr 19, 2023

/severity major

@Lily2025 Lily2025 changed the title ticdc lag is more than 10m after inject pdleader network partition ticdc lag is more than 10m after inject pdleader network partition to all Apr 19, 2023
@nongfushanquan
Copy link
Contributor

/unassign @overvenus
/assign @asddongmen

@ti-chi-bot ti-chi-bot bot assigned asddongmen and unassigned overvenus May 24, 2023
@asddongmen asddongmen mentioned this issue May 26, 2023
Closed
@ti-chi-bot ti-chi-bot bot closed this as completed in #9106 Jun 1, 2023
ti-chi-bot bot pushed a commit that referenced this issue Jun 1, 2023
@asddongmen
etcd (ticdc): add grpc keepalive params and add timeout for check pd …
e1826b3 
…version ctx. (#9106)

close #8808
This was referenced Jun 1, 2023
Closed
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this issue Jun 1, 2023
@asddongmen @ti-chi-bot
etcd (ticdc): add grpc keepalive params and add timeout for check pd …
470d826 
…version ctx. (pingcap#9106)

close pingcap#8808
@ti-chi-bot ti-chi-bot mentioned this issue Jun 1, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jun 2, 2023
@ti-chi-bot
etcd (ticdc): add grpc keepalive params and add timeout for check pd …
f827a89 
…version ctx. (#9106) (#9118)

close #8808
This was referenced Jun 9, 2023
Merged
Closed
@asddongmen asddongmen mentioned this issue Jun 13, 2023
Closed
@asddongmen asddongmen mentioned this issue Jun 27, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jun 30, 2023
@ti-chi-bot
etcd (ticdc): add grpc keepalive params and add timeout for check pd …
a217fc5 
…version ctx. (#9106) (#9119)

close #8808
@nongfushanquan nongfushanquan mentioned this issue Jul 14, 2023
Merged
17 tasks
@jebter jebter reopened this Sep 15, 2023
@fubinzh
Copy link

fubinzh commented Sep 15, 2023

This issue is seen when testing with 6.5 nightly build.
image

bash-5.1# /cdc version
Release Version: v6.5.0-nightly
Git Commit Hash: 2ab025e
Git Branch: heads/refs/tags/v6.5.0-nightly

@asddongmen asddongmen mentioned this issue Oct 31, 2023
Merged
@asddongmen
Copy link
Contributor

This issue is mitigated by #9661, so I change its severity to moderate.

@asddongmen
Copy link
Contributor

img_v2_fd85ed6b-0f1c-473e-aa08-130401676f8g
img_v2_d6217e6b-78d1-452e-8d32-d7d06f844a5g

@nongfushanquan
Copy link
Contributor

/remove-label affects-7.5

@ti-chi-bot ti-chi-bot bot removed the affects-7.5 label Nov 1, 2023
@asddongmen
Copy link
Contributor

The daily HA test results from the past year indicate that the lag consistently remains within 1 minute. This meets our RPO goal, leading me to close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@asddongmen asddongmen

Labels
affects-6.1 affects-6.5 affects-7.1 area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
8 participants
@overvenus @jebter @fubinzh @VelocityLight @asddongmen @ti-chi-bot @Lily2025 @nongfushanquan