Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TiCDC] Owner fails to clean up stale metrics leading to incorrect display in dashboard #4774

Closed
liuzix opened this issue Mar 4, 2022 · 2 comments · Fixed by #4775 or #4998
Closed

[TiCDC] Owner fails to clean up stale metrics leading to incorrect display in dashboard #4774

liuzix opened this issue Mar 4, 2022 · 2 comments · Fixed by #4775 or #4998
Assignees
@liuzix
Labels
affects-4.0 affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 affects-6.0 area/ticdc Issues or PRs related to TiCDC. severity/minor type/bug The issue is confirmed as a bug.

Comments

@liuzix
Copy link
Contributor

liuzix commented Mar 4, 2022 •
edited
Loading

If a owner has resigned, it may fail to clean up its metrics data. In some metric items, such as changefeed checkpoint lag, the dashboard tries to calculate a maximum of all existing time series, causing incorrect displayed values in the dashboard.

image

For example, in the screenshot above, the checkpoint lag remains unchanged after an old owner has resigned with a high lag.
@liuzix liuzix mentioned this issue Mar 4, 2022
Merged
@liuzix liuzix self-assigned this Mar 4, 2022
@liuzix liuzix added the type/bug The issue is confirmed as a bug. label Mar 4, 2022
ti-chi-bot pushed a commit that referenced this issue Mar 7, 2022
@liuzix
owner(cdc): clean up stale metrics (#4775)
3698b6e 
close #4774
This was referenced Mar 7, 2022
Closed
Closed
Closed
Closed
Closed
Closed
@liuzix
Copy link
Contributor Author

liuzix commented Mar 23, 2022

#4775 seems to be an incomplete fix.

@liuzix liuzix reopened this Mar 23, 2022
@liuzix liuzix mentioned this issue Mar 23, 2022
Merged
@liuzix
Copy link
Contributor Author

liuzix commented Mar 23, 2022

#4775 only mitigated the situation a little bit, and it assumed that Reset() methods on the prometheus metric items can clean up data from all instances, which is wrong. A new PR has been open, which is a complete fix.

ti-chi-bot pushed a commit that referenced this issue Mar 28, 2022
@liuzix
owner(cdc): clean up metrics when owner resigns (#4998)
809ad60 
close #4774
This was referenced Mar 28, 2022
Closed
Closed
Closed
Closed
Closed
Closed
Closed
liuzix added a commit to liuzix/ticdc that referenced this issue Apr 24, 2022
@liuzix
owner(cdc): clean up stale metrics (pingcap#4775)
57f85ed 
close pingcap#4774
liuzix added a commit to liuzix/ticdc that referenced this issue Apr 24, 2022
@liuzix
owner(cdc): clean up metrics when owner resigns (pingcap#4998)
235bbc8 
close pingcap#4774
@liuzix liuzix mentioned this issue Apr 24, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Apr 25, 2022
@liuzix
owner(cdc): backport stale metrics fix to 5.4 (#5260)
1cbf7f7 
ref #4774
liuzix added a commit to liuzix/ticdc that referenced this issue Jun 16, 2022
@liuzix
owner(cdc): backport stale metrics fix to 5.3
6129c89 
ref pingcap#4774
@liuzix liuzix mentioned this issue Jun 16, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Jun 16, 2022
@liuzix
owner(cdc): backport stale metrics fix to 5.3 (#5894)
dcb5b39 
ref #4774
This was referenced Jun 23, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liuzix liuzix

Labels
affects-4.0 affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 affects-6.0 area/ticdc Issues or PRs related to TiCDC. severity/minor type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
2 participants
@amyangfei @liuzix