Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gorouting & connection leak when relay meet error #6193

Closed
GMHDBJD opened this issue Jul 5, 2022 · 0 comments · Fixed by #6803
Closed

gorouting & connection leak when relay meet error #6193

GMHDBJD opened this issue Jul 5, 2022 · 0 comments · Fixed by #6803
Assignees
@lance6716
Labels
affects-2.0.5 affects-2.0.6 affects-2.0.7 affects-5.3 affects-5.4 affects-6.0 affects-6.1 area/dm Issues or PRs related to DM. severity/major type/bug The issue is confirmed as a bug.

Comments

@GMHDBJD
Copy link
Contributor

GMHDBJD commented Jul 5, 2022

What did you do?

same relay config for different worker.

What did you expect to see?

no gorouting/connection leak

What did you see instead?

[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta2204532292: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta2204532292: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta2039700688: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta2039700688: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta466240616: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta466240616: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1663811013: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1663811013: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1749840261: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1749840261: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta874705000: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta874705000: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta323006719: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta323006719: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1752378300: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1752378300: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1377781961: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1377781961: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta74353850: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta74353850: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta2873734196: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta2873734196: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta339862555: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta339862555: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:719\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2022/07/05 21:17:43.331 +08:00] [ERROR] [relay.go:721] ["flush meta"] [component="relay log"] [error="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1841116126: no such file or directory"] [errorVerbose="[code=30006:class=relay-unit:scope=internal:level=high], Message: flush local meta, RawCause: open /tidbrlc/relay-dir/f0752560-90de-11e9-b9f6-b496914ca318.000001/relay.meta1841116126: no such file or directory\ngithub.com/pingcap/ticdc/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/ticdc/dm/pkg/terror/terror.go:311\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).doFlush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:247\ngithub.com/pingcap/ticdc/dm/relay.(*LocalMeta).Flush\n\tgithub.com/pingcap/ticdc/dm/relay/meta.go:228\ngithub.com/pingcap/ticdc/dm/relay.(*Relay).FlushMeta\n\tgithub.com/pingcap/ticdc/dm/relay/relay.go:844\ngithub.com/pingcap/ticdc/dm/relay

goroutine 873481 [select]:
[github.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps](http://github.com/pingcap/ticdc/dm/relay.(*Relay).doIntervalOps)(0xc000cc2000, 0x3b9f908, 0xc001682d80)
	[github.com/pingcap/ticdc/dm/relay/relay.go:711](http://github.com/pingcap/ticdc/dm/relay/relay.go:711) +0x1f5
created by [github.com/pingcap/ticdc/dm/relay.(*Relay).process](http://github.com/pingcap/ticdc/dm/relay.(*Relay).process)
	[github.com/pingcap/ticdc/dm/relay/relay.go:305](http://github.com/pingcap/ticdc/dm/relay/relay.go:305) +0x57a

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

(paste DM version here, and you must ensure versions of dmctl, DM-worker and DM-master are same)

Upstream MySQL/MariaDB server version:

(paste upstream MySQL/MariaDB server version here)

Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

How did you deploy DM: tiup or manually?

(leave TiUP or manually here)

Other interesting information (system version, hardware config, etc):

>
>

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)
@lance6716 lance6716 assigned lance6716 and unassigned lance6716 Jul 7, 2022
@lance6716 lance6716 self-assigned this Aug 18, 2022
@lance6716 lance6716 mentioned this issue Aug 19, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Aug 19, 2022
@lance6716
relay(dm): cancel when relay meet error to close goroutine (#6803)
a9b5c0b 
close #6193
This was referenced Aug 19, 2022
Closed
Merged
Merged
Merged
ti-chi-bot added a commit that referenced this issue Aug 19, 2022
@ti-chi-bot
relay(dm): cancel when relay meet error to close goroutine (#6803) (#…
147051d 
…6812)

close #6193
ti-chi-bot added a commit that referenced this issue Aug 22, 2022
@ti-chi-bot
relay(dm): cancel when relay meet error to close goroutine (#6803) (#…
82a688e 
…6813)

close #6193
ti-chi-bot added a commit that referenced this issue Aug 23, 2022
@ti-chi-bot
relay(dm): cancel when relay meet error to close goroutine (#6803) (#…
246e1d8 
…6814)

close #6193
This was referenced Aug 29, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lance6716 lance6716

Labels
affects-2.0.5 affects-2.0.6 affects-2.0.7 affects-5.3 affects-5.4 affects-6.0 affects-6.1 area/dm Issues or PRs related to DM. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

relay(dm): cancel when relay meet error to close goroutine lance6716/ticdc
3 participants
@lance6716 @GMHDBJD @ti-chi-bot