Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data race when running leak test #4447

Closed
Tracked by #2246
buchuitoudegou opened this issue Jan 24, 2022 · 1 comment · Fixed by #4605
Closed
Tracked by #2246

data race when running leak test #4447

buchuitoudegou opened this issue Jan 24, 2022 · 1 comment · Fixed by #4605
Assignees
@liuzix
Labels
affects-4.0 affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 area/ticdc Issues or PRs related to TiCDC. severity/minor type/bug The issue is confirmed as a bug.

Comments

@buchuitoudegou
Copy link
Contributor

buchuitoudegou commented Jan 24, 2022 •
edited
Loading

What did you do?

/run-leak-test

log: https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_leak_test/detail/cdc_ghpr_leak_test/19037/pipeline

What did you expect to see?

no error

What did you see instead?


[2022-01-24T04:23:28.536Z] ==================

[2022-01-24T04:23:28.536Z] WARNING: DATA RACE

[2022-01-24T04:23:28.536Z] Write at 0x00c001d04970 by goroutine 75:

[2022-01-24T04:23:28.537Z]   runtime.closechan()

[2022-01-24T04:23:28.537Z]       /usr/local/go/src/runtime/chan.go:355 +0x0

[2022-01-24T04:23:28.537Z]   github.com/pingcap/tiflow/cdc/sorter/unified.(*Sorter).Run()

[2022-01-24T04:23:28.537Z]       /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/unified_sorter.go:235 +0x10c9

[2022-01-24T04:23:28.537Z]   github.com/pingcap/tiflow/cdc/sorter/unified.testSorter.func2()

[2022-01-24T04:23:28.537Z]       /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:144 +0x5c

[2022-01-24T04:23:28.537Z]   golang.org/x/sync/errgroup.(*Group).Go.func1()

[2022-01-24T04:23:28.537Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x94

[2022-01-24T04:23:28.537Z] 

[2022-01-24T04:23:28.537Z] Previous read at 0x00c001d04970 by goroutine 114:

[2022-01-24T04:23:28.537Z]   ??()

[2022-01-24T04:23:28.537Z]       -:0 +0xffffffffffffffff

[2022-01-24T04:23:28.537Z]   github.com/pingcap/tiflow/pkg/workerpool.(*defaultPoolImpl).Run.func1()

[2022-01-24T04:23:28.537Z]       /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/workerpool/pool_impl.go:67 +0x77

[2022-01-24T04:23:28.537Z]   golang.org/x/sync/errgroup.(*Group).Go.func1()

[2022-01-24T04:23:28.537Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x94

[2022-01-24T04:23:28.537Z] 

[2022-01-24T04:23:28.537Z] Goroutine 75 (running) created at:

[2022-01-24T04:23:28.537Z]   golang.org/x/sync/errgroup.(*Group).Go()

[2022-01-24T04:23:28.537Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x73

[2022-01-24T04:23:28.537Z]   github.com/pingcap/tiflow/cdc/sorter/unified.testSorter()

Versions of the cluster

dev latest
@buchuitoudegou buchuitoudegou added type/bug The issue is confirmed as a bug. area/ticdc Issues or PRs related to TiCDC. labels Jan 24, 2022
@lance6716 lance6716 removed the type/bug The issue is confirmed as a bug. label Jan 24, 2022
@lance6716 lance6716 mentioned this issue Jan 24, 2022
Closed
@maxshuang
Copy link
Contributor

2022-02-14T07:36:19.841Z] [2022/02/14 15:35:35.079 +08:00] [INFO] [unified_sorter.go:112] ["sorterDebug: Running Unified Sorter in debug mode"]

[2022-02-14T07:36:19.841Z] [2022/02/14 15:35:46.091 +08:00] [WARN] [cancel_monitor.go:43] ["MonitorCancelLatency: Cancellation is taking too long"] [identifier="Unified Sorter"] [duration=1] [error="context deadline exceeded"]

[2022-02-14T07:36:19.841Z] ==================

[2022-02-14T07:36:19.841Z] WARNING: DATA RACE

[2022-02-14T07:36:19.841Z] Write at 0x00c003218190 by goroutine 108:

[2022-02-14T07:36:19.841Z] runtime.closechan()

[2022-02-14T07:36:19.841Z] /usr/local/go/src/runtime/chan.go:355 +0x0

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/cdc/sorter/unified.(*Sorter).Run()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/unified_sorter.go:236 +0x1129

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/cdc/sorter/unified.testSorter.func2()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:144 +0x5c

[2022-02-14T07:36:19.841Z] golang.org/x/sync/errgroup.(*Group).Go.func1()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x94

[2022-02-14T07:36:19.841Z]

[2022-02-14T07:36:19.841Z] Previous read at 0x00c003218190 by goroutine 184:

[2022-02-14T07:36:19.841Z] runtime.chansend()

[2022-02-14T07:36:19.841Z] /usr/local/go/src/runtime/chan.go:158 +0x0

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/cdc/sorter/unified.(*Sorter).Run.func1.1()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/unified_sorter.go:141 +0x5d

[2022-02-14T07:36:19.841Z] sync.(*Once).doSlow()

[2022-02-14T07:36:19.841Z] /usr/local/go/src/sync/once.go:68 +0x109

[2022-02-14T07:36:19.841Z] sync.(*Once).Do()

[2022-02-14T07:36:19.841Z] /usr/local/go/src/sync/once.go:59 +0x68

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/cdc/sorter/unified.(*Sorter).Run.func1()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/unified_sorter.go:140 +0x85

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/pkg/workerpool.(*defaultEventHandle).doCancel()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/workerpool/pool_impl.go:229 +0x126

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/pkg/workerpool.(*defaultEventHandle).cancelWithErr()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/workerpool/pool_impl.go:258 +0x7a

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/pkg/workerpool.(*worker).run()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/workerpool/pool_impl.go:347 +0x545

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/pkg/workerpool.(*defaultPoolImpl).Run.func1()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/workerpool/pool_impl.go:67 +0x77

[2022-02-14T07:36:19.841Z] golang.org/x/sync/errgroup.(*Group).Go.func1()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x94

[2022-02-14T07:36:19.841Z]

[2022-02-14T07:36:19.841Z] Goroutine 108 (running) created at:

[2022-02-14T07:36:19.841Z] golang.org/x/sync/errgroup.(*Group).Go()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x73

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/cdc/sorter/unified.testSorter()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:143 +0x4e8

[2022-02-14T07:36:19.841Z] github.com/pingcap/failpoint.parseTerm()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:149 +0x256

[2022-02-14T07:36:19.841Z] github.com/pingcap/failpoint.parse()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xb3

[2022-02-14T07:36:19.841Z] github.com/pingcap/failpoint.newTerms()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x6f

[2022-02-14T07:36:19.841Z] github.com/pingcap/failpoint.(*Failpoint).Enable()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x53

[2022-02-14T07:36:19.841Z] github.com/pingcap/failpoint.(*Failpoints).Enable()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x171

[2022-02-14T07:36:19.841Z] github.com/pingcap/failpoint.Enable()

[2022-02-14T07:36:19.841Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x8a

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/cdc/sorter/unified.testSorter()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:131 +0x50

[2022-02-14T07:36:19.841Z] github.com/pingcap/tiflow/cdc/sorter/unified.(*sorterSuite).TestSorterCancelRestart()

[2022-02-14T07:36:19.841Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:299 +0x8b8

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.parse()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xb3

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.newTerms()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x6f

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.(*Failpoint).Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x53

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.(*Failpoints).Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x171

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x8a

[2022-02-14T07:36:19.842Z] github.com/pingcap/tiflow/cdc/sorter/unified.testSorter()

[2022-02-14T07:36:19.842Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:131 +0x50

[2022-02-14T07:36:19.842Z] github.com/pingcap/tiflow/cdc/sorter/unified.(*sorterSuite).TestSorterCancelRestart()

[2022-02-14T07:36:19.842Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:299 +0x8b8

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.parse()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xb3

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.newTerms()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x6f

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.(*Failpoint).Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x53

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.(*Failpoints).Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x171

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x8a

[2022-02-14T07:36:19.842Z] github.com/pingcap/tiflow/cdc/sorter/unified.testSorter()

[2022-02-14T07:36:19.842Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:131 +0x50

[2022-02-14T07:36:19.842Z] github.com/pingcap/tiflow/cdc/sorter/unified.(*sorterSuite).TestSorterCancelRestart()

[2022-02-14T07:36:19.842Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:299 +0x8b8

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.(*Failpoint).Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x53

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.(*Failpoints).Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x171

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x568

[2022-02-14T07:36:19.842Z] github.com/pingcap/tiflow/cdc/sorter/unified.(*sorterSuite).TestSorterCancelRestart()

[2022-02-14T07:36:19.842Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:289 +0x52e

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.(*Failpoint).Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x53

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.(*Failpoints).Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x171

[2022-02-14T07:36:19.842Z] github.com/pingcap/failpoint.Enable()

[2022-02-14T07:36:19.842Z] /go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x473

[2022-02-14T07:36:19.842Z] github.com/pingcap/tiflow/cdc/sorter/unified.(*sorterSuite).TestSorterCancelRestart()

[2022-02-14T07:36:19.843Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/sorter_test.go:282 +0x439

[2022-02-14T07:36:19.843Z] runtime.call16()

[2022-02-14T07:36:19.843Z] /usr/local/go/src/runtime/asm_amd64.s:550 +0x3d

[2022-02-14T07:36:19.843Z] reflect.Value.Call()

[2022-02-14T07:36:19.843Z] /usr/local/go/src/reflect/value.go:337 +0xd8

[2022-02-14T07:36:19.843Z] github.com/pingcap/check.(*suiteRunner).forkTest.func1()

[2022-02-14T07:36:19.843Z] /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0xb3b

[2022-02-14T07:36:19.843Z] github.com/pingcap/check.(*suiteRunner).forkCall.func1()

[2022-02-14T07:36:19.843Z] /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x11d

[2022-02-14T07:36:19.843Z]

[2022-02-14T07:36:19.843Z] Goroutine 184 (finished) created at:

[2022-02-14T07:36:19.843Z] golang.org/x/sync/errgroup.(*Group).Go()

[2022-02-14T07:36:19.843Z] /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x73

[2022-02-14T07:36:19.843Z] github.com/pingcap/tiflow/pkg/workerpool.(*defaultPoolImpl).Run()

[2022-02-14T07:36:19.843Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/workerpool/pool_impl.go:66 +0x1e4

[2022-02-14T07:36:19.843Z] github.com/pingcap/tiflow/cdc/sorter/unified.RunWorkerPool.func1()

[2022-02-14T07:36:19.843Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/sorter/unified/unified_sorter.go:278 +0x86

[2022-02-14T07:36:19.843Z] golang.org/x/sync/errgroup.(*Group).Go.func1()

[2022-02-14T07:36:19.843Z] /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x94

[2022-02-14T07:36:19.843Z] ==================

@liuzix liuzix mentioned this issue Feb 16, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Feb 24, 2022
@liuzix
pkg/workerpool(cdc): fix race between handling error and unregisterin…
79be937 
…g handler (#4605)

close #4447
This was referenced Feb 24, 2022
Closed
Closed
Merged
Merged
Merged
Closed
ti-chi-bot added a commit that referenced this issue Apr 14, 2022
@ti-chi-bot
pkg/workerpool(cdc): fix race between handling error and unregisterin…
d92f016 
…g handler (#4605) (#4681)

close #4447
ti-chi-bot added a commit that referenced this issue Apr 21, 2022
@ti-chi-bot
pkg/workerpool(cdc): fix race between handling error and unregisterin…
79b5af1 
…g handler (#4605) (#4683)

close #4447
ti-chi-bot added a commit that referenced this issue Jun 16, 2022
@ti-chi-bot
pkg/workerpool(cdc): fix race between handling error and unregisterin…
c6fdb3b 
…g handler (#4605) (#4682)

close #4447
This was referenced Jun 23, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liuzix liuzix

Labels
affects-4.0 affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 area/ticdc Issues or PRs related to TiCDC. severity/minor type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pkg/workerpool(cdc): fix race between handling error and unregistering handler liuzix/ticdc
4 participants
@lance6716 @maxshuang @liuzix @buchuitoudegou