Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data race found in the kv client on the relase-7.1 #10718

Closed
3AceShowHand opened this issue Mar 6, 2024 · 1 comment · Fixed by #10719
Closed

data race found in the kv client on the relase-7.1 #10718

3AceShowHand opened this issue Mar 6, 2024 · 1 comment · Fixed by #10719
Assignees
Labels
affects-6.5 affects-7.1 affects-7.5 area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.

Comments

@3AceShowHand
Copy link
Contributor

==================
WARNING: DATA RACE
Read at 0x00c004305a50 by goroutine 1766:
  runtime.chansend()
      /usr/local/go/src/runtime/chan.go:160 +0x0
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).scheduleRegionRequest.func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:544 +0x7c9
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).scheduleRegionRequest()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:594 +0x6d6
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).divideAndSendEventFeedToRegions()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:894 +0xfaf
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed.func5.1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:488 +0xda
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x82

Previous write at 0x00c004305a50 by goroutine 1750:
  runtime.closechan()
      /usr/local/go/src/runtime/chan.go:357 +0x0
  github.com/pingcap/tiflow/pkg/chann.(*Chann[...]).unboundedTerminate()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/chann.go:223 +0xe4
  github.com/pingcap/tiflow/pkg/chann.(*Chann[...]).unboundedProcessing()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/chann.go:190 +0x54c
  github.com/pingcap/tiflow/pkg/chann.New[...].func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/chann.go:149 +0x47

Goroutine 1766 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0x12e
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed.func5()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:487 +0x124
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x82

Goroutine 1750 (finished) created at:
  github.com/pingcap/tiflow/pkg/chann.New[...]()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/chann.go:149 +0x4ca
  github.com/pingcap/tiflow/pkg/chann.NewAutoDrainChann[...]()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/drainable_chann.go:27 +0xae
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:449 +0x124
  github.com/pingcap/tiflow/cdc/kv.(*CDCClient).EventFeed()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:311 +0x16b
  github.com/pingcap/tiflow/cdc/puller.(*pullerImpl).Run.func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/puller/puller.go:142 +0x176
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x82
==================
panic: send on closed channel

goroutine 1249 [running]:
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).scheduleRegionRequest.func1({0x0, 0x63847af382c0003, 0xc004b48258, 0x0, {0x0, 0x0, 0x0}})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:543 +0x80a
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).scheduleRegionRequest(0xc00439b7a0, {0xaa834a8, 0xc002787ae0}, {{0x91, 0x7, 0x37}, {0x0, {0xc002d0fd40, 0x12, 0x18}, ...}, ...})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:594 +0x6d7
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).divideAndSendEventFeedToRegions(0xc00439b7a0, {0xaa834a8, 0xc002787ae0}, {0x5e, {0xc002d0fd40, 0x12, 0x18}, {0xc002d0fd58, 0x12, 0x18}})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:894 +0xfb0
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed.func5.1()
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:488 +0xdb
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x83
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0x12f

@3AceShowHand 3AceShowHand added area/ticdc Issues or PRs related to TiCDC. affect-ticdc-config-docs Pull requests that affect TiCDC configuration docs. and removed affect-ticdc-config-docs Pull requests that affect TiCDC configuration docs. labels Mar 6, 2024
@asddongmen asddongmen self-assigned this Mar 6, 2024
@3AceShowHand 3AceShowHand added the type/bug The issue is confirmed as a bug. label Mar 6, 2024
@asddongmen
Copy link
Contributor

asddongmen commented Mar 6, 2024

It seems that a new group was started here at

g, ctx := errgroup.WithContext(ctx)
, causing the goroutine created on line 505
g.Go(func() error {
to not be waited for by the wait here at
return g.Wait()

So a data race issue could happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 affects-7.1 affects-7.5 area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.
Projects
Development

Successfully merging a pull request may close this issue.

3 participants