Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic occur in checkIneligibleTableDDL #11839

Closed
CharlesCheung96 opened this issue Dec 6, 2024 · 1 comment · Fixed by #11856
Closed

Panic occur in checkIneligibleTableDDL #11839

CharlesCheung96 opened this issue Dec 6, 2024 · 1 comment · Fixed by #11856
Assignees
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. impact/panic severity/major type/bug The issue is confirmed as a bug.

Comments

@CharlesCheung96
Copy link
Contributor

CharlesCheung96 commented Dec 6, 2024

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3f9ae71]

goroutine 604 [running]:
github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).checkIneligibleTableDDL(0xc004ccbd80, 0xc003b45ea0, 0xc00277d520)
	github.com/pingcap/tiflow/cdc/puller/ddl_puller.go:518 +0xb1
github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).handleJob(0xc004ccbd80, 0xc00277d520)
	github.com/pingcap/tiflow/cdc/puller/ddl_puller.go:488 +0x3157
github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).handleRawKVEntry(0xc004ccbd80, {0x6054320, 0xc003a1d770}, 0xc000e6e0e0)
	github.com/pingcap/tiflow/cdc/puller/ddl_puller.go:204 +0xdc
github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).Run.func3()
	github.com/pingcap/tiflow/cdc/puller/ddl_puller.go:151 +0xb5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync@v0.9.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 600
	golang.org/x/sync@v0.9.0/errgroup/errgroup.go:75 +0x96
@CharlesCheung96
Copy link
Contributor Author

CharlesCheung96 commented Dec 6, 2024

Investigate

The directly reason

Image

  1. Some DDL statements with ifNotExists/ifExists may not actually be executed because the condition is not met, resulting in Schemaversion being set to 0 after it is committed.
  2. After this fix, ddl job can not be filtered by Schemaversion.
  3. Then, the panic occus when access a uninitialized tableinfo here

The root cause

The ddl-related implementation of TiCDC is not aligned with TiDB's, so it is easy to cause inconsistencies due to miscalculations in some details such as ddl query, id, and schema related information.

Proposal

From @D3Hunter, we need to clearly define the contract for ddl replication and specify the constraints that ddl job meta must meet when the DDL event is set to the done state.

@CharlesCheung96 CharlesCheung96 added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. and removed may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 labels Dec 8, 2024
@ti-chi-bot ti-chi-bot bot closed this as completed in 7f57e1f Dec 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. impact/panic severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants