Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DPANIC with MariaDB 10.6 #10206

Closed
dveeden opened this issue Nov 30, 2023 · 5 comments · Fixed by #10207
Closed

DPANIC with MariaDB 10.6 #10206

dveeden opened this issue Nov 30, 2023 · 5 comments · Fixed by #10207
Labels
affects-6.3 affects-6.4 affects-7.1 affects-7.5 affects-8.1 area/dm Issues or PRs related to DM. MariaDB This issue is found with MariaDB upstream severity/moderate type/bug The issue is confirmed as a bug.

Comments

@dveeden
Copy link
Contributor

dveeden commented Nov 30, 2023

What did you do?

Setup DM with MariaDB 10.6.11

    [2023/11/29 11:17:55.229 +02:00]
    [DPANIC]
    [tracker.go:206]
    ["unexpected statement type"]
    [task=demo-mariadb-tidb]
    [unit="binlog replication"]
    [component=schema-tracker]
    [task=demo-mariadb-tidb]
    [type=*ast.SetStmt]
    [stack="github.com/pingcap/tiflow/dm/pkg/schema.(*Tracker).Exec\n\tgithub.com/pingcap/tiflow/dm/pkg/schema/tracker.go:206\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).loadTableStructureFromDump\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:3068\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).Run\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:1856\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).Process\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:757"]
    
    [2023/11/29 11:17:55.231 +02:00]
    [INFO]
    [syncer.go:1688]
    ["received ungraceful exit ctx, exit now"]
    [task=demo-mariadb-tidb]
    [unit="binlog replication"]
    panic: interface conversion: ast.StmtNode is *ast.SetStmt, not *ast.CreateTableStmt
    
    goroutine 1807 [running]:
    github.com/pingcap/tiflow/dm/syncer.(*Syncer).loadTableStructureFromDump(0xc000cb7200, {0x4a54438, 0xc00105dd10})
            github.com/pingcap/tiflow/dm/syncer/syncer.go:3079 +0x1ce9
    github.com/pingcap/tiflow/dm/syncer.(*Syncer).Run(0xc000cb7200, {0x4a54438?, 0xc00105dd10?})
            github.com/pingcap/tiflow/dm/syncer/syncer.go:1856 +0xcd6
    github.com/pingcap/tiflow/dm/syncer.(*Syncer).Process(0xc000cb7200, {0x4a54438, 0xc00105dcc0}, 0x0?)
            github.com/pingcap/tiflow/dm/syncer/syncer.go:757 +0x325
    created by github.com/pingcap/tiflow/dm/worker.(*SubTask).run in goroutine 1707
            github.com/pingcap/tiflow/dm/worker/subtask.go:242 +0x53b

What did you expect to see?

No response

What did you see instead?

No response

Versions of the cluster

Master @ 13499f4

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)
@dveeden dveeden added area/dm Issues or PRs related to DM. type/bug The issue is confirmed as a bug. labels Nov 30, 2023
@dveeden dveeden mentioned this issue Nov 30, 2023
Merged
@dveeden
Copy link
Contributor Author

dveeden commented Nov 30, 2023

Without any formatting:

[2023/11/30 13:24:16.218 +01:00] [DPANIC] [tracker.go:206] ["unexpected statement type"] [task=demo-mariadb-tidb] [unit="binlog replication"] [component=schema-tracker] [task=demo-mariadb-tidb] [type=*ast.SetStmt] [stack="github.com/pingcap/tiflow/dm/pkg/schema.(*Tracker).Exec\n\tgithub.com/pingcap/tiflow/dm/pkg/schema/tracker.go:206\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).loadTableStructureFromDump\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:3068\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).Run\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:1856\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).Process\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:757"]
[2023/11/30 13:24:16.219 +01:00] [INFO] [syncer.go:1688] ["received ungraceful exit ctx, exit now"] [task=demo-mariadb-tidb] [unit="binlog replication"]
panic: interface conversion: ast.StmtNode is *ast.SetStmt, not *ast.CreateTableStmt

goroutine 2068 [running]:
github.com/pingcap/tiflow/dm/syncer.(*Syncer).loadTableStructureFromDump(0xc000d9c000, {0x4a54438, 0xc003f57720})
	github.com/pingcap/tiflow/dm/syncer/syncer.go:3079 +0x1ce9
github.com/pingcap/tiflow/dm/syncer.(*Syncer).Run(0xc000d9c000, {0x4a54438?, 0xc003f57720?})
	github.com/pingcap/tiflow/dm/syncer/syncer.go:1856 +0xcd6
github.com/pingcap/tiflow/dm/syncer.(*Syncer).Process(0xc000d9c000, {0x4a54438, 0xc003f576d0}, 0x0?)
	github.com/pingcap/tiflow/dm/syncer/syncer.go:757 +0x325
created by github.com/pingcap/tiflow/dm/worker.(*SubTask).run in goroutine 1949
	github.com/pingcap/tiflow/dm/worker/subtask.go:242 +0x53b

@dveeden
Copy link
Contributor Author

dveeden commented Nov 30, 2023

With a small change to show what the SET statement is:

[2023/11/30 13:30:16.896 +01:00] [DPANIC] [tracker.go:206] ["Encountered SET statement"] [task=demo-mariadb-tidb] [unit="binlog replication"] [component=schema-tracker] [task=demo-mariadb-tidb] [stmt="SET FOREIGN_KEY_CHECKS=0"] [stack="github.com/pingcap/tiflow/dm/pkg/schema.(*Tracker).Exec\n\tgithub.com/pingcap/tiflow/dm/pkg/schema/tracker.go:206\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).loadTableStructureFromDump\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:3068\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).Run\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:1856\ngithub.com/pingcap/tiflow/dm/syncer.(*Syncer).Process\n\tgithub.com/pingcap/tiflow/dm/syncer/syncer.go:757"]
[2023/11/30 13:30:16.896 +01:00] [INFO] [syncer.go:1688] ["received ungraceful exit ctx, exit now"] [task=demo-mariadb-tidb] [unit="binlog replication"]
panic: interface conversion: ast.StmtNode is *ast.SetStmt, not *ast.CreateTableStmt

goroutine 2082 [running]:
github.com/pingcap/tiflow/dm/syncer.(*Syncer).loadTableStructureFromDump(0xc000b0e900, {0x4a54478, 0xc0033f57c0})
	github.com/pingcap/tiflow/dm/syncer/syncer.go:3079 +0x1ce9
github.com/pingcap/tiflow/dm/syncer.(*Syncer).Run(0xc000b0e900, {0x4a54478?, 0xc0033f57c0?})
	github.com/pingcap/tiflow/dm/syncer/syncer.go:1856 +0xcd6
github.com/pingcap/tiflow/dm/syncer.(*Syncer).Process(0xc000b0e900, {0x4a54478, 0xc0033f5770}, 0x0?)
	github.com/pingcap/tiflow/dm/syncer/syncer.go:757 +0x325
created by github.com/pingcap/tiflow/dm/worker.(*SubTask).run in goroutine 1952
	github.com/pingcap/tiflow/dm/worker/subtask.go:242 +0x53b

diff --git a/dm/pkg/schema/tracker.go b/dm/pkg/schema/tracker.go
index 9da3f49ff..063498e10 100644
--- a/dm/pkg/schema/tracker.go
+++ b/dm/pkg/schema/tracker.go
@@ -202,6 +202,10 @@ func (tr *Tracker) Exec(ctx context.Context, db string, stmt ast.StmtNode) (errR
        case *ast.TruncateTableStmt:
                ident := ast.Ident{Schema: v.Table.Schema, Name: v.Table.Name}
                return tr.upstreamTracker.TruncateTable(tr.se, ident)
+       case *ast.SetStmt:
+               tr.logger.DPanic("Encountered SET statement",
+                       zap.String("stmt", v.Text()),
+               )
        default:
                tr.logger.DPanic("unexpected statement type", zap.String("type", fmt.Sprintf("%T", v)))
        }

So the statement is SET FOREIGN_KEY_CHECKS=0

@dveeden
Copy link
Contributor Author

dveeden commented Nov 30, 2023 

$ find dumped_data.demo-mariadb-tidb/ -type f -exec grep SET\ FOREIGN_KEY_CHECKS {} + | tail -5
dumped_data.demo-mariadb-tidb/performance_schema.user_variables_by_thread-schema.sql:SET FOREIGN_KEY_CHECKS=0;
dumped_data.demo-mariadb-tidb/performance_schema.users-schema.sql:SET FOREIGN_KEY_CHECKS=0;
dumped_data.demo-mariadb-tidb/sys-schema-create.sql:SET FOREIGN_KEY_CHECKS=0;
dumped_data.demo-mariadb-tidb/sys.sys_config-schema.sql:SET FOREIGN_KEY_CHECKS=0;
dumped_data.demo-mariadb-tidb/test-schema-create.sql:SET FOREIGN_KEY_CHECKS=0;
$ cat dumped_data.demo-mariadb-tidb/test-schema-create.sql
/*!40101 SET NAMES binary*/;
SET FOREIGN_KEY_CHECKS=0;
CREATE DATABASE `test` /*!40100 DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci */;

dumped_data.demo-mariadb-tidb.zip

@lance6716 lance6716 added the MariaDB This issue is found with MariaDB upstream label Dec 1, 2023
@fubinzh
Copy link

fubinzh commented Dec 1, 2023

/severity major

@dveeden
Copy link
Contributor Author

dveeden commented May 8, 2024

Some notes about how to reproduce this:

MariaDB

podman run --network=host --env MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 -it mariadb:10.6 \
--port 3307 \
--log-bin \
--binlog_format=ROW

TiDB

./bin/tidb-server

DM

./bin/dm-master -master-addr 127.0.0.1:8261
./bin/dm-worker -worker-addr 127.0.0.1:8262 -join 127.0.0.1:8261

DM Config

./bin/dmctl --master-addr 127.0.0.1:8261
operate-source create source-mariadb.yaml
start-task task-mariadb.yaml
query-status mariadb-to-tidb

Source

source-id: "mariadb-01"
from:
  host: "127.0.0.1"
  port: 3307
  user: "root"

Task

name: mariadb-to-tidb
mysql-instances:
  - source-id: "mariadb-01"
    block-allow-list: "test"
target-database:
  host: "127.0.0.1"
  port: 4000
  user: "root"
block-allow-list:
  test:
    do-dbs: ["*"]

ti-chi-bot bot pushed a commit that referenced this issue May 9, 2024
@dveeden
syncer(dm): Ignore SET statement for MariaDB (#10207)
6f0f2e3 
close #10206
@ti-chi-bot ti-chi-bot mentioned this issue Jul 30, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jul 31, 2024
@ti-chi-bot
syncer(dm): Ignore SET statement for MariaDB (#10207) (#11440)
1cbc3ed 
close #10206
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.3 affects-6.4 affects-7.1 affects-7.5 affects-8.1 area/dm Issues or PRs related to DM. MariaDB This issue is found with MariaDB upstream severity/moderate type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

syncer(dm): Ignore SET statement for MariaDB dveeden/tiflow
4 participants
@dveeden @lance6716 @fubinzh @ti-chi-bot