Err: table id xxx: invalid table id, no corresponding table map event

[GMHDBJD]

What did you do?

start task with relay

What did you expect to see?

task runs as normal

What did you see instead?

Err: table id xxx: invalid table id, no corresponding table map event

#5731 (comment)

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

nightly

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)

[GMHDBJD]

task will retry with remote binlog

[lance6716]

[2022-08-01T15:08:24.870Z]                         "errors": [

[2022-08-01T15:08:24.870Z]                             {

[2022-08-01T15:08:24.870Z]                                 "ErrCode": 36069,

[2022-08-01T15:08:24.870Z]                                 "ErrClass": "sync-unit",

[2022-08-01T15:08:24.870Z]                                 "ErrScope": "upstream",

[2022-08-01T15:08:24.870Z]                                 "ErrLevel": "high",

[2022-08-01T15:08:24.870Z]                                 "Message": "get binlog event error: [code=11038:class=functional:scope=internal:level=high], Message: parse relay log file dm-integration-test-go1180-8458-vlf20-gch73-bin.000001 from offset 4 in dir /tmp/dm_test/sequence_sharding_optimistic/worker2/relay_log/7ad31c83-11a9-11ed-8db2-c2cb83fdf274.000001: parse relay log file dm-integration-test-go1180-8458-vlf20-gch73-bin.000001 from offset 3681 in dir /tmp/dm_test/sequence_sharding_optimistic/worker2/relay_log/7ad31c83-11a9-11ed-8db2-c2cb83fdf274.000001: parse relay log file /tmp/dm_test/sequence_sharding_optimistic/worker2/relay_log/7ad31c83-11a9-11ed-8db2-c2cb83fdf274.000001/dm-integration-test-go1180-8458-vlf20-gch73-bin.000001, RawCause: Header \u0026replication.EventHeader{Timestamp:0x62e7ec4d, EventType:0x1e, ServerID:0x1, EventSize:0x3a, LogPos:0xe61, Flags:0x0}, Data \"k\\x02\\x00\\x00\\x00\\x00\\x01\\x00\\x02\\x00\\x03\\xff\\x00\\x84\\x1a\\x06\\x00\\x00\\x00\\x00\\x00\\x06410004\\x06420004\", Err: table id 619: invalid table id, no corresponding table map event",

[2022-08-01T15:08:24.870Z]                                 "RawCause": "",

[2022-08-01T15:08:24.870Z]                                 "Workaround": "Please check if the binlog file could be parsed by `mysqlbinlog`."

[2022-08-01T15:08:24.870Z]                             }

[2022-08-01T15:08:24.870Z]                         ],

https://ci2.pingcap.net/blue/organizations/jenkins/dm_ghpr_integration_test/detail/dm_ghpr_integration_test/8458/pipeline

[lance6716]

[2022-08-01T15:08:24.870Z]                         "errors": [

[2022-08-01T15:08:24.870Z]                             {

[2022-08-01T15:08:24.870Z]                                 "ErrCode": 36069,

[2022-08-01T15:08:24.870Z]                                 "ErrClass": "sync-unit",

[2022-08-01T15:08:24.870Z]                                 "ErrScope": "upstream",

[2022-08-01T15:08:24.870Z]                                 "ErrLevel": "high",

[2022-08-01T15:08:24.870Z]                                 "Message": "get binlog event error: [code=11038:class=functional:scope=internal:level=high], Message: parse relay log file dm-integration-test-go1180-8458-vlf20-gch73-bin.000001 from offset 4 in dir /tmp/dm_test/sequence_sharding_optimistic/worker2/relay_log/7ad31c83-11a9-11ed-8db2-c2cb83fdf274.000001: parse relay log file dm-integration-test-go1180-8458-vlf20-gch73-bin.000001 from offset 3681 in dir /tmp/dm_test/sequence_sharding_optimistic/worker2/relay_log/7ad31c83-11a9-11ed-8db2-c2cb83fdf274.000001: parse relay log file /tmp/dm_test/sequence_sharding_optimistic/worker2/relay_log/7ad31c83-11a9-11ed-8db2-c2cb83fdf274.000001/dm-integration-test-go1180-8458-vlf20-gch73-bin.000001, RawCause: Header \u0026replication.EventHeader{Timestamp:0x62e7ec4d, EventType:0x1e, ServerID:0x1, EventSize:0x3a, LogPos:0xe61, Flags:0x0}, Data \"k\\x02\\x00\\x00\\x00\\x00\\x01\\x00\\x02\\x00\\x03\\xff\\x00\\x84\\x1a\\x06\\x00\\x00\\x00\\x00\\x00\\x06410004\\x06420004\", Err: table id 619: invalid table id, no corresponding table map event",

[2022-08-01T15:08:24.870Z]                                 "RawCause": "",

[2022-08-01T15:08:24.870Z]                                 "Workaround": "Please check if the binlog file could be parsed by `mysqlbinlog`."

[2022-08-01T15:08:24.870Z]                             }

[2022-08-01T15:08:24.870Z]                         ],

https://ci2.pingcap.net/blue/organizations/jenkins/dm_ghpr_integration_test/detail/dm_ghpr_integration_test/8458/pipeline

in this integration test, task are frequently paused and resumed by failpoint, "relay log switch to remote binlog" has a cooldown so the error remains.

[lance6716]

For the cause of "invalid table id", found that

[2022-08-01T15:07:56.657Z] restart dm-worker1

[2022-08-01T15:07:56.657Z] process dm-worker1 already exit

[2022-08-01T15:07:56.657Z] [Mon Aug  1 23:07:56 CST 2022] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/tests/sequence_sharding_optimistic/conf/dm-worker1.toml >>>>>>

but

dm-worker exit 2022-08-01 23:08:00.049842502 +0800 CST m=+12.956042139

Worker exits too slowly, there're two writers to write relay log