Skip to content

Conversation

@tejanshrana
Copy link
Contributor

@tejanshrana tejanshrana commented Nov 4, 2025

Transaction BEGIN/END events (when provide.transaction.metadata is enabled) are currently treated as unknown and skipped, causing loss of transaction boundaries.

Solution

  • Detect transaction metadata events via key schema name
  • Update binlog offset when processing these events
  • Emit events to downstream consumers if includeTransactionMetadataEvents() is set to true (default = false)

Logs from before the change:
2025-11-04 14:52:28 2025-11-04 14:52:28,827 INFO org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter [] - Meet unknown element SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=4541146d-b988-11f0-87f6-0242ac140006:33, ts_sec=1762267948, file=mysql-bin.000003, pos=13381, gtids=4541146d-b988-11f0-87f6-0242ac140006:1-32, server_id=1}} ConnectRecord{topic='mysql_binlog_source.transaction', kafkaPartition=null, key=Struct{id=4541146d-b988-11f0-87f6-0242ac140006:33}, keySchema=Schema{io.debezium.connector.common.TransactionMetadataKey:STRUCT}, value=Struct{status=BEGIN,id=4541146d-b988-11f0-87f6-0242ac140006:33}, valueSchema=Schema{io.debezium.connector.common.TransactionMetadataValue:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}, just skip.

After the change, the transaction start and end events look like below:
{"status":"BEGIN","id":"4541146d-b988-11f0-87f6-0242ac140006:47","event_count":null,"data_collections":null} {"status":"END","id":"4541146d-b988-11f0-87f6-0242ac140006:47","event_count":1,"data_collections":[{"data_collection":"sample.sample","event_count":1}]}

@tejanshrana tejanshrana changed the title [PROPOSAL] [cdc connector mysql] Support for MySQL Transaction Boundary Events in Flink CDC Connector [FLINK-38691] [cdc connector mysql] Support for MySQL Transaction Boundary Events in Flink CDC Connector Nov 24, 2025
@tejanshrana tejanshrana marked this pull request as ready for review November 24, 2025 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants