Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hotfix] [connector-cdc-oracle ] support read partition table #8265

Merged
merged 6 commits into from
Dec 27, 2024

Conversation

zhangshenghang
Copy link
Member

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@hailin0
Copy link
Member

hailin0 commented Dec 10, 2024

waiting for this pr start ci
#8262

@github-actions github-actions bot added transform-v2 and removed cdc labels Dec 11, 2024
@dailai
Copy link
Contributor

dailai commented Dec 11, 2024

Please use rebase to merge the code

@github-actions github-actions bot added cdc and removed transform-v2 labels Dec 11, 2024
@zhangshenghang zhangshenghang marked this pull request as draft December 11, 2024 05:46
@zhangshenghang
Copy link
Member Author

@hailin0 @dailai @Carl-Zhou-CN @Hisoka-X

I encountered an issue where the update data in the e2e is:

> Insert delete can run successfully

image

After debugging and printing the logs, I found that Debezium modified it to the following:

update "DEBEZIUM"."PARTITION_SOURCE_TABLE" set "VAL_VARCHAR" = 'vc3' where "VAL_VARCHAR" = 'vc2';

The full content:


LogMinerEventRow{scn=2168886, tableId='ORCLCDB.DEBEZIUM.PARTITION_SOURCE_TABLE', tableName='PARTITION_SOURCE_TABLE', tablespaceName='DEBEZIUM', eventType=UPDATE, changeTime=2024-12-11T10:53:49Z, transactionId='0a000700fe040000', operation='UPDATE', userName='DEBEZIUM', rowId='AAAR4PAAHAAABGeAAA', rollbackFlag=false, rsId= 0x000006.0000264f.0010 , ssn=0, redoSql='update "DEBEZIUM"."PARTITION_SOURCE_TABLE" set "VAL_VARCHAR" = 'vc3' where "VAL_VARCHAR" = 'vc2';'}



However, for non-partitioned tables, Debezium can correctly capture the SQL.

image

I wonder if this is a bug in Debezium?

Flink CDC also does not support this feature, although there is PR, it has not been merged: apache/flink-cdc#2479

Looking forward to your advice!

@hailin0
Copy link
Member

hailin0 commented Dec 11, 2024

Try to start logminer query sql logs form sql cmd

ref
https://www.modb.pro/db/394933

@hailin0
Copy link
Member

hailin0 commented Dec 25, 2024

I tested everything is ok

image
2024-12-25 18:42:54,602 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0  rowIndex=4:  SeaTunnelRow#tableId=ORCL.QA_SOURCE.TEST_PARTITION SeaTunnelRow#kind=UPDATE_BEFORE : 1, 1, true
2024-12-25 18:42:54,602 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0  rowIndex=5:  SeaTunnelRow#tableId=ORCL.QA_SOURCE.TEST_PARTITION SeaTunnelRow#kind=UPDATE_AFTER : 1, 1, false
2024-12-25 18:43:44,499 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0  rowIndex=6:  SeaTunnelRow#tableId=ORCL.QA_SOURCE.TEST_PARTITION SeaTunnelRow#kind=UPDATE_BEFORE : 2, 1, true
2024-12-25 18:43:44,499 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0  rowIndex=7:  SeaTunnelRow#tableId=ORCL.QA_SOURCE.TEST_PARTITION SeaTunnelRow#kind=UPDATE_AFTER : 2, 1, false
2024-12-25 18:51:30,481 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0  rowIndex=8:  SeaTunnelRow#tableId=ORCL.QA_SOURCE.TEST_PARTITION SeaTunnelRow#kind=DELETE : 1, 1, false

@zhangshenghang zhangshenghang marked this pull request as ready for review December 26, 2024 12:01
Copy link
Member

@hailin0 hailin0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if ci passes

@hailin0 hailin0 merged commit 91b86b2 into apache:dev Dec 27, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] [connector-cdc-oracle ] Unsupport read partition table
4 participants