Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink(ticdc): add event key output for debezium #11649

Merged
merged 17 commits into from
Oct 21, 2024
Merged

sink(ticdc): add event key output for debezium #11649

merged 17 commits into from
Oct 21, 2024

Conversation

wk989898
Copy link
Collaborator

@wk989898 wk989898 commented Oct 11, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #11652

What is changed and how it works?

Debezium send event key and event value to kafka. I added event key output to this PR.
Also modify function SanitizeName for keeping same with official code.

Check List

Tests

  • Unit test
  • Integration test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

Fix the issue that the Key of Kafka messages has't been filled in Debezium protocol.

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 11, 2024
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 11, 2024
@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/needs-triage-completed and removed release-note-none Denotes a PR that doesn't merit a release note. labels Oct 12, 2024
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 12, 2024
@codecov Codecov
Copy link

codecov bot commented Oct 12, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 24.44444% with 238 lines in your changes missing coverage. Please review.

Project coverage is 55.1512%. Comparing base (cd277f5) to head (2dc3baf).
Report is 6 commits behind head on master.

Additional details and impacted files
Components Coverage Δ
cdc 59.6168% <24.4444%> (-0.2386%) ⬇️
dm 50.0616% <ø> (-0.0261%) ⬇️
engine 53.2336% <ø> (-0.0087%) ⬇️
Flag Coverage Δ
unit 55.1512% <24.4444%> (-0.1186%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #11649        +/-   ##
================================================
- Coverage   55.2697%   55.1512%   -0.1186%     
================================================
  Files          1001       1001                
  Lines        136163     136472       +309     
================================================
+ Hits          75257      75266         +9     
- Misses        55417      55697       +280     
- Partials       5489       5509        +20

@wk989898 wk989898 changed the title sink(ticdc): add key message output on debezium sink(ticdc): add event key output for debezium Oct 14, 2024
hongyunyan
hongyunyan reviewed Oct 16, 2024
View reviewed changes
tests/integration_tests/debezium/sql/debezium/real_test.sql
@@ -1,3 +1,5 @@
-- set sql_mode REAL_AS_FLOAT is necessary
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why set REAL_AS_FLOAT is neccesary?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because Debezium treats REAL as a synonym for FLOAT.

tiflow/pkg/sink/codec/debezium/codec.go

Lines 385 to 386 in 45b439b

// https://dev.mysql.com/doc/refman/8.4/en/numeric-types.html
// MySQL also treats REAL as a synonym for DOUBLE PRECISION (a nonstandard variation), unless the REAL_AS_FLOAT SQL mode is enabled.

pkg/sink/codec/debezium/codec.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Oct 17, 2024
@wk989898
Copy link
Collaborator Author

/retest

3 similar comments
@wk989898
Copy link
Collaborator Author

/retest

@wk989898
Copy link
Collaborator Author

/retest

@wk989898
Copy link
Collaborator Author

/retest

@wk989898
Copy link
Collaborator Author

/retest

hongyunyan
hongyunyan approved these changes Oct 21, 2024
View reviewed changes
Copy link
Collaborator

@hongyunyan hongyunyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot bot added the lgtm label Oct 21, 2024
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 21, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, hongyunyan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [3AceShowHand,hongyunyan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Oct 21, 2024
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 21, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-10-17 02:52:44.780260683 +0000 UTC m=+498761.929170515: ☑️ agreed by 3AceShowHand.
  • 2024-10-21 06:24:05.975844173 +0000 UTC m=+245846.672634802: ☑️ agreed by hongyunyan.

@ti-chi-bot ti-chi-bot bot merged commit 76ebb37 into pingcap:master Oct 21, 2024
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

@hongyunyan hongyunyan hongyunyan approved these changes

@breezewish breezewish Awaiting requested review from breezewish

Assignees

@wk989898 wk989898

Labels
approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

The Key field is not filled in the Kafka event of Debezium protocol
3 participants
@wk989898 @3AceShowHand @hongyunyan