Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink: fix kafka max message size inaccurate issue. #3002

Merged
merged 32 commits into from
Oct 14, 2021
Merged

sink: fix kafka max message size inaccurate issue. #3002

merged 32 commits into from
Oct 14, 2021

Conversation

3AceShowHand
Copy link
Contributor

@3AceShowHand 3AceShowHand commented Oct 9, 2021 •
edited
Loading

What problem does this PR solve?

try to close #2962

What is changed and how it works?

  • When calculating the length of encoded MQMessage, also consider the maximumRecordOverhead, which was introduced by the sarama client.
  • Ignoring max-message-size at the producer level, default to 512m. max-message-size only works for the encoder.
  • Change log level to Debug when AppendRowChangedEvent in json encoder.

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)

Code changes
Side effects

Related changes

Release note 

fix kafka sink can not send message due to constraint by `max-message-size` option.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Oct 9, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 9, 2021
@ti-chi-bot ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 11, 2021
@ti-chi-bot ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 11, 2021
@ti-chi-bot ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 11, 2021
overvenus
overvenus reviewed Oct 11, 2021
View reviewed changes
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

cdc/sink/codec/interface.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 11, 2021
3AceShowHand and others added 2 commits October 11, 2021 15:17
@3AceShowHand
no need to have a mock broker.
9f50c1a
@3AceShowHand @overvenus
Update cdc/sink/codec/interface.go
fc37d0a 
Co-authored-by: Neil Shen <overvenus@gmail.com>
@3AceShowHand 3AceShowHand changed the title [DNM]sink: fix kafka max message size inaccurate issue. sink: fix kafka max message size inaccurate issue. Oct 11, 2021
@3AceShowHand
Copy link
Contributor Author

/run-all-tests

@3AceShowHand
Copy link
Contributor Author

/run-integration-tests

@codecov-commenter
Copy link

Codecov Report

Merging #3002 (c27a5eb) into master (3c3b915) will decrease coverage by 0.1649%.
The diff coverage is 51.3194%.

@@               Coverage Diff                @@
##             master      #3002        +/-   ##
================================================
- Coverage   57.2251%   57.0601%   -0.1650%     
================================================
  Files           163        188        +25     
  Lines         19453      20736      +1283     
================================================
+ Hits          11132      11832       +700     
- Misses         7261       7735       +474     
- Partials       1060       1169       +109

@amyangfei
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: c27a5eb

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Oct 14, 2021
@amyangfei
Copy link
Contributor

/run-kafka-tests

1 similar comment
@amyangfei
Copy link
Contributor

/run-kafka-tests

@ti-chi-bot ti-chi-bot merged commit b7c783f into pingcap:master Oct 14, 2021
@ti-chi-bot ti-chi-bot mentioned this pull request Oct 14, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3046.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Oct 14, 2021
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#3002
c93676b 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Oct 14, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3047.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3048.

This was referenced Oct 14, 2021
Merged
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3049.

ti-chi-bot added a commit that referenced this pull request Oct 14, 2021
@ti-chi-bot
sink: fix kafka max message size inaccurate issue. (#3002) (#3049)
1553069
ti-chi-bot added a commit that referenced this pull request Oct 15, 2021
@ti-chi-bot
sink: fix kafka max message size inaccurate issue. (#3002) (#3046)
5cb4dc7
ti-chi-bot added a commit that referenced this pull request Oct 15, 2021
@ti-chi-bot
sink: fix kafka max message size inaccurate issue. (#3002) (#3047)
9b5dbbc
@3AceShowHand 3AceShowHand mentioned this pull request Oct 15, 2021
Merged
ti-chi-bot added a commit that referenced this pull request Oct 15, 2021
@ti-chi-bot
sink: fix kafka max message size inaccurate issue. (#3002) (#3048)
e123b03
@3AceShowHand 3AceShowHand mentioned this pull request Oct 18, 2021
Merged
This was referenced Oct 20, 2021
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@overvenus overvenus overvenus approved these changes

@amyangfei amyangfei amyangfei approved these changes

@lance6716 lance6716 Awaiting requested review from lance6716

@liuzix liuzix Awaiting requested review from liuzix

Assignees
No one assigned
Labels
component/sink Sink component. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
v5.3.0
Development

Successfully merging this pull request may close these issues.

mq_sink: unexpected behavior of max-message-bytes
7 participants
@3AceShowHand @ti-chi-bot @overvenus @codecov-commenter @amyangfei @lance6716 @liuzix