Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink(ticdc): set max-message-bytes default to 10m (#4036) #4061

Closed
wants to merge 37 commits into from
Closed

sink(ticdc): set max-message-bytes default to 10m (#4036) #4061

wants to merge 37 commits into from

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #4036

What problem does this PR solve?

close #4041
set Kafka producer's default max-message-bytes to 10M, and use the minimum value among sink-uri's max-message-bytes, broker's message.max.bytes, topic's max.message.bytes to initialize the producer.

What is changed and how it works?

  • set the default value of max-message-bytes to 10M
  • use the min value of the 3 mentioned above to initialize the producer.

Check List

Tests

  • Unit test

Code changes

Side effects

Related changes

Release note 

Set `max-message-bytes` default to 10M, and use the min value with topic and broker to initialize the producer.

3AceShowHand and others added 3 commits November 22, 2021 05:33
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#3192
0187fdd 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@3AceShowHand
fix conflicts.
e1a50d6
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#4036
806ceae 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member Author

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/cherry-pick-not-approved release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/LGT2 Indicates that a PR has LGTM 2. type/cherry-pick-for-release-5.1 This PR is cherry-picked to release-5.1 from a source PR. labels Dec 24, 2021
@ti-chi-bot ti-chi-bot mentioned this pull request Dec 24, 2021
Merged
@ti-chi-bot ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 24, 2021
3AceShowHand and others added 17 commits December 24, 2021 23:49
@3AceShowHand
remove craft.
f51e1a9
@3AceShowHand
remove useless files.
1996bd6
@3AceShowHand
fix conflicts in json.
9dd01b8
@ti-chi-bot @3AceShowHand
test(ticdc): separate statistics for unit tests and integration tests (…
2913f7b 
…pingcap#3570) (pingcap#3579)
@ti-chi-bot @3AceShowHand
ticdc/metrics: fix processor checkpoint-ts lag expr (pingcap#3543) (p…
9f3f8cf 
…ingcap#3559)
@ti-chi-bot @3AceShowHand
test: add new ci collation without old value test (pingcap#3114) (pin…
6fad381 
…gcap#3368)
@ti-chi-bot @3AceShowHand
processor,sink(cdc): let sink report resolved ts and do not skip buff…
3136a20 
…er sink flush (pingcap#3540) (pingcap#3563)
@ti-chi-bot @3AceShowHand
integration_tests(ticdc): move integration tests into integration_tes…
d928a2f 
…ts dir (pingcap#3646) (pingcap#3690)
@ti-chi-bot @3AceShowHand
sink(ticdc): Set config.Metadata.Timeout correctly (pingcap#3665) (pi…
50545de 
…ngcap#3671)
@ti-chi-bot @3AceShowHand
*: fix changefeed checkpoint lag negative value error (pingcap#3013) (p…
47b2b2d 
…ingcap#3534)
@ti-chi-bot @3AceShowHand
ticdc.rules: fix alertmanager rules (pingcap#3422) (pingcap#3595)
666ac9e
@ti-chi-bot @3AceShowHand
Fix handling of JSON columns (pingcap#3643) (pingcap#3652)
643fb1b
@ti-chi-bot @3AceShowHand
workerpool: limit the rate to output deadlock warning (pingcap#3775) (p…
974aaa1 
…ingcap#3798)
@ti-chi-bot @3AceShowHand
ticdc/alert: add no owner alert rule (pingcap#3809) (pingcap#3835)
e5155ed
@ti-chi-bot @3AceShowHand
pkg,cdc: do not use log package (pingcap#3902) (pingcap#3943)
daf1adb
@amyangfei @3AceShowHand
*: rename repo from pingcap/ticdc to pingcap/tiflow (pingcap#3956)
ab1661e
@ti-chi-bot @3AceShowHand
tz (ticdc): fix timezone error (pingcap#3887) (pingcap#3909)
f7a8919
ti-chi-bot and others added 12 commits December 25, 2021 23:11
@ti-chi-bot @3AceShowHand
kvclient(ticdc): fix kvclient takes too long time to recover (pingcap…
92b5021 
…#3612) (pingcap#3661)
@ti-chi-bot @3AceShowHand
etcd_worker: batch etcd patch (pingcap#3277) (pingcap#3392)
fea07be
@ti-chi-bot @3AceShowHand
Clean old owner and old processor in release 5.2 branch (pingcap#4019) (
b5178e6 
pingcap#4023)
@ti-chi-bot @3AceShowHand
tests(ticdc): set up the sync diff output directory correctly (pingca…
4ef3deb 
…p#3725) (pingcap#3744)
@ti-chi-bot @3AceShowHand
owner,scheduler(cdc): fix nil pointer panic in owner scheduler (pingc…
af3c44f 
…ap#2980) (pingcap#4007) (pingcap#4014)
@ti-chi-bot @3AceShowHand
config(ticdc): Fix old value configuration check for maxwell protocol (…
f51db06 
…pingcap#3747) (pingcap#3781)
@3AceShowHand
This is an automated cherry-pick of pingcap#4036
8b5360c 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@3AceShowHand
remove craft.
f35ae50
@3AceShowHand
remove useless files.
9da3fb7
@3AceShowHand
fix conflicts in json.
380feac
@3AceShowHand
fix conflict.
e3b5105
@3AceShowHand
remove config.
803c407
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 25, 2021
@3AceShowHand
Copy link
Contributor

/run-all-tests

@3AceShowHand 3AceShowHand mentioned this pull request Dec 27, 2021
Merged
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 27, 2021
@overvenus overvenus added this to the v5.1.4 milestone Jan 6, 2022
@VelocityLight VelocityLight added the cherry-pick-approved Cherry pick PR approved by release team. label Jan 19, 2022
@ti-chi-bot ti-chi-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 20, 2022
@3AceShowHand
Copy link
Contributor

merged by #4077

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@3AceShowHand 3AceShowHand

Labels
cherry-pick-approved Cherry pick PR approved by release team. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. status/LGT2 Indicates that a PR has LGTM 2. type/cherry-pick-for-release-5.1 This PR is cherry-picked to release-5.1 from a source PR.
Projects
None yet
Milestone
v5.1.4
Development

Successfully merging this pull request may close these issues.
5 participants
@ti-chi-bot @3AceShowHand @overvenus @VelocityLight @amyangfei