Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perform max.poll.interval.ms considering max.poll.records #2432

Merged
merged 1 commit into from
Jul 28, 2022
Merged

Perform max.poll.interval.ms considering max.poll.records #2432

merged 1 commit into from
Jul 28, 2022

Conversation

pierDipi
Copy link
Member

Without considering max.poll.records in the calculation of
the max.poll.interval.ms we might cause unnecessary rebalances
since without this patch slow consumers might trigger a rebalance
at every handled record (when their processing time ~= timeout).

Signed-off-by: Pierangelo Di Pilato pierdipi@redhat.com

When using ordered delivery with slow consumers, unnecessary consumer group rebalances might be triggered, which causes higher e2e latency and increased duplicates.

/kind bug

@knative-prow knative-prow bot added kind/bug Categorizes issue or PR as related to a bug. area/data-plane approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 28, 2022
@codecov
Copy link

codecov bot commented Jul 28, 2022 •
edited
Loading

Codecov Report

Merging #2432 (581fd58) into main (d536fe1) will decrease coverage by 0.01%.
The diff coverage is 88.88%.

@@             Coverage Diff              @@
##               main    #2432      +/-   ##
============================================
- Coverage     64.60%   64.58%   -0.02%     
+ Complexity      706      705       -1     
============================================
  Files           146      146              
  Lines          9783     9786       +3     
  Branches        215      216       +1     
============================================
  Hits           6320     6320              
- Misses         3038     3039       +1     
- Partials        425      427       +2
Flag Coverage Δ
java-unittests 81.39% <88.88%> (-0.10%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...r/dispatcher/main/ConsumerVerticleFactoryImpl.java 79.16% <88.88%> (-0.98%) ⬇️
...broker/dispatcher/impl/consumer/OffsetManager.java 95.69% <0.00%> (-1.08%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us.

@pierDipi
Copy link
Member Author

/cc @aliok @aavarghese @aslom

@knative-prow knative-prow bot requested review from aavarghese, aliok and aslom July 28, 2022 10:43
@pierDipi pierDipi force-pushed the max-processing-time-based-on-max-records branch from 2993a8c to 9d6a765 Compare July 28, 2022 10:44
@pierDipi
Perform max.poll.interval.ms considering max.poll.records
581fd58 
Without considering `max.poll.records` in the calculation of
the `max.poll.interval.ms` we might cause unnecessary rebalances
since without this patch slow consumers might trigger a rebalance
at every handled record (when their processing time ~= timeout).

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
@pierDipi pierDipi force-pushed the max-processing-time-based-on-max-records branch from 9d6a765 to 581fd58 Compare July 28, 2022 10:47
@knative-prow knative-prow bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 28, 2022
@pierDipi
Copy link
Member Author

/cherry-pick release-1.6

@knative-prow-robot
Copy link
Contributor

@pierDipi: once the present PR merges, I will cherry-pick it on top of release-1.6 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@pierDipi
Copy link
Member Author

/retest-required

1 similar comment
@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/test unit-tests_eventing-kafka-broker_main

@pierDipi
Copy link
Member Author

/retes-required

@pierDipi
Copy link
Member Author

/retest-required

3 similar comments
@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/retest-required

@aavarghese
Copy link
Contributor

@pierDipi @aslom since we're computing the max.poll.interval.ms value for both Ordered and Unordered, that implies this parameter in consumer properties will get overridden. Should we add this info as comment or remove that parameter?

@pierDipi
Copy link
Member Author

@pierDipi @aslom since we're computing the max.poll.interval.ms value for both Ordered and Unordered, that implies this parameter in consumer properties will get overridden. Should we add this info as comment or remove that parameter?

+1, let's work on that in a separate issue since it's an existing behavior.

@pierDipi
Copy link
Member Author

/retest-required

aavarghese
aavarghese approved these changes Jul 28, 2022
View reviewed changes
Copy link
Contributor

@aavarghese aavarghese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests in-progress....

/lgtm

@knative-prow knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jul 28, 2022
@knative-prow
Copy link

knative-prow bot commented Jul 28, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aavarghese, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pierDipi
Copy link
Member Author

/retest-required

@knative-prow knative-prow bot merged commit 72299ab into knative-extensions:main Jul 28, 2022
@knative-prow-robot knative-prow-robot mentioned this pull request Jul 28, 2022
Merged
@knative-prow-robot
Copy link
Contributor

@pierDipi: new pull request created: #2439

In response to this:

/cherry-pick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aavarghese aavarghese aavarghese approved these changes

@aliok aliok Awaiting requested review from aliok

@aslom aslom Awaiting requested review from aslom

Assignees

@aavarghese aavarghese

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/data-plane kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pierDipi @knative-prow-robot @aavarghese