Skip to content
This repository has been archived by the owner on Sep 2, 2024. It is now read-only.

[main] KafkaChannel to init offsets before dispatcher #911

Merged
merged 7 commits into from
Oct 11, 2021

Conversation

aliok
Copy link
Member

@aliok aliok commented Oct 1, 2021

Manual cherry pick of the same fix in 0.24: #886

Release Note

- 🐛  Fixed a bug that was causing Kafka Channel dispatcher to miss some events in case the offset for the consumer group was not initialized

Docs

@google-cla google-cla bot added the cla: yes Indicates the PR's author has signed the CLA. label Oct 1, 2021
@knative-prow-robot knative-prow-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 1, 2021
@codecov
Copy link

codecov bot commented Oct 1, 2021

Codecov Report

Merging #911 (598969b) into main (9fe562d) will decrease coverage by 0.48%.
The diff coverage is 44.44%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #911      +/-   ##
==========================================
- Coverage   76.00%   75.51%   -0.49%     
==========================================
  Files         139      137       -2     
  Lines        6775     6554     -221     
==========================================
- Hits         5149     4949     -200     
+ Misses       1345     1320      -25     
- Partials      281      285       +4     
Impacted Files Coverage Δ
...l/consolidated/reconciler/controller/controller.go 0.00% <0.00%> (ø)
...g/common/consumer/consumergroup_offsets_checker.go 4.16% <4.16%> (ø)
pkg/channel/consolidated/dispatcher/dispatcher.go 64.13% <33.33%> (+2.54%) ⬆️
...consolidated/reconciler/controller/kafkachannel.go 47.56% <37.03%> (-2.12%) ⬇️
pkg/common/consumer/consumer_factory.go 78.00% <45.45%> (-7.37%) ⬇️
pkg/common/kafka/offset/offsets.go 59.18% <63.63%> (+0.36%) ⬆️
pkg/source/adapter/adapter.go 54.16% <66.66%> (-0.58%) ⬇️
.../distributed/dispatcher/controller/kafkachannel.go 85.12% <100.00%> (+0.25%) ⬆️
...el/distributed/dispatcher/dispatcher/dispatcher.go 96.77% <100.00%> (ø)
pkg/common/consumer/consumer_handler.go 54.83% <100.00%> (ø)
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fe562d...598969b. Read the comment docs.

@matzew
Copy link
Contributor

matzew commented Oct 1, 2021

/hold

let's use 🍒-pick of the "merge sha" ...

@knative-prow-robot knative-prow-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 1, 2021
@aliok
Copy link
Member Author

aliok commented Oct 1, 2021

let's use 🍒-pick of the "merge sha" ...

If it works yes, that's nicer.

@google-cla
Copy link

google-cla bot commented Oct 7, 2021

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added cla: no Indicates the PR's author has not signed the CLA. and removed cla: yes Indicates the PR's author has signed the CLA. labels Oct 7, 2021
@pierDipi
Copy link
Member

pierDipi commented Oct 7, 2021

@googlebot I consent.

@google-cla google-cla bot added cla: yes Indicates the PR's author has signed the CLA. and removed cla: no Indicates the PR's author has not signed the CLA. labels Oct 7, 2021
@aliok
Copy link
Member Author

aliok commented Oct 7, 2021

Let's see this one passes the tests then I will cherry pick the changes and do the dance from 0.24 -> ... -> main

aliok and others added 4 commits October 8, 2021 13:41
Signed-off-by: Pierangelo Di Pilato <pdipilat@redhat.com>
Signed-off-by: Pierangelo Di Pilato <pdipilat@redhat.com>
@pierDipi
Copy link
Member

pierDipi commented Oct 8, 2021

/test pull-knative-sandbox-eventing-kafka-upgrade-tests

@aliok
Copy link
Member Author

aliok commented Oct 8, 2021

I rebased this on top of main head now

@pierDipi
Copy link
Member

pierDipi commented Oct 8, 2021

This is green again ...

@aliok
Copy link
Member Author

aliok commented Oct 8, 2021

Ok, let's see what happens to the 0.26 backport: #928

@aliok
Copy link
Member Author

aliok commented Oct 11, 2021

/hold

Some renaming to be done here on this PR

@knative-metrics-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-knative-sandbox-eventing-kafka-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/channel/consolidated/dispatcher/dispatcher.go 64.5% 66.1% 1.6
pkg/channel/consolidated/reconciler/controller/kafkachannel.go 54.8% 54.3% -0.5
pkg/channel/distributed/dispatcher/controller/kafkachannel.go 87.5% 87.6% 0.1
pkg/common/consumer/consumer_factory.go 87.2% 81.2% -5.9
pkg/common/consumer/consumer_manager.go 96.9% 96.9% 0.0
pkg/common/consumer/consumergroup_offsets_checker.go Do not exist 4.3%
pkg/common/kafka/offset/offsets.go 81.1% 81.5% 0.4

@knative-prow-robot
Copy link
Contributor

knative-prow-robot commented Oct 11, 2021

@aliok: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-knative-sandbox-eventing-kafka-go-coverage 598969b link false /test pull-knative-sandbox-eventing-kafka-go-coverage

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@aliok
Copy link
Member Author

aliok commented Oct 11, 2021

/test pull-knative-sandbox-eventing-kafka-integration-test-channel-distributed

Unable to connect to the server: net/http: TLS handshake timeout

Copy link
Contributor

@matzew matzew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@knative-prow-robot knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 11, 2021
@knative-prow-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aliok, matzew

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@aliok
Copy link
Member Author

aliok commented Oct 11, 2021

/unhold

@knative-prow-robot knative-prow-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 11, 2021
@knative-prow-robot knative-prow-robot merged commit a8f8dec into knative-extensions:main Oct 11, 2021
@aliok aliok deleted the kafka-ch-event-loss branch October 11, 2021 08:46
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cla: yes Indicates the PR's author has signed the CLA. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants