Avoid waiting for offsets initialization in dispatcher #382
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Knative Automation <automation@knative.team>
bumping knative.dev/pkg 6040b3a...0d31134: > 0d31134 Fix nil pointer panic in kvstore (# 2002) bumping knative.dev/eventing 81ce898...ea452b5: > ea452b5 go 1.15 in go.mod, drop k8s 1.17 e2e tests (# 4795) > ce099cf new cloudevent extension attribute with dispatch error (# 4760) > 83ecc7c Set failure policy to ignore for webhook (# 4793) > ae17ca3 Adding HPA for eventing-webhook (# 4792) Signed-off-by: Knative Automation <automation@knative.team>
bumping knative.dev/eventing ea452b5...1725902: > 1725902 Update COMMUNITY_CONTACTS.md test grid link (# 4798) > b75d03a Adding new source conformance testcase for CRD Source Registry Spec (# 4780) > 77fc350 upgrade to latest dependencies (# 4797) Signed-off-by: Knative Automation <automation@knative.team>
Signed-off-by: Matthias Wessendorf <mwessend@redhat.com>
* Remove 400 from the list of reasons to retry an event. * Updated explanatory comment for retries
bumping knative.dev/eventing 1725902...b20c96b: > b20c96b sinkbing implement bindable (# 4794) > 8d51418 Bump golang linter version and timeout (# 4808) > f882ff0 Add availability rate to upgrade test report (# 4555) > a6635bb Patch HPA eventing-webhook min replicas (# 4811) > d46c1d0 It is already 2021 (# 4809) > fb8fbf7 Adding new source conformance testcase for CRD RBAC (# 4787) bumping knative.dev/pkg 0d31134...ca02ef7: > ca02ef7 Genreconciler properly generates reconciler for Resources with Status (# 2004) Signed-off-by: Knative Automation <automation@knative.team>
* multi-tenant KafkaSource * initial scheduler * added autoscaler * recompute state each time a scheduling is requested * remove config * fix unit tests * fix vendored files * allow statefulsets/scale to be read and updated * eviction implemented * add rate limiter * run codegen/update-deps * rename schedulable to vpod * get some ut in there * fix typo * fix ut data race * moar ut * make sure lint is happy * fix config files * changes after review * make lgtm happy * update copyright year and softlink some config files * fix goimport * update deps * fix config files * isolate the mt kafka source implementation * more tests * fix roles * revert single roles * revert single config * run goimport * run update-deps * miss one file
* Update Owners * only add myself to reviewer list
bumping knative.dev/eventing b20c96b...321a839: > 321a839 Format markdown (# 4823) > ec63881 Fix spec # 4515 (# 4654) > 1528750 upgrade to latest dependencies (# 4818) > 9dc57e2 Update common github actions (# 4817) Signed-off-by: Knative Automation <automation@knative.team>
bumping knative.dev/eventing 321a839...d58bd38: > d58bd38 fix the link to head (# 4826) > 39d7a9e remove unused functions (# 4824) > 6edcbdd Panic in shared main to propagate error to exit code (# 4820) > 012a9bd v1beta1.Trigger delivery should be v1.DeliverySpec (# 4822) Signed-off-by: Knative Automation <automation@knative.team>
…nsions#350) * e2e source tests can now run against the mt source impl * rerun codegen.
Signed-off-by: Francesco Guardiani <francescoguard@gmail.com>
Signed-off-by: Francesco Guardiani <francescoguard@gmail.com>
Signed-off-by: Knative Automation <automation@knative.team>
…sions#366) * Increase idle connection limits from 100/2 to 1000/100 * go imports
* Fix knative-extensions#342 Signed-off-by: Francesco Guardiani <francescoguard@gmail.com> * Commit mistake Signed-off-by: Francesco Guardiani <francescoguard@gmail.com>
…native-extensions#344) * Groundwork for dispatcher subscriber status w/ consumers * Add functionality to serve http of subscribers * Drop callback functions * Fix failing unit test, add unsub check for chanref * Rework http handler to be dispatcher local (not kafkasubscription) * Variable typo fix * Fix copyright years * Change header name to constant * Move subscription handler to its own ServeHTTP func * Remove channelRef in KafkaSubscription * Change bad channelref request to http.StatusNotFound * Add namespace to subscriptions http output * Add Unit tests for servehttp & setready * Split uriSplit into channelRefName{,space} vars * Expose dispatcher http-sub-status port in disatcher svc * Add servehttp diagnostic messages * One more uriSplit -> channelRefName variable rename * Change how we write the http response * Add empty SetReady() method to source RA * Fix consumer_handler_test * more linting * Add back ObserveKind until controller implements substatus scraper * Add more ServeHTTP unit tests * slightly alter where we mark a handler as ready or not
…certs (knative-extensions#359) * Adding check that requires us to just enable TLS, no need for custom certs Signed-off-by: Matthias Wessendorf <mwessend@redhat.com> * Bit of refactorings Signed-off-by: Matthias Wessendorf <mwessend@redhat.com> * Formatting Signed-off-by: Matthias Wessendorf <mwessend@redhat.com>
…native-extensions#371) Follow the eventing naming scheme which doesn't have a '_' in the prefix
bumping knative.dev/pkg ca02ef7...a02dcff: > a02dcff Bump a few assorted dependencies to their latest versions (# 2013) > 4b2ae07 Replace deprecated github.com/markbates/inflect with github.com/gobuffalo/flect (# 2014) > 8878069 upgrade to latest dependencies (# 2010) > c493a9e Update common github actions (# 2009) > 6045ed4 Allow setting DisableCompression in NewAutoTransport (# 2007) bumping go.opencensus.io 3fb168f...13369a4: > 13369a4 Adds an exported function to flush internal reader (# 1248) > e736602 fix memory leak cause by the spanStore.(census-instrumentation/opence… (# 1246) > 61c7a2b Remove Travis CI for GH Actions. (# 1244) > 1f9ef3b Set up Github Actions for pull request validation (# 1237) > 4ee1fc6 Upgrade all dependencies (# 1243) > ff2df8e Lock only when needed, remove duplicate code (# 1242) > 380f407 provide accessor to the span implementation (# 1240) > 0f7c5f5 allow replaceing trace SDK (# 1238) > de37041 Revert "Allow replacing trace SDK (# 1234)" (# 1235) > fc3822b Allow replacing trace SDK (# 1234) bumping knative.dev/eventing d58bd38...37eaf71: > 37eaf71 switch Harwayne with antoineco (# 4850) > b887ac4 upgrade to latest dependencies (# 4848) > daa085d Format markdown (# 4846) > 3d51c72 Change trigger spec to indicate assigned broker should be immutable (# 4828) > 1774243 Add immutable fields validation to v1beta1 broker (# 4816) > 935d5fb Format markdown (# 4840) > 269b061 Update common github actions (# 4839) > 152e608 upgrade to latest dependencies (# 4841) > 8171448 Adding Logline for when a cloudevent has been successfully sent from APIServerSource to the sink (# 4723) > fd2688d Use SHOULD for TriggerSpec to keep compatibility (# 4838) > 1fa597e [# 4796] Bump golang lint timeout to 10m (# 4830) > a4a9f48 Format markdown (# 4829) bumping github.com/google/uuid 0e4e311...bfb86fa: > bfb86fa Add NewString Func (# 73) > b5d0d36 hash.go hash error covered and linter error fixed (# 71) > 9377784 Update sql.go (# 72) > 85223fa Reduce custom error allocation (# 70) > edef28d Use a custom error type for invalid lengths, replacing `fmt.Errorf` (# 69) bumping knative.dev/hack 8d623a0...8368e1f: > 8368e1f guard against set -o unset (# 49) > 2b4f6fc disable go's proxy and sumdb only for knative deps (# 47) Signed-off-by: Knative Automation <automation@knative.team>
bumping knative.dev/eventing 37eaf71...7ce8283: > 7ce8283 fix(PingSource)!: event consumer should receive base64-decoded dataBase64 (# 4851) > c7a114c Format markdown (# 4857) > f777623 Remove Harwayne from OWNERS. (# 4860) > f1d55ad remove json-specific logic useless for binary mode (# 4858) > 83da202 💄 Latest of kindst image for e2e tests w/ 1.20.2 (# 4854) > 3fcb645 🌀 Remvoing contrib refs (# 4855) > f05561d upgrade to latest dependencies (# 4853) Signed-off-by: Knative Automation <automation@knative.team>
Signed-off-by: Knative Automation <automation@knative.team>
…native-extensions#376) Change the validation of the kafkasource api to allow for changing the topics/servers after creation
bumping knative.dev/pkg a02dcff...e38bb89: > e38bb89 Update common github actions (# 2017) bumping knative.dev/eventing 7ce8283...1c771ce: > 1c771ce Change broker spec to indicate brokerclass should be immutable (# 4827) > d7cf344 tracing fixed for apiserversource (# 4842) > 1efec2b expose ChannelReference in UnknownChannelError (# 4875) > 47d7a02 Update deps (# 4874) > 47ee8fa Update common github actions (# 4869) > c4e9c3b ut for trigger/broker updates check immutable (# 4843) Signed-off-by: Knative Automation <automation@knative.team>
* Initial close consumergroup control-protocol implementation * Fix controller_test with pod fake dep * support startconsumergroups and fix dataplane labels * preliminary controller control-protocol communication to stop/start consumergroups * Clean notificationstore * cleanup * lint cleanup * PR feedback * PR Feedback
…e-extensions#724) * Extremely bad version (but does pause/resume with opcodes 3/4) * Added manager function to factory * Added consumergroup error channel wait * WIP * Added some manager tests * Consumer Group Manager unit tests 100% * Unit tests passing * remove debug code * Removed passthrough consumer group manager * revert accidents * undo accidents * Removed manager's unnecessary knowledge of customConsumerGroup * comments/spelling * Updates from PR review * Logging * imports * logger field redundancy * More logging * even more logging * Moved mocks to more appropriate places * fix mocks * Fixed forever-running test * Addressed data race in managedGroup functions * imports * Use atomics * imports * spelling * Added some debug messages * Changed from PR review * Use a copy of the router when calling MessageHandler()
* Extremely bad version (but does pause/resume with opcodes 3/4) * Added manager function to factory * Added consumergroup error channel wait * WIP * Added some manager tests * Consumer Group Manager unit tests 100% * Unit tests passing * remove debug code * Removed passthrough consumer group manager * revert accidents * undo accidents * Removed manager's unnecessary knowledge of customConsumerGroup * comments/spelling * Updates from PR review * Logging * imports * logger field redundancy * More logging * even more logging * Moved mocks to more appropriate places * fix mocks * Fixed forever-running test * Addressed data race in managedGroup functions * imports * Use atomics * imports * spelling * Added some debug messages * Changed from PR review * Use a copy of the router when calling MessageHandler() * First attempt * comments * comments * Debug messages * Don't change processAsyncGroupNotification for no reason * It wasn't for no reason, oops * imports * imports and comments * Tiny timeouts too finicky for unit tests * imports * Timeouts too short * Timing experiments * missed one * Can't win * Changes from PR review
bumping knative.dev/eventing 2fcbc0a...a574b7b: > a574b7b Fix flakiness & slowness of imc dispatcher test (# 5537) > b2f4322 Removing deprecated code from test/upgrade (# 5555) > 14ae489 Update community files (# 5553) > 6addfef Update community files (# 5546) > 21651d4 Knativehistory was removed (# 5550) > e614155 [PingSource] Degrade warning to normal event (# 5549) > c84fe22 Extend rotation calendar (# 5538) Signed-off-by: Knative Automation <automation@knative.team>
* optimize InitOffsets * attempt to fix race Co-authored-by: Lionel Villard <villard@us.ibm.com>
* add missing condition types Co-authored-by: Steven DOng <steven.dong@ibm.com>
…ons#793) Signed-off-by: Ahmed Abdalla <aabdelre@redhat.com> Co-authored-by: Ahmed Abdalla <aabdelre@redhat.com>
…ted dispatcher OwenrRef (knative-extensions#802) * Set KafkaChannel consolidated dispatcher OwenrRef Signed-off-by: Ahmed Abdalla <aabdelre@redhat.com> * Remove the rest of superficial tests Signed-off-by: Ahmed Abdalla <aabdelre@redhat.com> * Use a k8s go-client to fetch controller deployment Signed-off-by: Ahmed Abdalla <aabdelre@redhat.com> * Refactor dispatcher reconciliation logic for easier reasoning Signed-off-by: Ahmed Abdalla <aabdelre@redhat.com> * Remove apps.kubernetes.io labels * fix bad comment * Use apimachinary wait package instead of manual backoff * fix typo Co-authored-by: Ahmed Abdalla <aabdelre@redhat.com>
Co-authored-by: Steven DOng <steven.dong@ibm.com>
bumping knative.dev/networking 53f45d6...4ee76ca: > 4ee76ca upgrade to latest dependencies (# 500) bumping knative.dev/pkg dd0db4b...953af01: > 953af01 [release-0.24] allow unknown metadata fields (# 2255) > 03e7ca5 Drop redundant pointers and decoders (# 2260) bumping knative.dev/eventing a574b7b...a8533da: > a8533da 🔥 Fix broken job name for 0.24 release (# 5665) > b4cabbf [release-0.24] [PingSource] disable @every (# 5590) Signed-off-by: Knative Automation <automation@knative.team>
…tensions#885) * Move initOffsets functionality to a common package (knative-extensions#884) Co-authored-by: Ali Ok <aliok@redhat.com> * Move initOffsets functionality to a common package (knative-extensions#884) Co-authored-by: Ali Ok <aliok@redhat.com> Co-authored-by: Knative Prow Robot <knative-prow-robot@google.com>
…ons#869) Co-authored-by: Ali Ok <aliok@redhat.com>
…ions#886) * Init offsets in Kafka channel controller - first iteration * Do not check prober for subscription readiness * Better code style * Get rid of probing for checking subscription readiness * Get rid of unused deps * Ooops, fixed the tests * Pass context along the functions, which will be necessary later * Fix unit test * Move partition retrieval into a separate function, which is going to be used later * Check if offsets are initialized in the consumerFactory * IDE "extracts" method LOL * Make unit tests working * Move MockClusterAdmin into the common package and reuse it * Copy paste tests for CheckIfAllOffsetsInitialized * Unify tests for CheckIfAllOffsetsInitialized and InitOffsets * Separate tests for CheckIfAllOffsetsInitialized and InitOffsets * Do not block main reconciliation thread for offset checking * Remove last crumbs of probing * Change log level for offset init message * Move some consts to right place * Rename checkOffsets.. func to WaitForOffsets... * Rename consumerOffsetInitializer to ConsumerGroupOffsetsChecker * Do not handle deleted topics or partitions when checking the offsets * Copy the partitions array when retrieving partitions * Address comments * Separate client and clusteradminclient creation * Stash kafka client * Revert "Stash kafka client" This reverts commit 4d57547 * Do not do any offset initialization when subscription is already marked ready
Signed-off-by: Pierangelo Di Pilato <pdipilat@redhat.com>
|
@pierDipi: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: pierDipi The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
This mess was created by the GH CLI that I'm using to open PR from the command line, so far with only 2 remotes it works but with 3 remotes it chooses a random one 😄 |
|
:-) yeah - main is only here to stash things |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #
Proposed Changes
Release Note
Docs