Admin examples for available APIs #84
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
milindl
approved these changes
Sep 15, 2024
emasab
merged commit d014984
into
dev_early_access_development_branch
1 of 2 checks passed
PratRanj07
added a commit
that referenced
this pull request
Nov 4, 2024
* 1st commit * 2nd commit * 3rd commit * Added tests and examples * Formatting * Formatting * little change * some small changes * changes requested * requested changes * requested changes * name change * indentation * indentation * Add data-governance to code owners for schema registry clients (#52) * Add data-governance to code owners for schema registry clients * Fix ownership * Fix deprecation warning * Separate eachMessage and eachBatch internal consume loop * Update performance example with more cases * Add per-partition cache with global expiry * Add per-partition cache expiry logic * Allow cache to disburse multiple messages at once * Add per-partition concurrency * Add partition level concurrency to faux-eachBatch * Create persistent workers for per-partition concurrency, prevents excessive Promise spawning * Fix tests for Per Partition Concurrency * Add message set capability to message cache * Add naive batching (without resolution handling) * Add batch staleness, resolution, and offset management to eachBatch * Update tests for true eachBatch Also cleans up some excessive parametrization. Separates tests into files so they can be parallelized better. wq * Remove debug-only properties * Update MIGRATION.md for eachBatch * Bump version * Fix linting and Makefile issues (#2) * Add single-offset-storage method for performance In promisified mode, offsets are stored after every single message Calling into FFI is already not cheap and on top of that, the offsetsStore method does a lot of allocations because it needs to make a vector etc. etc. This method change yields a benefit of 8%. Also reduces dynamic casts by storing a reference to the rdkafka consumer. * Remove jshint and expand eslint coverage Many changes within, but only cosmetic: 1. remove jshint as it's rather unmaintained and we don't need 2 linters. 2. update eslint to latest possible version. 3. update eslint coverage to include old API and promisified tests. 4. update makefile accordingly 5. fix lint errors. * Update Makefule, update cpplint to latest version * Comply with cpplint * Add SchemaRegistryClient, RestService, and testing (#1) * Add SchemaRegistryClient, RestService, and testing * Add new Makefile for schema registry * Merging * Revert to throwing exceptions * Add mock client for testing * Remove testing artifacts * Fix flaky e2e tests (#54) * fixing empty message and key * indentation * 1st commit * changes * changes * alignment * Preset fix (#6) * Fix jest to unblock tests * Add git history * Do not modify RegExps which don't start with a ^ 1. Adds test. Also fixes some flakiness in the subscribe test. 2. Fixes typing issue for assignment() * Fix argument mutation in run, pause and resume * Dekregistry client (#67) * Add Mock Schema Registry Client (#9) (#66) * Add mock client for testing * Remove testing artifacts * Dekregistry client (#10) * Add mock client for testing * Remove testing artifacts * Add dekregistry client, mock dekregistry client --------- Co-authored-by: Robert Yokota <rayokota@gmail.com> --------- Co-authored-by: Robert Yokota <rayokota@gmail.com> * Add clientConfig, baseUrl retry, RestError, encodeURIComponent (#12) (#68) * Add clientConfig, baseUrl retry, RestError * refactor such that RestService takes in necessary dependencies * Update tsconfig.json (#69) * Fix broken tests (#70) Co-authored-by: Robert Yokota <rayokota@gmail.com> * Add commitCb method (#59) to avoid blocking while committing and return a Promise without having to call consume() * Fix eslint config (#71) * Add eslint rules (#72) * Add some ts eslint rules * Fix makefile * First cut at JavaScript serdes (#73) * First cut at serdes * Checkpoint - no errs * Minor cleanup * Add siv * Fix eslint errs * Minor cleanup * Minor cleanup * Fix configs * Fix execute calls * Clean up public, compat levels * Fix test * Incorporate review feedback * Add assign/unassign within rebalance callbacks Also remove onPartitionsAssigned and onPartitionsRevoked. * Add performance benchmarking script modes and README * Add performance benchmarking script modes and README * Fix createTopics return * Add topic creation to benchmarks * Remove needless batch size (msgs) 1 * Add performance example to semaphore * Clean up the perf runner script * Add confluent debian repo for performance benchmark * Remove store from promisified API * Add binding level debug logging and client name to logs * Add binding level debug logging and client name to logs * Add C++ changes for binding-level logging * Fix typo in script name * First cut at Data Contract rules (#77) * First cut at encryption rules (#74) * First cut at encryption rules * Add tests * Clean up package.json * Clean up package.json * Add kms clients * Minor fix * First cut at additional serde tests (#75) * First cut at additional serde tests * Add JSON test * Add protobuf ref test * Add format params * Checkpoint * Add coverage to gitignore * Remove coverage * Minor fix * Minor fix * Avro ref test * Add json ref test * Add nested tests * First cut at Data Contract rules * Remove CEL executors for now * Minor refactor to use RuleRegistry * Move DEK registry under encryption * Clean up package.json * Add CSFLE test with logical type * Minor fix * Add CEL executors * Revert "Add CEL executors" This reverts commit 850c3de. * Minor fixes * Minor fixes * Separate SR into a different workspace (#78) Co-authored-by: Milind L <miluthra@confluent.io> * Refactor to always use a barrier for pending operation (#26) Readers writer lock with reentrant calls Refactor to always use a barrier for pending operation to avoid problems or regressions with async pause or resume calls that will be introduced later. Uses linked lists for the cache. Removes heap-js code. Test name improvement to run single parametric tests Seek signature and add partitionsConsumedConcurrently to ConsumerRunConfig type definition Make final max poll interval double of configured value so even last message processed before cache cleanup can take that time to process. Fix to restart max poll interval timer on fetch. Marking batch stale after cache clear was requested and max poll interval is reached before it's cleared. Add assignmentLost function to the rebalance callback. Fix to nextN size, version with max.poll.interval.ms applied to each message or batch (only for messages after cache reset) Performance test, removing outliers Start performance timer from first message received after resuming * Schemaregistry rebase (#33) (#80) * rebase dev_early_access * Add OAuth Support to Rest Service (#25) * Add OAuth client support * Add optional chaining for token * Fix merge conflict * add simple-oauth2 dependency * Add Docker environment for integration tests (#34) (#81) * Add docker files * Add docker env for integ tests * Fix log level config in light of binding logs * Also add rule for no trailing spaces and enforce it * Fix log level config in light of binding logs * Remove consumerGroupId argument from sendOffsets and add tests (#82) * Performance measurement improvements CKJS performance improved through fetch.queue.backoff.ms * Admin examples for available APIs (#84) * Fix listGroups segfault when passing an undefined matchConsumerGroupStates (#85) * Add more unit tests; minor fixes for KMS clients (#86) * Minor fixes for KMS clients * Add JSON 2020-12 test * Bump version to 0.1.17-devel * Add complex encryption tests (#89) * Add index.ts (#91) * Enhance HighLevelProducer to take schema serializers (#92) * Add auth features (#47) (#94) * Add auth features * Update schemaregistry/rest-service.ts * Minor changes * Minor changes (missed a few) --------- Co-authored-by: Robert Yokota <rayokota@gmail.com> * Add more JSON Schema validation tests (#95) * Move ts-jest to dev dependencies (#96) * Add JSON integration tests (#46) (#97) * Add JSON integration tests * remove random * Unsubscribe before disconnecting to mitigate hangs on destroy (#98) Co-authored-by: Emanuele Sabellico <esabellico@confluent.io> * Pass creds to DEK Registry client (#99) * Bump version to 0.2.0 and drop -devel (#100) * Remove mandatory basic or bearer auth credentials (#57) (#101) * Add build script and readme (#104) Co-authored-by: claimundefine <juswang@confluent.io> * Add license (#105) * Add license * Clean up scripts * Add clearLatestCaches/clearCaches API, fix test to call clearLatestCaches (#102) * Add clearLatestCaches for use in tests * Minor cleanup * Minor cleanup * Add avro integration tests (#56) (#106) * Add tsdoc (#107) * Add tsdocs * More tsdoc fixes * Enhance docs (#108) * Update kafkajs README and sr example * Minor formatting * Update schemaregistry README (#109) * Add restService interfaces to exported types (#110) Co-authored-by: claimundefine <juswang@confluent.io> * Rename DekClient to avoid conflict with Client (#112) * Minor cleanup * Rename DekClient * Schemaregistry examples (#69) (#113) * Add restService interfaces to exported types * Bugfix for rest service and oauth * Add schemaregistry examples workspace with avro, json, and csfle examples (#70) (#114) * Add schemaregistry examples workspace with avro, json, and csfle examples * change file name * Change schemaregistry name * Add extra descriptor for constants * Add content type by default * Change to basic auth for most examples * bugfix integ tests for registering -value (#71) (#115) * bugfix integ tests for registering -value * Add autoRegister and useLatestVersion tests * Bump version to v0.2.1 (#116) * Update version to 0.2.1 for EA release (#72) (#117) * Add Kafka Oauth implementation (#74) (#119) * Upgrade librdkafka to v2.6.0 (#120) * Upgrade librdkafka to v2.6.0 * Add install-from-source * Dynamic linking for tests * Speed up librdkafka build * Bump version to 0.3.0-RC1 and: (#122) - change software version to be without v as other clients - use relative dev or example dependencies to avoid changing version in multiple places - updates to RELEASE section * v0.3.0 (#126) * npmignore more files and fix to the update versions script * Improved Releasing section * Minor optimization to reduce schema ID lookups (#123) * v0.3.0-RC2 (#127) * v0.3.0 final release (#128) * Changelog entry for #123 * Fix header conversion in eachBatch (#130) * 1st commit * 2nd commit * 3rd commit * changes requested * requested changes * required Changes * remove unnecessary changes * indentation and unnecessary changes * indentation * comment removed * comment added * changelog entry * Changed topic partition js to c conversion structure * refactoring * Requested changes * final changes --------- Co-authored-by: Emanuele Sabellico <esabellico@confluent.io> Co-authored-by: Milind L <miluthra@confluent.io> Co-authored-by: Milind L <milindl@users.noreply.github.com> Co-authored-by: Justin Wang <35274216+Claimundefine@users.noreply.github.com> Co-authored-by: claimundefine <juswang@confluent.io> Co-authored-by: Robert Yokota <rayokota@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.