Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade librdkafka to v2.6.0 #120

Merged
merged 5 commits into from
Oct 14, 2024
Merged

Upgrade librdkafka to v2.6.0 #120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b7a5d40
Upgrade librdkafka to v2.6.0
emasab Oct 14, 2024
f53b17d
Fix install-from-source
emasab Oct 14, 2024
a4cb696
Dynamic linking for tests
emasab Oct 14, 2024
be9cc9e
Speed up librdkafka build
emasab Oct 14, 2024
ff7033f
Fix performance test installation
emasab Oct 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 24 additions & 24 deletions .semaphore/semaphore.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ global_job_config:
- git submodule update --init --recursive
- cd deps/librdkafka
- git fetch origin
- git checkout v2.5.3
- git checkout v2.6.0
- cd ../../
- cache clear

Expand Down Expand Up @@ -70,7 +70,8 @@ blocks:
jobs:
- name: "Build from source and test"
commands:
- npm install # this will actually not build anything if we have a release, but rather, fetch things using node-pre-gyp - so change this later.
- export CKJS_LINKING=dynamic
- npm run install-from-source
- make test

- name: 'macOS arm64/m1: Build and test'
Expand All @@ -82,7 +83,8 @@ blocks:
jobs:
- name: 'Build from source and test'
commands:
- npm install # this will actually not build anything if we have a release, but rather, fetch things using node-pre-gyp - so change this later.
- export CKJS_LINKING=dynamic
- npm run install-from-source
- make test

- name: "Linux amd64: Build, test, lint"
Expand All @@ -93,7 +95,8 @@ blocks:
type: s1-prod-ubuntu20-04-amd64-2
prologue:
commands:
- npm install # this will actually not build anything if we have a release, but rather, fetch things using node-pre-gyp - so change this later.
- export CKJS_LINKING=dynamic
- npm run install-from-source
jobs:
- name: "Test"
commands:
Expand Down Expand Up @@ -130,10 +133,7 @@ blocks:
- sudo apt-get install -y librdkafka-dev bc
- export CKJS_LINKING=dynamic
- export BUILD_LIBRDKAFKA=0
- npm install
- npx node-pre-gyp --build-from-source clean
- npx node-pre-gyp --build-from-source configure
- npx node-pre-gyp --build-from-source build
- npm run install-from-source
jobs:
- name: "Performance Test"
commands:
Expand Down Expand Up @@ -168,7 +168,7 @@ blocks:
- sem-version node 18.19.0
- export NODE_ABI=108
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -177,7 +177,7 @@ blocks:
- sem-version node 20.10.0
- export NODE_ABI=115
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -186,7 +186,7 @@ blocks:
- sem-version node 21.4.0
- export NODE_ABI=120
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -195,7 +195,7 @@ blocks:
- sem-version node 22.2.0
- export NODE_ABI=127
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand Down Expand Up @@ -224,7 +224,7 @@ blocks:
- sem-version node 18.19.0
- export NODE_ABI=108
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -233,7 +233,7 @@ blocks:
- sem-version node 20.10.0
- export NODE_ABI=115
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -242,7 +242,7 @@ blocks:
- sem-version node 21.4.0
- export NODE_ABI=120
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -251,7 +251,7 @@ blocks:
- sem-version node 22.2.0
- export NODE_ABI=127
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand Down Expand Up @@ -373,7 +373,7 @@ blocks:
- sem-version node 18.19.0
- export NODE_ABI=108
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -382,7 +382,7 @@ blocks:
- sem-version node 20.10.0
- export NODE_ABI=115
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -391,7 +391,7 @@ blocks:
- sem-version node 21.4.0
- export NODE_ABI=120
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand All @@ -400,7 +400,7 @@ blocks:
- sem-version node 22.2.0
- export NODE_ABI=127
- export ARTIFACT_KEY="confluent-kafka-javascript-${SEMAPHORE_GIT_TAG_NAME}-node-v${NODE_ABI}-${PLATFORM}-${LIBC}-${ARCHITECTURE}.tar.gz"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}
- artifact push workflow "build/stage/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}" --destination "releases/${SEMAPHORE_GIT_TAG_NAME}/${ARTIFACT_KEY}"
Expand Down Expand Up @@ -450,7 +450,7 @@ blocks:
- $env:NODE_ABI = 108
- $env:ARTIFACT_KEY = "confluent-kafka-javascript-${env:SEMAPHORE_GIT_TAG_NAME}-node-v${env:NODE_ABI}-${env:PLATFORM}-${env:LIBC}-${env:ARCHITECTURE}.tar.gz"
- echo "$env:ARTIFACT_KEY"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls "build/stage/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}"
- artifact push workflow "build/stage/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}" --destination "releases/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}"
Expand All @@ -463,7 +463,7 @@ blocks:
- $env:NODE_ABI = 115
- $env:ARTIFACT_KEY = "confluent-kafka-javascript-${env:SEMAPHORE_GIT_TAG_NAME}-node-v${env:NODE_ABI}-${env:PLATFORM}-${env:LIBC}-${env:ARCHITECTURE}.tar.gz"
- echo "$env:ARTIFACT_KEY"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls "build/stage/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}"
- artifact push workflow "build/stage/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}" --destination "releases/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}"
Expand All @@ -476,7 +476,7 @@ blocks:
- $env:NODE_ABI = 120
- $env:ARTIFACT_KEY = "confluent-kafka-javascript-${env:SEMAPHORE_GIT_TAG_NAME}-node-v${env:NODE_ABI}-${env:PLATFORM}-${env:LIBC}-${env:ARCHITECTURE}.tar.gz"
- echo "$env:ARTIFACT_KEY"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls "build/stage/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}"
- artifact push workflow "build/stage/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}" --destination "releases/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}"
Expand All @@ -489,7 +489,7 @@ blocks:
- $env:NODE_ABI = 127
- $env:ARTIFACT_KEY = "confluent-kafka-javascript-${env:SEMAPHORE_GIT_TAG_NAME}-node-v${env:NODE_ABI}-${env:PLATFORM}-${env:LIBC}-${env:ARCHITECTURE}.tar.gz"
- echo "$env:ARTIFACT_KEY"
- npm install # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npm ci # node-pre-gyp will fallback to build here, because new tag implies no release yet.
- npx node-pre-gyp package
- ls "build/stage/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}"
- artifact push workflow "build/stage/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}" --destination "releases/${env:SEMAPHORE_GIT_TAG_NAME}/${env:ARTIFACT_KEY}"
Expand Down
2 changes: 1 addition & 1 deletion deps/librdkafka
Open in desktop
Submodule librdkafka updated 39 files
+28 −0 CHANGELOG.md
+2 −1 INTRODUCTION.md
+2 −1 examples/.gitignore
+3 −0 examples/CMakeLists.txt
+5 −0 examples/Makefile
+317 −0 examples/elect_leaders.c
+46 −11 examples/list_consumer_groups.c
+1 −1 src-cpp/rdkafkacpp.h
+62 −8 src/rdkafka.c
+214 −7 src/rdkafka.h
+420 −14 src/rdkafka_admin.c
+30 −0 src/rdkafka_admin.h
+75 −0 src/rdkafka_aux.c
+43 −0 src/rdkafka_aux.h
+55 −0 src/rdkafka_broker.c
+15 −4 src/rdkafka_broker.h
+15 −1 src/rdkafka_cgrp.c
+3 −0 src/rdkafka_cgrp.h
+10 −0 src/rdkafka_event.c
+1 −0 src/rdkafka_event.h
+21 −1 src/rdkafka_fetcher.c
+64 −5 src/rdkafka_int.h
+10 −1 src/rdkafka_mock.c
+9 −0 src/rdkafka_mock.h
+5 −3 src/rdkafka_mock_cgrp.c
+2 −0 src/rdkafka_mock_int.h
+8 −1 src/rdkafka_op.c
+3 −0 src/rdkafka_op.h
+6 −7 src/rdkafka_queue.c
+102 −5 src/rdkafka_request.c
+12 −0 src/rdkafka_request.h
+529 −35 src/rdkafka_telemetry_decode.c
+223 −67 src/rdkafka_telemetry_encode.c
+122 −35 src/rdkafka_telemetry_encode.h
+19 −3 src/rdtime.h
+154 −5 tests/0080-admin_ut.c
+4 −4 tests/0113-cooperative_rebalance.cpp
+278 −180 tests/0150-telemetry_mock.c
+1 −1 vcpkg.json
2 changes: 1 addition & 1 deletion deps/librdkafka.gyp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
"deps/librdkafka/src/librdkafka.so",
],
"action": [
"make", "-C", "librdkafka", "libs", "install"
"make", "-j", "-C", "librdkafka", "libs", "install"
],
"conditions": [
[
Expand Down
2 changes: 1 addition & 1 deletion lib/error.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ LibrdKafkaError.wrap = errorWrap;
* @enum {number}
* @constant
*/
// ====== Generated from librdkafka master file src-cpp/rdkafkacpp.h ======
// ====== Generated from librdkafka 2.6.0 file src-cpp/rdkafkacpp.h ======
LibrdKafkaError.codes = {

/* Internal errors to rdkafka: */
Expand Down
5 changes: 3 additions & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,16 @@
"name": "@confluentinc/kafka-javascript",
"version": "v0.2.1",
"description": "Node.js bindings for librdkafka",
"librdkafka": "2.5.3",
"librdkafka_win": "2.5.3",
"librdkafka": "2.6.0",
"librdkafka_win": "2.6.0",
"main": "lib/index.js",
"types": "types/index.d.ts",
"scripts": {
"configure": "node-gyp configure",
"build": "node-gyp build",
"test": "make test",
"install": "node-pre-gyp install --fallback-to-build",
"install-from-source": "npm ci && node-pre-gyp install --build-from-source=@confluentinc/kafka-javascript --fallback-to-build",
"prepack": "node ./ci/prepublish.js",
"test:types": "tsc -p ."
},
Expand Down
Loading