Conversation
The previous wording is not 100% clear, a little example clarifies it.
…s with 0; patched by Milosz Tanski; reviewed by Jun Rao
Fix issue with SSLContext initialized globally at JVM level
Bootstrap Kafka system tests
|
This pull request doesn't merge cleanly anymore, can you please close it? Sorry for not replying to this earlier, but pull requests are not currently monitored as the Kafka project uses JIRA and Review Board for contributions. There is a plan to change this and we would like to close stale PRs before we start. Unfortunately we can't do it ourselves (a JIRA needs to be filed with Apache Infra) so your help is appreciated. If this change is still relevant, please see http://kafka.apache.org/contributing.html. Alternatively wait until the new approach based on GitHub pull requests is in place (hopefully soon). |
|
I believe you can close it, this is a really old PR. |
|
@fsaintjacques, for Apache projects, no-one but Apache Infra has write permission to the repository (including pull requests and issues). So, either you (the author) must close it, or we must file a ticket asking Infra to do it. |
Initial patch for KIP-25 Note that to install ducktape, do *not* use pip to install ducktape. Instead: ``` $ git clone gitgithub.com:confluentinc/ducktape.git $ cd ducktape $ python setup.py install ``` Author: Geoff Anderson <geoff@confluent.io> Author: Geoff <granders@gmail.com> Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ewen, Gwen, Jun, Guozhang Closes #70 from granders/KAFKA-2276 and squashes the following commits: a62fb6c [Geoff Anderson] fixed checkstyle errors a70f0f8 [Geoff Anderson] Merged in upstream trunk. 8b62019 [Geoff Anderson] Merged in upstream trunk. 47b7b64 [Geoff Anderson] Created separate tools jar so that the clients package does not pull in dependencies on the Jackson JSON tools or argparse4j. a9e6a14 [Geoff Anderson] Merged in upstream changes d18db7b [Geoff Anderson] fixed :rat errors (needed to add licenses) 321fdf8 [Geoff Anderson] Ignore tests/ and vagrant/ directories when running rat build task 795fc75 [Geoff Anderson] Merged in changes from upstream trunk. 1d93f06 [Geoff Anderson] Updated provisioning to use java 7 in light of KAFKA-2316 2ea4e29 [Geoff Anderson] Tweaked README, changed default log collection behavior on VerifiableProducer 0eb6fdc [Geoff Anderson] Merged in system-tests 69dd7be [Geoff Anderson] Merged in trunk 4034dd6 [Geoff Anderson] Merged in upstream trunk ede6450 [Geoff] Merge pull request #4 from confluentinc/move_muckrake 7751545 [Geoff Anderson] Corrected license headers e6d532f [Geoff Anderson] java 7 -> java 6 8c61e2d [Geoff Anderson] Reverted jdk back to 6 f14c507 [Geoff Anderson] Removed mode = "test" from Vagrantfile and Vagrantfile.local examples. Updated testing README to clarify aws setup. 98b7253 [Geoff Anderson] Updated consumer tests to pre-populate kafka logs e6a41f1 [Geoff Anderson] removed stray println b15b24f [Geoff Anderson] leftover KafkaBenchmark in super call 0f75187 [Geoff Anderson] Rmoved stray allow_fail. kafka_benchmark_test -> benchmark_test f469f84 [Geoff Anderson] Tweaked readme, added example Vagrantfile.local 3d73857 [Geoff Anderson] Merged downstream changes 42dcdb1 [Geoff Anderson] Tweaked behavior of stop_node, clean_node to generally fail fast 7f7c3e0 [Geoff Anderson] Updated setup.py for kafkatest c60125c [Geoff Anderson] TestEndToEndLatency -> EndToEndLatency 4f476fe [Geoff Anderson] Moved aws scripts to vagrant directory 5af88fc [Geoff Anderson] Updated README to include aws quickstart e5edf03 [Geoff Anderson] Updated example aws Vagrantfile.local 96533c3 [Geoff] Update aws-access-keys-commands 25a413d [Geoff] Update aws-example-Vagrantfile.local 884b20e [Geoff Anderson] Moved a bunch of files to kafkatest directory fc7c81c [Geoff Anderson] added setup.py 632be12 [Geoff] Merge pull request #3 from confluentinc/verbose-client 51a94fd [Geoff Anderson] Use argparse4j instead of joptsimple. ThroughputThrottler now has more intuitive behavior when targetThroughput is 0. a80a428 [Geoff Anderson] Added shell program for VerifiableProducer. d586fb0 [Geoff Anderson] Updated comments to reflect that throttler is not message-specific 6842ed1 [Geoff Anderson] left out a file from last commit 1228eef [Geoff Anderson] Renamed throttler 9100417 [Geoff Anderson] Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable. 0a5de8e [Geoff Anderson] Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements. 475423b [Geoff Anderson] Convert class to string before adding to json object. bc009f2 [Geoff Anderson] Got rid of VerboseProducer in core (moved to clients) c0526fe [Geoff Anderson] Updates per review comments. 8b4b1f2 [Geoff Anderson] Minor updates to VerboseProducer 2777712 [Geoff Anderson] Added some metadata to producer output. da94b8c [Geoff Anderson] Added number of messages option. 07cd1c6 [Geoff Anderson] Added simple producer which prints status of produced messages to stdout. a278988 [Geoff Anderson] fixed typos f1914c3 [Liquan Pei] Merge pull request #2 from confluentinc/system_tests 81e4156 [Liquan Pei] Bootstrap Kafka system tests
…reams-tech-preview Backport recent changes from trunk/streams: batch apache#2
[LOGBROKER-726] Fix debianization
Add ability to prevent broker from taking leadership for any partition
…er receiving offset commit exception TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit apache#2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
…AndTimestamp and ListOffsetResult & Tests
|
This issue can be caused by a non-existing path but also a misunderstanding from the config file. A short example will help the user. |
for testcontainers based integration tests in SMM, Kafka Connectors (kafka_connect_ext/kafka_connect_parent) [CDPD-24861] Adding Docker files for building Docker images. The script will be triggered as part of the build process to create Docker images. Change-Id: I401c0b3a49bb885d7aaad7691a8587873bf7a255 [CDPD-25577] Adding environment variables to docker image for configuring Kafka properties Change-Id: I37eddd212d78f929e1cac94cc6e5f7ca265b46bb CDPD-24861: Making a separate image for Kafka and Connect Base and also adding flexibility to the property configuration. Change-Id: I7725744d5fea0237868604fd36f9d450f6bf7ada CDPD-26355 tagging images with stack latest Change-Id: I6ddb44276b3641a3e6c9e729f5dc1ba0e64cadb7 [docker] CDPD-39536 Update Kafka Connect docker images configuration to use listeners property Change-Id: Ifad3e0369ec7e93b094d27945dc05b2810d810f5 CDPD-40406: Use JDK 11 for docker images Change-Id: Ib899cb7105b8f8ab5ae4094f297d237bb802b006 [docker] CDPD-39536 Update Kafka Connect docker images configuration to use listeners property apache#2 Change-Id: I94202c3f7607219fccb4562b68b1671f6fdee54b [docker] CDPD-41325 Simplifying local build of Kafka, Kafka Connect base docker images Change-Id: Id51eb15fd198ffa929e3eb3d98d66ce087959c9f [docker] CDPD-56944: Build Cruise Control Metrics Reporter into Kafka docker image and configure if available Change-Id: I065eaaee0db8ee6301b63d88c7f54a3fe814c811 [docker, ci] CDPD-60283: Run storage initialization when KRaft mode of Kafka is used and use KRaft mode in CI where possible acls are still not supported in KRaft mode, those are still using Zookeeper also some simplifications for M1 local docker image build Change-Id: I4154024a22c920f77ce5277d500d5d3553b540c2 CDPD-60488: Test flakiness corrected by introducing delay Change-Id: Id27fcb371853361ce8f683e16e093b35c123e3da CDPD-68964: Remove tests from under ci module Change-Id: I22173bd078f47f18f166ede936817db2bd8ddc91 [CLOUDERA-BUILD] CI This commit contains the following items: [CLOUDERA-BUILD] [CDPDFX-221] Add kafka CI Change-Id: I0686081492480db93fbca3d3cf194466073deeb4 [CLOUDERA-BUILD] CI Fixing gate job Change-Id: Ifa861e0621f58e6db200c0521223462bfc9b90a7 Change ci tests to java Change-Id: Ia133df85a9300a9f96c1f2d6c27bd9f8c64ca539 CI - Rename helm repo name since cloudera/kafka became an existing directory Change-Id: Iff3bfa926793f0c1664fa9c4ff69976338642aea CDPDFX-756 collect ci logs Change-Id: Iee58b0a6f082a64c9c208a6b116c14731339993f CDPD-35263 Use AclAuthorizer Change-Id: Ifed7f7b531cac988a171fea9690789c8e1b86bbf CDPD-42095 Upgrade junit in CI tests Change-Id: Ia19c231860f7f83fd51ab5e9c7340b486ede0095 CDPD-45541 Remove kafka zuul ci ignore_commit_convention_check Change-Id: If5e6a2f09cca5e61a180a3ed6f9209ecadd49903 CDPD-45541 Include CI tests into test scope Change-Id: I3520813d1bc5f08fc0d39c5bfa2feff91ec6ffa1 CDPD-45541: Canary breaks with older hwqe-kafka-client versions e.g. org.hw.qe:hwqe-kafka-client:0.4.128 is not present in public-non-IN-QA repo - which is used by canary Change-Id: If4cd05cb3e73605168b8d348344863a11f8806e2 [ci] CDPD-27134 Groovy postbuild script to handle gerrit voting Change-Id: If8d26250c504b4c5ae99f075810e1ba1a51897b2 CDPD-35705: Increase timeout for post-unit-test execution Change-Id: I1c136ed3c3eeb3ceec8c430e201abafe7e76e92d fixup! [ci] CDPD-27134 Groovy postbuild script to handle gerrit voting CDPD-48246: Add Unit test+1 label for CDF/smm_app_private Change-Id: I0ae6a32d119931c47d24e3a893d7e11dface06f3 CDPD-55718: Adding file to execute java11 unit tests Change-Id: If15a35d82277b67f942c3b1a4712cacee4757d36 Netcat is used by testcontainers to check the ports in the container Change-Id: Idc98aa472fe5706fb751943e6502d0d8c6f15abe
…ker provisioning (#21394) ## Summary Fixes bugs where `--jdk-version` and `--jdk-arch` parameters were ignored during system test worker provisioning, and refactors `vagrant/base.sh` to support flexible JDK versions without code changes. --- ## Problem The Vagrant provisioning script (`vagrant/base.sh`) had two bugs that caused JDK version parameters to be ignored: | Bug | Problem | |-----|---------| | **#1: `--jdk-version` ignored** | `JDK_FULL` was hardcoded to `17-linux-x64`, so passing `--jdk-version 25` still downloaded JDK 17 | | **#2: `--jdk-arch` ignored** | Architecture parameter was passed but never used in the S3 download URL | --- ## Solution - Validate `JDK_MAJOR` and `JDK_ARCH` input parameters with regex - Dynamically construct `JDK_FULL` from `JDK_MAJOR` and `JDK_ARCH` - Update S3 path to use `/jdk/` subdirectory - Add logging for debugging --- ## Changes ### `vagrant/base.sh` | Change | Description | |--------|-------------| | **Input validation** | Added regex validation for `JDK_MAJOR` and `JDK_ARCH` with sensible defaults | | **Dynamic construction** | `JDK_FULL` is now constructed from `JDK_MAJOR` and `JDK_ARCH` if not explicitly provided | | **Updated S3 path** | Changed URL from `/kafka-packages/jdk-{version}.tar.gz` to `/kafka-packages/jdk/jdk-{version}.tar.gz` | | **Logging** | Added debug output for JDK configuration | | **Backward compatibility** | Vagrantfile can still pass `JDK_FULL` directly; the script validates and uses it if valid | --- ## S3 Path Change ### Old Path ``` s3://kafka-packages/jdk-{version}.tar.gz ``` ### New Path ``` s3://kafka-packages/jdk/jdk-{version}.tar.gz ``` ### Available JDKs in `s3://kafka-packages/jdk/` | File | Version | Architecture | |------|---------|--------------| | `jdk-7u80-linux-x64.tar.gz` | 7u80 | x64 | | `jdk-8u144-linux-x64.tar.gz` | 8u144 | x64 | | `jdk-8u161-linux-x64.tar.gz` | 8u161 | x64 | | `jdk-8u171-linux-x64.tar.gz` | 8u171 | x64 | | `jdk-8u191-linux-x64.tar.gz` | 8u191 | x64 | | `jdk-8u202-linux-x64.tar.gz` | 8u202 | x64 | | `jdk-11.0.2-linux-x64.tar.gz` | 11.0.2 | x64 | | `jdk-17-linux-x64.tar.gz` | 17 | x64 | | `jdk-18.0.2-linux-x64.tar.gz` | 18.0.2 | x64 | | `jdk-21.0.1-linux-x64.tar.gz` | 21.0.1 | x64 | | `jdk-21.0.1-linux-aarch64.tar.gz` | 21.0.1 | aarch64 | | `jdk-25-linux-x64.tar.gz` | 25 | x64 | | `jdk-25-linux-aarch64.tar.gz` | 25 | aarch64 | | `jdk-25.0.1-linux-x64.tar.gz` | 25.0.1 | x64 | | `jdk-25.0.1-linux-aarch64.tar.gz` | 25.0.1 | aarch64 | | `jdk-25.0.2-linux-x64.tar.gz` | 25.0.2 | x64 | | `jdk-25.0.2-linux-aarch64.tar.gz` | 25.0.2 | aarch64 | --- ## Future JDK Releases > **IMPORTANT: No code changes required for future Java major/minor releases!** The validation regex supports all version formats: - **Major versions**: `17`, `25`, `26` - **Minor versions**: `25.0.1`, `25.0.2`, `26.0.1` - **Legacy format**: `8u144`, `8u202` ### Adding New JDK Versions To add support for a new JDK version (e.g., JDK 26, 25.0.3): 1. Download the JDK tarball from Oracle/Adoptium 2. Rename to follow naming convention: `jdk-{VERSION}-linux-{ARCH}.tar.gz` 3. Upload to S3: `aws s3 cp jdk-{VERSION}-linux-{ARCH}.tar.gz s3://kafka-packages/jdk/` 4. Use in tests: `--jdk-version {VERSION} --jdk-arch {ARCH}` No modifications to `base.sh` or any other scripts are needed. --- ## Benefits | Before | After | |--------|-------| | `--jdk-version` ignored | ✅ Correctly uses specified version | | `--jdk-arch` ignored | ✅ Correctly uses specified architecture | | Only major version support | ✅ Full version support (e.g., `25.0.2`) | | Code change needed for new JDK | ✅ Just upload to S3 and pass version | --- ## Testing Tested with different JDK versions to confirm the fix works correctly: | Test | JDK_MAJOR | Expected | Actual | Result | Test Report | |------|-----------|----------|--------|--------|-------------| | JDK 17 | `17` | javac 17.0.4 | javac 17.0.4 | ✅ | | JDK 25 | `25` | javac 25.0.2 | javac 25.0.2 | ✅ | --- ## Backward Compatibility - **Vagrantfile**: Continues to work as before - **Existing workflows**: Default behavior unchanged (JDK 17 on x64 architecture) - **No breaking changes**: All existing configurations continue to work --- Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
…ker provisioning (#21394) ## Summary Fixes bugs where `--jdk-version` and `--jdk-arch` parameters were ignored during system test worker provisioning, and refactors `vagrant/base.sh` to support flexible JDK versions without code changes. --- ## Problem The Vagrant provisioning script (`vagrant/base.sh`) had two bugs that caused JDK version parameters to be ignored: | Bug | Problem | |-----|---------| | **#1: `--jdk-version` ignored** | `JDK_FULL` was hardcoded to `17-linux-x64`, so passing `--jdk-version 25` still downloaded JDK 17 | | **#2: `--jdk-arch` ignored** | Architecture parameter was passed but never used in the S3 download URL | --- ## Solution - Validate `JDK_MAJOR` and `JDK_ARCH` input parameters with regex - Dynamically construct `JDK_FULL` from `JDK_MAJOR` and `JDK_ARCH` - Update S3 path to use `/jdk/` subdirectory - Add logging for debugging --- ## Changes ### `vagrant/base.sh` | Change | Description | |--------|-------------| | **Input validation** | Added regex validation for `JDK_MAJOR` and `JDK_ARCH` with sensible defaults | | **Dynamic construction** | `JDK_FULL` is now constructed from `JDK_MAJOR` and `JDK_ARCH` if not explicitly provided | | **Updated S3 path** | Changed URL from `/kafka-packages/jdk-{version}.tar.gz` to `/kafka-packages/jdk/jdk-{version}.tar.gz` | | **Logging** | Added debug output for JDK configuration | | **Backward compatibility** | Vagrantfile can still pass `JDK_FULL` directly; the script validates and uses it if valid | --- ## S3 Path Change ### Old Path ``` s3://kafka-packages/jdk-{version}.tar.gz ``` ### New Path ``` s3://kafka-packages/jdk/jdk-{version}.tar.gz ``` ### Available JDKs in `s3://kafka-packages/jdk/` | File | Version | Architecture | |------|---------|--------------| | `jdk-7u80-linux-x64.tar.gz` | 7u80 | x64 | | `jdk-8u144-linux-x64.tar.gz` | 8u144 | x64 | | `jdk-8u161-linux-x64.tar.gz` | 8u161 | x64 | | `jdk-8u171-linux-x64.tar.gz` | 8u171 | x64 | | `jdk-8u191-linux-x64.tar.gz` | 8u191 | x64 | | `jdk-8u202-linux-x64.tar.gz` | 8u202 | x64 | | `jdk-11.0.2-linux-x64.tar.gz` | 11.0.2 | x64 | | `jdk-17-linux-x64.tar.gz` | 17 | x64 | | `jdk-18.0.2-linux-x64.tar.gz` | 18.0.2 | x64 | | `jdk-21.0.1-linux-x64.tar.gz` | 21.0.1 | x64 | | `jdk-21.0.1-linux-aarch64.tar.gz` | 21.0.1 | aarch64 | | `jdk-25-linux-x64.tar.gz` | 25 | x64 | | `jdk-25-linux-aarch64.tar.gz` | 25 | aarch64 | | `jdk-25.0.1-linux-x64.tar.gz` | 25.0.1 | x64 | | `jdk-25.0.1-linux-aarch64.tar.gz` | 25.0.1 | aarch64 | | `jdk-25.0.2-linux-x64.tar.gz` | 25.0.2 | x64 | | `jdk-25.0.2-linux-aarch64.tar.gz` | 25.0.2 | aarch64 | --- ## Future JDK Releases > **IMPORTANT: No code changes required for future Java major/minor releases!** The validation regex supports all version formats: - **Major versions**: `17`, `25`, `26` - **Minor versions**: `25.0.1`, `25.0.2`, `26.0.1` - **Legacy format**: `8u144`, `8u202` ### Adding New JDK Versions To add support for a new JDK version (e.g., JDK 26, 25.0.3): 1. Download the JDK tarball from Oracle/Adoptium 2. Rename to follow naming convention: `jdk-{VERSION}-linux-{ARCH}.tar.gz` 3. Upload to S3: `aws s3 cp jdk-{VERSION}-linux-{ARCH}.tar.gz s3://kafka-packages/jdk/` 4. Use in tests: `--jdk-version {VERSION} --jdk-arch {ARCH}` No modifications to `base.sh` or any other scripts are needed. --- ## Benefits | Before | After | |--------|-------| | `--jdk-version` ignored | ✅ Correctly uses specified version | | `--jdk-arch` ignored | ✅ Correctly uses specified architecture | | Only major version support | ✅ Full version support (e.g., `25.0.2`) | | Code change needed for new JDK | ✅ Just upload to S3 and pass version | --- ## Testing Tested with different JDK versions to confirm the fix works correctly: | Test | JDK_MAJOR | Expected | Actual | Result | Test Report | |------|-----------|----------|--------|--------|-------------| | JDK 17 | `17` | javac 17.0.4 | javac 17.0.4 | ✅ | | JDK 25 | `25` | javac 25.0.2 | javac 25.0.2 | ✅ | --- ## Backward Compatibility - **Vagrantfile**: Continues to work as before - **Existing workflows**: Default behavior unchanged (JDK 17 on x64 architecture) - **No breaking changes**: All existing configurations continue to work --- Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
This issue can be caused by a non-existing path but also a misunderstanding from the config file. A short example will help the user.