MINOR: Fix JDK version and architecture parameters in system test worker provisioning#21466
Closed
tirthooo7 wants to merge 1 commit intoapache:4.2from
Closed
MINOR: Fix JDK version and architecture parameters in system test worker provisioning#21466tirthooo7 wants to merge 1 commit intoapache:4.2from
tirthooo7 wants to merge 1 commit intoapache:4.2from
Conversation
…ker provisioning (apache#21394) ## Summary Fixes bugs where `--jdk-version` and `--jdk-arch` parameters were ignored during system test worker provisioning, and refactors `vagrant/base.sh` to support flexible JDK versions without code changes. --- ## Problem The Vagrant provisioning script (`vagrant/base.sh`) had two bugs that caused JDK version parameters to be ignored: | Bug | Problem | |-----|---------| | **#1: `--jdk-version` ignored** | `JDK_FULL` was hardcoded to `17-linux-x64`, so passing `--jdk-version 25` still downloaded JDK 17 | | **#2: `--jdk-arch` ignored** | Architecture parameter was passed but never used in the S3 download URL | --- ## Solution - Validate `JDK_MAJOR` and `JDK_ARCH` input parameters with regex - Dynamically construct `JDK_FULL` from `JDK_MAJOR` and `JDK_ARCH` - Update S3 path to use `/jdk/` subdirectory - Add logging for debugging --- ## Changes ### `vagrant/base.sh` | Change | Description | |--------|-------------| | **Input validation** | Added regex validation for `JDK_MAJOR` and `JDK_ARCH` with sensible defaults | | **Dynamic construction** | `JDK_FULL` is now constructed from `JDK_MAJOR` and `JDK_ARCH` if not explicitly provided | | **Updated S3 path** | Changed URL from `/kafka-packages/jdk-{version}.tar.gz` to `/kafka-packages/jdk/jdk-{version}.tar.gz` | | **Logging** | Added debug output for JDK configuration | | **Backward compatibility** | Vagrantfile can still pass `JDK_FULL` directly; the script validates and uses it if valid | --- ## S3 Path Change ### Old Path ``` s3://kafka-packages/jdk-{version}.tar.gz ``` ### New Path ``` s3://kafka-packages/jdk/jdk-{version}.tar.gz ``` ### Available JDKs in `s3://kafka-packages/jdk/` | File | Version | Architecture | |------|---------|--------------| | `jdk-7u80-linux-x64.tar.gz` | 7u80 | x64 | | `jdk-8u144-linux-x64.tar.gz` | 8u144 | x64 | | `jdk-8u161-linux-x64.tar.gz` | 8u161 | x64 | | `jdk-8u171-linux-x64.tar.gz` | 8u171 | x64 | | `jdk-8u191-linux-x64.tar.gz` | 8u191 | x64 | | `jdk-8u202-linux-x64.tar.gz` | 8u202 | x64 | | `jdk-11.0.2-linux-x64.tar.gz` | 11.0.2 | x64 | | `jdk-17-linux-x64.tar.gz` | 17 | x64 | | `jdk-18.0.2-linux-x64.tar.gz` | 18.0.2 | x64 | | `jdk-21.0.1-linux-x64.tar.gz` | 21.0.1 | x64 | | `jdk-21.0.1-linux-aarch64.tar.gz` | 21.0.1 | aarch64 | | `jdk-25-linux-x64.tar.gz` | 25 | x64 | | `jdk-25-linux-aarch64.tar.gz` | 25 | aarch64 | | `jdk-25.0.1-linux-x64.tar.gz` | 25.0.1 | x64 | | `jdk-25.0.1-linux-aarch64.tar.gz` | 25.0.1 | aarch64 | | `jdk-25.0.2-linux-x64.tar.gz` | 25.0.2 | x64 | | `jdk-25.0.2-linux-aarch64.tar.gz` | 25.0.2 | aarch64 | --- ## Future JDK Releases > **IMPORTANT: No code changes required for future Java major/minor releases!** The validation regex supports all version formats: - **Major versions**: `17`, `25`, `26` - **Minor versions**: `25.0.1`, `25.0.2`, `26.0.1` - **Legacy format**: `8u144`, `8u202` ### Adding New JDK Versions To add support for a new JDK version (e.g., JDK 26, 25.0.3): 1. Download the JDK tarball from Oracle/Adoptium 2. Rename to follow naming convention: `jdk-{VERSION}-linux-{ARCH}.tar.gz` 3. Upload to S3: `aws s3 cp jdk-{VERSION}-linux-{ARCH}.tar.gz s3://kafka-packages/jdk/` 4. Use in tests: `--jdk-version {VERSION} --jdk-arch {ARCH}` No modifications to `base.sh` or any other scripts are needed. --- ## Benefits | Before | After | |--------|-------| | `--jdk-version` ignored | ✅ Correctly uses specified version | | `--jdk-arch` ignored | ✅ Correctly uses specified architecture | | Only major version support | ✅ Full version support (e.g., `25.0.2`) | | Code change needed for new JDK | ✅ Just upload to S3 and pass version | --- ## Testing Tested with different JDK versions to confirm the fix works correctly: | Test | JDK_MAJOR | Expected | Actual | Result | Test Report | |------|-----------|----------|--------|--------|-------------| | JDK 17 | `17` | javac 17.0.4 | javac 17.0.4 | ✅ | | JDK 25 | `25` | javac 25.0.2 | javac 25.0.2 | ✅ | --- ## Backward Compatibility - **Vagrantfile**: Continues to work as before - **Existing workflows**: Default behavior unchanged (JDK 17 on x64 architecture) - **No breaking changes**: All existing configurations continue to work --- Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Member
|
@tirthooo7 Thanks for the backport. It looks like I can cherry-pick it directly instead of opening a PR. I'll do that once 4.2 is officially released |
Member
|
cherry-pick c1a930a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry pick from trunk : #21394
Summary
Fixes bugs where
--jdk-versionand--jdk-archparameters were ignored during system test worker provisioning, and refactorsvagrant/base.shto support flexible JDK versions without code changes.Problem
The Vagrant provisioning script (
vagrant/base.sh) had two bugs that caused JDK version parameters to be ignored:--jdk-versionignoredJDK_FULLwas hardcoded to17-linux-x64, so passing--jdk-version 25still downloaded JDK 17Solution
JDK_MAJORandJDK_ARCHinput parameters with regexJDK_FULLfromJDK_MAJORandJDK_ARCH/jdk/subdirectoryChanges
vagrant/base.shJDK_MAJORandJDK_ARCHwith sensible defaultsJDK_FULLis now constructed fromJDK_MAJORandJDK_ARCHif not explicitly provided/kafka-packages/jdk-{version}.tar.gzto/kafka-packages/jdk/jdk-{version}.tar.gzS3 Path Change
Old Path
New Path
Available JDKs in
s3://kafka-packages/jdk/jdk-7u80-linux-x64.tar.gzjdk-8u144-linux-x64.tar.gzjdk-8u161-linux-x64.tar.gzjdk-8u171-linux-x64.tar.gzjdk-8u191-linux-x64.tar.gzjdk-8u202-linux-x64.tar.gzjdk-11.0.2-linux-x64.tar.gzjdk-17-linux-x64.tar.gzjdk-18.0.2-linux-x64.tar.gzjdk-21.0.1-linux-x64.tar.gzjdk-21.0.1-linux-aarch64.tar.gzjdk-25-linux-aarch64.tar.gzjdk-25.0.1-linux-x64.tar.gzjdk-25.0.1-linux-aarch64.tar.gzjdk-25.0.2-linux-aarch64.tar.gzFuture JDK Releases
The validation regex supports all version formats:
17,25,2625.0.1,25.0.2,26.0.18u144,8u202Adding New JDK Versions
To add support for a new JDK version (e.g., JDK 26, 25.0.3):
jdk-{VERSION}-linux-{ARCH}.tar.gzaws s3 cp jdk-{VERSION}-linux-{ARCH}.tar.gz s3://kafka-packages/jdk/--jdk-version {VERSION} --jdk-arch {ARCH}No modifications to
base.shor any other scripts are needed.Benefits
--jdk-versionignoredTesting
Tested with different JDK versions to confirm the fix works correctly:
| Test | JDK_MAJOR | Expected | Actual | Result | Test Report | |------|-----------|----------|--------|--------|-------------| | JDK 17 |
17| javac 17.0.4 | javac 17.0.4 | ✅ | | JDK 25 |25| javac 25.0.2 | javac 25.0.2 | ✅ |Backward Compatibility
Reviewers: Manikumar Reddy manikumar.reddy@gmail.com, Chia-Ping Tsai chia7712@gmail.com