Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup Java 21 container #28918

Merged
merged 3 commits into from
Oct 20, 2023
Merged

Setup Java 21 container #28918

merged 3 commits into from
Oct 20, 2023

Conversation

Abacn
Copy link
Contributor

@Abacn Abacn commented Oct 10, 2023

Part of #28120

roll forward of #28833 and fix issues revealed after it is merged; also fix relevant java21 logic that found in #28969 (needed for build java agent for container). By the time of #28833 there was not a chance to test java21 build logic because gradle 8.3 not yet support compile java21. Now they are tested on #28969

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@Abacn
Copy link
Contributor Author

Abacn commented Oct 10, 2023

wordcount tested on Dataflow runner: https://console.cloud.google.com/dataflow/jobs/us-central1/2023-10-10_10_33_46-1503830796761633932?project=google.com%3Aclouddfe

jamm warnings are fixed by add-open jvm flags

@Abacn Abacn force-pushed the java21containeronly branch from 5c01659 to a67b78f Compare October 10, 2023 17:49
@Abacn Abacn marked this pull request as draft October 10, 2023 17:56
@codecov
Copy link

codecov bot commented Oct 10, 2023

Codecov Report

Merging #28918 (1c4b2d1) into master (0b50a17) will decrease coverage by 0.01%.
Report is 47 commits behind head on master.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #28918      +/-   ##
==========================================
- Coverage   38.38%   38.38%   -0.01%     
==========================================
  Files         686      686              
  Lines      101640   101652      +12     
==========================================
- Hits        39019    39018       -1     
- Misses      61041    61054      +13     
  Partials     1580     1580              
Flag Coverage Δ
python 30.00% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Abacn Abacn closed this Oct 12, 2023
@Abacn Abacn deleted the java21containeronly branch October 12, 2023 03:37
@Abacn Abacn restored the java21containeronly branch October 13, 2023 14:26
@Abacn Abacn reopened this Oct 13, 2023
@Abacn Abacn force-pushed the java21containeronly branch from a67b78f to 6f616d7 Compare October 13, 2023 14:27
@github-actions github-actions bot added the spark label Oct 13, 2023
@Abacn
Copy link
Contributor Author

Abacn commented Oct 13, 2023

Run Jpms Direct Java 11 PostCommit

@Abacn Abacn force-pushed the java21containeronly branch from 6f616d7 to f86c519 Compare October 17, 2023 18:43
Abacn added 2 commits October 17, 2023 14:45
* Add Java21 container

* fix spotless

* Update website

* Fix jamm

* align configs in recent change
@Abacn Abacn force-pushed the java21containeronly branch from f86c519 to 71fb124 Compare October 17, 2023 18:46
@Abacn
Copy link
Contributor Author

Abacn commented Oct 17, 2023

Run Seed Job

publish SDK container job succeeded: https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/6308/

this included java21 container: gcr.io/apache-beam-testing/beam-sdk/beam_java21_sdk

@Abacn Abacn marked this pull request as ready for review October 17, 2023 20:04
@Abacn
Copy link
Contributor Author

Abacn commented Oct 17, 2023

previously run jmps 11 test passed: https://ci-beam.apache.org/job/beam_PostCommit_Java_Jpms_Direct_Java11_PR/30/

@Abacn
Copy link
Contributor Author

Abacn commented Oct 17, 2023

R: @damccorm

@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@Abacn
Copy link
Contributor Author

Abacn commented Oct 18, 2023

R: @AnandInguva (since new Python support was done by you which was similar workflow)

Copy link
Contributor

@damccorm damccorm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good, just had a couple minor questions

@@ -120,7 +121,7 @@ plugins.withType(JavaPlugin).configureEach{
// JPMS requires JDK > 8
project.tasks.each {
it.onlyIf {
project.hasProperty("compileAndRunTestsWithJava17")
project.hasProperty('testJavaVersion')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this was just running on 17 previously - do we need tighter filtering here to avoid running on 11?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently Java11 JPMS test is done by setting

switches("-Dorg.gradle.java.home=${commonJobProperties.JAVA_11_HOME}")

so everything runs on Java11;

while Java17 test is done by setting

switches("-PcompileAndRunTestsWithJava17")

so all but compilation of jpms-test project are still run on Java8.

i.e. currently the combination of javahome=java 8+compileAndRunTestsWithJava11 option does not run test (where it supposed to). This change fixed the issue and make the behavior consistent for Java11/17/21

@@ -39,6 +40,7 @@ public class JvmVerification {
versionMapping.put("0034", v1_8);
versionMapping.put("0037", v11);
versionMapping.put("003d", v17);
versionMapping.put("0041", v21);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nonblocking since it seems to be consistently hex 002c + version, but where are these coming from?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are java bytecode versions. Yes from Java8 its been 44 (2c in hex) + version though no guarantee this trand continues. ref: https://javaalmanac.io/bytecode/versions/

Copy link
Contributor

@damccorm damccorm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Abacn
Copy link
Contributor Author

Abacn commented Oct 20, 2023

Run Java PreCommit

@Abacn
Copy link
Contributor Author

Abacn commented Oct 20, 2023

PreCommit Java paased on Jenkins, github action fn-execution test failure unrelated

@Abacn Abacn merged commit e7a6405 into apache:master Oct 20, 2023
41 of 45 checks passed
@Abacn Abacn deleted the java21containeronly branch October 20, 2023 15:39
kkdoon pushed a commit to twitter-forks/beam that referenced this pull request Oct 21, 2023
* Setup Java 21 container (apache#28833)

* Add Java21 container

* fix spotless

* Update website

* Fix jamm

* align configs in recent change

* incremental fixes

* add more comments for when jpms test enabled
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants