Jenkins agents need newer Java (11+)

[richardlau]

It looks like after updating to the latest Jenkins LTS (https://www.jenkins.io/security/advisory/2022-09-09/) several agents are failing to connect to the server.
e.g. example failure from one of the agent logs:

WARNING: LinkageError while performing UserRequest:hudson.slaves.SlaveComputer$SlaveVersion@435cd082
java.lang.UnsupportedClassVersionError: Failed to load hudson.slaves.SlaveComputer$SlaveVersion
...
Caused by: java.lang.UnsupportedClassVersionError: JVMCFRE003 bad major version; class=hudson/slaves/SlaveComputer$SlaveVersion, offset=6
...
Sep 14, 2022 11:38:51 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated

Looks like now that Jenkins has dropped support for Java 8 (#2984) the agents also require Java 11+.

[richardlau]

For the time being I've rolled back the Jenkins update by backing up the jenkins.war file on the test CI server and copying over the one from the release server (which hadn't been updated yet). We'll need to update eventually but this will buy some time to get Java updated on the test machines.

[richardlau]

Via ansible --list-hosts "test-*" | cut -d "-" -f 3 | sort -u these are the OS's we currently have in the test matrix:

os	current Java	most recent Java from distro	Notes
aix71	8 ➡️ 17	-	IBM Semuru Java 11/17 requires XL C++ Runtime environment 16
aix72	8 ➡️ 17	-	IBM Semuru
aix73	8 ➡️ 17	-	IBM Semuru
centos7	8 ➡️ 11	11	ppc64 install Adoptium binaries and has been updated to Java 17
debian10	11	11
debian8	8	8	Should be removed from CI
debian9	8	8	Should be removed from CI
fedora32	8 ➡️ 11	11	Should be updated to Fedora 35
fedora34	8 ➡️ 17	17	Should be updated to Fedora 36
freebsd12	8 ➡️ 11	16
ibmi73	11	11
macos10.14	8 ➡️ 17	19
macos10.15	8 ➡️ 17	19
macos11	8 ➡️ 17	19
rhel7	11	11
rhel8	17
smartos18	8 ➡️ 11	11
smartos20	8 ➡️ 11	15	Move to 11
ubuntu1404	8		Should be removed from CI
ubuntu1604	8	9	Should be removed from CI
ubuntu1804	8 ➡️ 17	17
ubuntu2004
win10	8 ➡️ 17	19 (Temurin)
win2012r2	8 ➡️ 17	19 (Temurin)
win2016	8 ➡️ 17	19 (Temurin)
zos24

[richardlau]

Just finished updating the Windows machines. Remaining platforms:

• macos
• smartos
• End-of-Life Linux (e.g. debian 8) which should be removed from CI

[targos]

Is this something I may be able to do as a first task?

[richardlau]

Is this something I may be able to do as a first task?

Yes 🙂. We use Ansible to manage our CI hosts. In theory updating the Java being used for an existing platform would be:

• Update https://github.com/nodejs/build/blob/main/ansible/roles/java-base/vars/main.yml
• Check if the platform uses a non-default path to the Java binary and update if necessary:

  build/ansible/roles/jenkins-worker/vars/main.yml

  Lines 80 to 97 in 7fdcf57

  	java_path: {
  	'ibmi73': '/QOpenSys/pkgs/lib/jvm/openjdk-11/bin/java',
  	'macos10.10': 'java',
  	'macos10.11': 'java',
  	'macos10.12': 'java',
  	'macos10.13': 'java',
  	'macos10.14': 'java',
  	'macos10.15': 'java',
  	'macos11': 'java',
  	# Currently hardcoded untill adopt have their build available
  	'macos11.0': '/Users/{{ ansible_user }}/zulu8.52.0.23-ca-jdk8.0.282-macosx_aarch64/bin/java',
  	'smartos15': '/opt/local/java/openjdk8/bin/java',
  	'smartos16': '/opt/local/java/openjdk8/bin/java',
  	'smartos17': '/opt/local/java/openjdk8/bin/java',
  	'smartos18': '/opt/local/java/openjdk8/bin/java',
  	'smartos20': '/opt/local/java/openjdk8/bin/java',
  	'zos24': '/usr/lpp/java/J8.0_64/bin/java'
  	}

When actually running the Ansible scripts against a CI host it's usually better to check that the host isn't running a build (otherwise the script may interrupt the in progress build) -- marking the host as offline in Jenkins will stop any jobs being scheduled on it (just remember to set it back to being online in Jenkins afterwards).

Specifically for the macs one caveat is that when @AshCripps added the M1's we had to use a Java SDK from Azul as that was the only native Apple Silicon Java available at the time -- I understand that there are now Adoptium binaries available so in theory we could simplify the playbook by removing the M1 specific tasks:

build/ansible/roles/java-base/tasks/main.yml

Lines 68 to 78 in 7fdcf57

	- name: Fetch java (Apple Silicon)
	when: java.rc > 0 and os|startswith("macos11") and arch == "arm64"
	shell:
	chdir: "/Users/{{ ansible_user }}"
	cmd: "curl -L -o zulu8.52.0.23-ca-jdk8.0.282-macosx_aarch64.tar.gz https://cdn.azul.com/zulu/bin/zulu8.52.0.23-ca-jdk8.0.282-macosx_aarch64.tar.gz"
	- name: Extract java (Apple Silicon)
	when: java.rc > 0 and os|startswith("macos11") and arch == "arm64"
	shell:
	chdir: "/Users/{{ ansible_user }}"
	cmd: "tar -xf zulu8.52.0.23-ca-jdk8.0.282-macosx_aarch64.tar.gz"

[targos]

According to https://github.com/AdoptOpenJDK/homebrew-openjdk, we should move all macs to the temurin cask.
PR: #3085

[targos]

From #3085:

macOS update status

Updated to Java 17

• release-nearform-macos10.15-x64-1
• release-nearform-macos11.0-arm64-1
• release-orka-macos10.15-x64-1
• test-nearform-macos10.15-x64-1
• test-nearform-macos10.15-x64-2
• test-nearform-macos11.0-arm64-1
• test-orka-macos10.14-x64-1
• test-orka-macos10.14-x64-2
• test-orka-macos10.14-x64-3
• test-orka-macos11-x64-1
• test-orka-macos11-x64-2

Impossible to connect via SSH

• release-macstadium-macos11.0-arm64-1
• test-macstadium-macos11.0-arm64-3
• test-macstadium-macos11.0-arm64-4

Offline

• test-nearform-macos10.15-x64-3
• test-orka-macos10.15-x64-1
• test-orka-macos10.15-x64-2