Jenkins will require Java 11 from September LTS release

[richardlau]

We probably need to start planning for this:

https://www.jenkins.io/blog/2022/06/28/require-java-11/

Beginning with Jenkins 2.357 (released on June 28, 2022) and the forthcoming September LTS release, Jenkins requires Java 11.

Both CI servers are currently running Java 8.

cc @nodejs/build-infra

[richardlau]

Looking at the release schedules (nodejs/Release#737, nodejs/Release#658, nodejs/Release#567) I'm going to plan to do the updates on 25 Aug 2022, with the proviso that if a security release is needed/happens during August that requires a CI lockdown I would shift doing the updates to between after the security release and before lifting the CI lockdown.

[richardlau]

Maintenance has started. I've shutdown the test Jenkins CI, powered off the server and am in the process of taking a snapshot in DigitalOcean (infra-digitalocean-ubuntu14-x64-1-1661424784910).

[richardlau]

As a (self-)reminder: just before shutting down the server I unselected all "build" (can start builds) permissions to prevent builds from being started during the updates and will need to put them back when the maintenance is over.

[richardlau]

I've managed to upgrade the server from Ubuntu 16.04->18.04->20.04->22.04.1. Had a nameserver issue, which we've run into before with DO droplets and required the same fix as #2745 (comment). Working on switching to Java 17 now.

[richardlau]

Also the initial update to Ubuntu 18.04 still had the server on a very old kernel:

Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 3.13.0-57-generic x86_64)

There's a "Kernel" page for the droplet in DigitalOcean which I switched to "Grubloader" and after a restart the server then used the expected kernel for the Ubuntu release. We're now on:

Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-46-generic x86_64)

[richardlau]

Have had to apply afonsof/jenkins-material-theme#193. It looks like the theme were using has some issues with the latest LTS Jenkins release 😞 .

[richardlau]

Run into an issue with Java 17 -- when I start Jenkins with it, attempting to start a build that uses our VersionSelectorScript.groovy is now erroring:
e.g. https://ci.nodejs.org/job/node-test-commit-linuxone/33786/console

17:34:16 FATAL: For input string: "[49, 57]"
17:34:16 java.lang.NumberFormatException: For input string: "[49, 57]"
17:34:16 	at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
17:34:16 	at java.base/java.lang.Integer.parseInt(Integer.java:668)
17:34:16 	at java.base/java.lang.Integer.valueOf(Integer.java:999)
17:34:16 	at org.codehaus.groovy.runtime.StringGroovyMethods.toInteger(StringGroovyMethods.java:3319)
17:34:16 	at org.codehaus.groovy.runtime.dgm$1162.invoke(Unknown Source)
17:34:16 	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
17:34:16 	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
17:34:16 	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
17:34:16 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
17:34:16 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
17:34:16 	at Script1.run(Script1.groovy:161)
17:34:16 	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:574)
17:34:16 	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:612)
17:34:16 	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:583)
17:34:16 	at groovy.lang.Script.evaluate(Script.java:210)
17:34:16 	at groovy.lang.Script$evaluate$0.callCurrent(Unknown Source)
17:34:16 	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
17:34:16 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
17:34:16 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:169)
17:34:16 	at Script1.run(Script1.groovy:11)

49 and 57 are ASCII codes for "1" and "9", e.g. "19". Problem goes away (i.e. job successfully starts) if I switch back to Java 8 and restart Jenkins 😞 .

[richardlau]

Going to try Java 11 for comparison.

[richardlau]

Nope, same error with Java 11 😞.
https://ci.nodejs.org/job/node-test-commit-linuxone/33788/console

[richardlau]

Looks like we're running into jenkinsci/matrix-groovy-execution-strategy-plugin#20 😞 .

[richardlau]

Looks like the current LTS version of Jenkins and the custom theme we have don't play well together -- black text on black background for console output in jobs and several missing icons. I've removed our custom theme for now.

[richardlau]

Looks like we're running into jenkinsci/matrix-groovy-execution-strategy-plugin#20 😞 .

Implemented a workaround in #3019.
Needs a small fixup: #3020

[richardlau]

Summary:

• Test CI server has been updated from Ubuntu 16.04->18.04->20.04->22.04.1. Jenkins is now running on Java 17.
• Release CI server has been updated from Ubuntu 18.04->20.04. Jenkins is now running on Java 17. There was a disk space issue when attempting upgrading to Ubuntu 22.04 that I haven't had time to look into, but I have no immediate concerns about running on Ubuntu 20.04.

[tniessen]

Great work @richardlau, thank you!