BWC builds failing with gradle error (irregularly)

[tvernum]

This seems to have been failing once or twice per day for the past week.

I think the first of the recent failures was https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/937/console
The most recent is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu&&virtual/155/console

There was also an earlier failure here which was presumed to be due to the gradle 5 bump, but it seems to have been a one-off so we didn't pin down the root cause.

The failure is (from build 155):

23:25:54 1: Task failed with an exception.
23:25:54 -----------
23:25:54 * Where:
23:25:54 Build file '/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/ubuntu&&virtual/distribution/bwc/build.gradle' line: 146
23:25:54 
23:25:54 * What went wrong:
23:25:54 Execution failed for task ':distribution:bwc:staged:buildBwcVersion'.
23:25:54 > Cannot invoke method add() on null object
23:25:54 
23:25:54 * Try:
23:25:54 Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
23:25:54 ==============================================================================
23:25:54 
23:25:54 2: Task failed with an exception.
23:25:54 -----------
23:25:54 * Where:
23:25:54 Build file '/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/ubuntu&&virtual/distribution/bwc/build.gradle' line: 146
23:25:54 
23:25:54 * What went wrong:
23:25:54 Execution failed for task ':distribution:bwc:minor:buildBwcVersion'.
23:25:54 > Cannot invoke method add() on null object
23:25:54 

[elasticmachine]

Pinging @elastic/es-core-infra

[tvernum]

Also https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+corretto-periodic/ES_BUILD_JAVA=java11,label=amazon/27/console

[alpar-t]

The scan for the failures that are still available:
https://gradle.com/s/q2gygthoxrmkg
https://gradle.com/s/iiz7yqtj3ne7w

[alpar-t]

The relevant stack trace is:

at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:34)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at java_util_List$add$0.call(Unknown Source)
at org.elasticsearch.gradle.BuildPlugin.requireJavaHome(BuildPlugin.groovy:412)
at org.elasticsearch.gradle.BuildPlugin$requireJavaHome$13.callStatic(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:55)
at org.elasticsearch.gradle.BuildPlugin$requireJavaHome$13.callStatic(Unknown Source)
at org.elasticsearch.gradle.BuildPlugin.getJavaHome(BuildPlugin.groovy:417)
at org.elasticsearch.gradle.BuildPlugin$getJavaHome$12.callStatic(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:55)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:196)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:216)

We use a map and [:].withDefault{key -> return []} , then we do requiredJavaVersions.get(version).add(task). The get should retrun an empty list instead of null, but this doesn't seem to be the case. Relevant SO: https://stackoverflow.com/questions/24738903/creating-map-using-withdefault-causing-null-when-putting-element

[alpar-t]

https://elasticsearch-ci.elastic.co/view/Elasticsearch/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/193/console